當前位置：首頁 > 编程资源 > 编程问答 >内容正文

编程问答

Beautifulsoup官方文档

發(fā)布時間：2023/12/20 编程问答 31 豆豆

生活随笔收集整理的這篇文章主要介紹了 Beautifulsoup官方文档小編覺得挺不錯的,現(xiàn)在分享給大家,幫大家做個參考.

Beautiful Soup 中文文檔

原文 by?Leonard Richardson?(leonardr@segfault.org)?
翻譯 by?Richie Yan?(richieyan@gmail.com)?
###如果有些翻譯的不準確或者難以理解，直接看例子吧。###?

英文原文點這里

Beautiful Soup?是用Python寫的一個HTML/XML的解析器，它可以很好的處理不規(guī)范標記并生成剖析樹(parse tree)。它提供簡單又常用的導航（navigating），搜索以及修改剖析樹的操作。它可以大大節(jié)省你的編程時間。對于Ruby，使用Rubyful Soup。

這個文檔說明了Beautiful Soup 3.0主要的功能特性，并附有例子。從中你可以知道這個庫有哪些好處，它是怎樣工作的，怎樣讓它幫做你想做的事以及你該怎樣做當它做的和你期待不一樣。

快速開始
剖析文檔
- 剖析 HTML
- 剖析 XML
- 如果它不工作
使用Unicode的Beautiful Soup, Dammit
輸出文檔
剖析樹
- Tags的屬性
Navigating 剖析樹
- parent
- contents
- string
- nextSibling?and?previousSibling
- next?and?previous
- 遍歷Tag
- 使用標簽名作為成員
Searching 剖析樹
- The basic find method:?findAll(name, attrs, recursive, text, limit, **kwargs)
  - 使用CSS類查找
  - 像?findall一樣調(diào)用tag
- find(name, attrs, recursive, text, **kwargs)
- first哪里去了?
Searching 剖析樹內(nèi)部
- findNextSiblings(name, attrs, text, limit, **kwargs)?and?findNextSibling(name, attrs, text, **kwargs)
- findPreviousSiblings(name, attrs, text, limit, **kwargs)?and?findPreviousSibling(name, attrs, text, **kwargs)
- findAllNext(name, attrs, text, limit, **kwargs)?and?findNext(name, attrs, text, **kwargs)
- findAllPrevious(name, attrs, text, limit, **kwargs)?and?findPrevious(name, attrs, text, **kwargs)
Modifying 剖析樹
- 改變屬性值
- 刪除元素
- 替換元素
- 添加新元素
常見問題(Troubleshooting)
- 為什么Beautiful Soup不能打印我的no-ASCII字符?
- Beautiful Soup 弄丟了我給的數(shù)據(jù)!為什么?為什么?????
- Beautiful Soup 太慢了!
高級主題
- 產(chǎn)生器(Generators)
- 其他的內(nèi)部剖析器
- 定制剖析器(Parser)
- 實體轉(zhuǎn)換
- 使用正則式處理糟糕的數(shù)據(jù)
- 玩玩SoupStrainers
- 通過剖析部分文檔來提升效率
- 使用extract改進內(nèi)存使用
其它
- 使用Beautiful Soup的其他應(yīng)用
- 類似的庫
小結(jié)

快速開始

從這里獲得 Beautiful Soup。?變更日志?描述了3.0 版本與之前版本的不同。

在程序中中導入 Beautiful Soup庫:

from BeautifulSoup import BeautifulSoup # For processing HTML from BeautifulSoup import BeautifulStoneSoup # For processing XML import BeautifulSoup # To get everything

下面的代碼是Beautiful Soup基本功能的示范。你可以復(fù)制粘貼到你的python文件中，自己運行看看。

from BeautifulSoup import BeautifulSoup import re doc = ['<html><head><title>Page title</title></head>', '<body>This is paragraph one.', 'This is paragraph two.', '</html>'] soup = BeautifulSoup(''.join(doc)) print soup.prettify() #<html> # <head> # <title> # Page title # </title> # </head> # <body> # # This is paragraph # # one # # . # # # This is paragraph # # two # # . # # </body> #</html>

navigate soup的一些方法:

soup.contents[0].name #u'html' soup.contents[0].contents[0].name #u'head' head = soup.contents[0].contents[0] head.parent.name #u'html' head.next #<title>Page title</title> head.nextSibling.name #u'body' head.nextSibling.contents[0] #This is paragraph one. head.nextSibling.contents[0].nextSibling #This is paragraph two.

下面是一些方法搜索soup，獲得特定標簽或有著特定屬性的標簽：

titleTag = soup.html.head.title titleTag #<title>Page title</title> titleTag.string #u'Page title' len(soup('p')) #2 soup.findAll('p', align="center") #[This is paragraph one. ] soup.find('p', align="center") #This is paragraph one. soup('p', align="center")[0]['id'] #u'firstpara' soup.find('p', align=re.compile('^b.*'))['id'] #u'secondpara' soup.find('p').b.string #u'one' soup('p')[1].b.string #u'two'

修改soup也很簡單：

titleTag['id'] = 'theTitle' titleTag.contents[0].replaceWith("New title") soup.html.head #<head><title id="theTitle">New title</title></head> soup.p.extract() soup.prettify() #<html> # <head> # <title id="theTitle"> # New title # </title> # </head> # <body> # # This is paragraph # # two # # . # # </body> #</html> soup.p.replaceWith(soup.b) #<html> # <head> # <title id="theTitle"> # New title # </title> # </head> # <body> # # two # # </body> #</html> soup.body.insert(0, "This page used to have ") soup.body.insert(2, " tags!") soup.body #<body>This page used to have two tags!</body>

一個實際例子，用于抓取?ICC Commercial Crime Services weekly piracy report頁面, 使用Beautiful Soup剖析并獲得發(fā)生的盜版事件:

import urllib2 from BeautifulSoup import BeautifulSoup page = urllib2.urlopen("http://www.icc-ccs.org/prc/piracyreport.php") soup = BeautifulSoup(page) for incident in soup('td', width="90%"): where, linebreak, what = incident.contents[:3] print where.strip() print what.strip() print

剖析文檔

Beautiful Soup使用XML或HTML文檔以字符串的方式(或類文件對象)構(gòu)造。它剖析文檔并在內(nèi)存中創(chuàng)建通訊的數(shù)據(jù)結(jié)構(gòu)

如果你的文檔格式是非常標準的，解析出來的數(shù)據(jù)結(jié)構(gòu)正如你的原始文檔。但是如果你的文檔有問題，Beautiful Soup會使用heuristics修復(fù)可能的結(jié)構(gòu)問題。

剖析 HTML

使用?BeautifulSoup?類剖析HTML文檔。?BeautifulSoup會得出以下一些信息：

有些標簽可以內(nèi)嵌 (<BLOCKQUOTE>) ，有些不行 ().
table和list標簽有一個自然的內(nèi)嵌順序。例如，<TD> 標簽內(nèi)為 <TR> 標簽，而不會相反。
<SCRIPT> 標簽的內(nèi)容不會被剖析為HTML。
<META> 標簽可以知道文檔的編碼類型。

這是運行例子:

from BeautifulSoup import BeautifulSoup html = "<html>Para 1Para 2<blockquote>Quote 1<blockquote>Quote 2" soup = BeautifulSoup(html) print soup.prettify() #<html> # # Para 1 # # # Para 2 # <blockquote> # Quote 1 # <blockquote> # Quote 2 # </blockquote> # </blockquote> # #</html>

注意：BeautifulSoup?會智能判斷那些需要添加關(guān)閉標簽的位置，即使原始的文檔沒有。

也就是說那個文檔不是一個有效的HTML，但是它也不是太糟糕。下面是一個比較糟糕的文檔。在一些問題中，它的<FORM>的開始在 <TABLE> 外面，結(jié)束在<TABLE>里面。 (這種HTML在一些大公司的頁面上也屢見不鮮)

from BeautifulSoup import BeautifulSoup html = """ <html> <form> <table> <td><input name="input1">Row 1 cell 1 <tr><td>Row 2 cell 1 </form> <td>Row 2 cell 2 This sure is a long cell </body> </html>"""

Beautiful Soup 也可以處理這個文檔：

print BeautifulSoup(html).prettify() #<html> # <form> # <table> # <td> # <input name="input1" /> # Row 1 cell 1 # </td> # <tr> # <td> # Row 2 cell 1 # </td> # </tr> # </table> # </form> # <td> # Row 2 cell 2 # # This # sure is a long cell # </td> #</html>

table的最后一個單元格已經(jīng)在標簽<TABLE>外了；Beautiful Soup 決定關(guān)閉<TABLE>標簽當它在<FORM>標簽?zāi)睦镪P(guān)閉了。寫這個文檔家伙原本打算使用<FORM>標簽擴展到table的結(jié)尾，但是Beautiful Soup 肯定不知道這些。即使遇到這樣糟糕的情況, Beautiful Soup 仍可以剖析這個不合格文檔，使你開業(yè)存取所有數(shù)據(jù)。

剖析 XML

BeautifulSoup?類似瀏覽器，是個具有啟發(fā)性的類，可以盡可能的推測HTML文檔作者的意圖。但是XML沒有固定的標簽集合，因此這些啟發(fā)式的功能沒有作用。因此BeautifulSoup處理XML不是很好。

使用BeautifulStoneSoup類剖析XML文檔。它是一個概括的類，沒有任何特定的XML方言已經(jīng)簡單的標簽內(nèi)嵌規(guī)則。下面是范例：

from BeautifulSoup import BeautifulStoneSoup xml = "<doc><tag1>Contents 1<tag2>Contents 2<tag1>Contents 3" soup = BeautifulStoneSoup(xml) print soup.prettify() #<doc> # <tag1> # Contents 1 # <tag2> # Contents 2 # </tag2> # </tag1> # <tag1> # Contents 3 # </tag1> #</doc>

BeautifulStoneSoup的一個主要缺點就是它不知道如何處理自結(jié)束標簽。 HTML 有固定的自結(jié)束標簽集合，但是XML取決對應(yīng)的DTD文件。你可以通過傳遞selfClosingTags?的參數(shù)的名字到?BeautifulStoneSoup的構(gòu)造器中，指定自結(jié)束標簽:

from BeautifulSoup import BeautifulStoneSoup xml = "<tag>Text 1<selfclosing>Text 2" print BeautifulStoneSoup(xml).prettify() #<tag> # Text 1 # <selfclosing> # Text 2 # </selfclosing> #</tag> print BeautifulStoneSoup(xml, selfClosingTags=['selfclosing']).prettify() #<tag> # Text 1 # <selfclosing /> # Text 2 #</tag>

如果它不工作

這里有?一些其他的剖析類?使用與上述兩個類不同的智能感應(yīng)。你也可以子類化以及定制一個剖析器?使用你自己的智能感應(yīng)方法。

使用Unicode的Beautiful Soup，Dammit

當你的文檔被剖析之后，它就自動被轉(zhuǎn)換為unicode。 Beautiful Soup 只存儲Unicode字符串。

from BeautifulSoup import BeautifulSoup soup = BeautifulSoup("Hello") soup.contents[0] #u'Hello' soup.originalEncoding #'ascii'

使用UTF-8編碼的日文文檔例子：

from BeautifulSoup import BeautifulSoup soup = BeautifulSoup("\xe3\x81\x93\xe3\x82\x8c\xe3\x81\xaf") soup.contents[0] #u'\u3053\u308c\u306f' soup.originalEncoding #'utf-8' str(soup) #'\xe3\x81\x93\xe3\x82\x8c\xe3\x81\xaf' #Note: this bit uses EUC-JP, so it only works if you have cjkcodecs #installed, or are running Python 2.4. soup.__str__('euc-jp') #'\xa4\xb3\xa4\xec\xa4\xcf'

Beautiful Soup 使用一個稱為UnicodeDammit?的類去來檢測文檔的編碼，并將其轉(zhuǎn)換為Unicode。如果你需要為其他文檔（沒有石油Beautiful Soup剖析過得文檔）使用這轉(zhuǎn)換，你也可以直接使用UnicodeDammit。它是基于Universal Feed Parser開發(fā)的。

如果你使用Python2.4之前的版本，請下載和安裝cjkcodecs?以及iconvcodec?是python支持更多的編碼，特別是CJK編碼。要想更好地自動檢測，你也要安裝chardet

Beautiful Soup 會按順序嘗試不同的編碼將你的文檔轉(zhuǎn)換為Unicode：

可以通過fromEncoding參數(shù)傳遞編碼類型給soup的構(gòu)造器
通過文檔本身找到編碼類型：例如XML的聲明或者HTML文檔http-equiv的META標簽。如果Beautiful Soup在文檔中發(fā)現(xiàn)編碼類型，它試著使用找到的類型轉(zhuǎn)換文檔。但是，如果你明顯的指定一個編碼類型，并且成功使用了編碼：這時它會忽略任何它在文檔中發(fā)現(xiàn)的編碼類型。
通過嗅探文件開頭的一下數(shù)據(jù)，判斷編碼。如果編碼類型可以被檢測到，它將是這些中的一個：UTF-*編碼，EBCDIC或者ASCII。
通過chardet?庫,嗅探編碼，如果你安裝了這個庫。
UTF-8
Windows-1252

Beautiful Soup總是會猜對它可以猜測的。但是對于那些沒有聲明以及有著奇怪編碼的文檔，它會常常會失敗。這時，它會選擇Windows-1252編碼，這個可能是錯誤的編碼。下面是EUC-JP的例子，Beautiful Soup猜錯了編碼。(重申一下：因為它使用了EUC-JP，這個例子只會在 python 2.4或者你安裝了cjkcodecs的情況下才工作。)：

from BeautifulSoup import BeautifulSoup euc_jp = '\xa4\xb3\xa4\xec\xa4\xcf' soup = BeautifulSoup(euc_jp) soup.originalEncoding #'windows-1252' str(soup) #'\xc2\xa4\xc2\xb3\xc2\xa4\xc3\xac\xc2\xa4\xc3\x8f' # Wrong!

但如果你使用fromEncoding參數(shù)指定編碼，它可以正確的剖析文檔，并可以將文檔轉(zhuǎn)換為UTF-8或者轉(zhuǎn)回EUC-JP。

soup = BeautifulSoup(euc_jp, fromEncoding="euc-jp") soup.originalEncoding #'windows-1252' str(soup) #'\xe3\x81\x93\xe3\x82\x8c\xe3\x81\xaf' # Right! soup.__str__(self, 'euc-jp') == euc_jp #True

如果你指定Beautiful Soup使用 Windows-1252編碼（或者類似的編碼如ISO-8859-1，ISO-8859-2), Beautiful Soup會找到并破壞文檔的smart quotes以及其他的Windows-specific 字符。這些字符不會轉(zhuǎn)換為相應(yīng)的Unicode,而是將它們變?yōu)镠TML entities(BeautifulSoup) 或者XML entitis(BeautifulStoneSoup)。

但是，你可以指定參數(shù)smartQuotesTo=None?到soup構(gòu)造器：這時 smart quotes會被正確的轉(zhuǎn)換為Unicode。你也可以指定smartQuotesTo為"xml"或"html" 去改變BeautifulSoup和BeautifulStoneSoup的默認操作。

from BeautifulSoup import BeautifulSoup, BeautifulStoneSoup text = "Deploy the \x91SMART QUOTES\x92!" str(BeautifulSoup(text)) #'Deploy the ‘SMART QUOTES’!' str(BeautifulStoneSoup(text)) #'Deploy the ‘SMART QUOTES’!' str(BeautifulSoup(text, smartQuotesTo="xml")) #'Deploy the ‘SMART QUOTES’!' BeautifulSoup(text, smartQuotesTo=None).contents[0] #u'Deploy the \u2018SMART QUOTES\u2019!'

輸出文檔

你可以使用?str函數(shù)將Beautiful Soup文檔（或者它的子集）轉(zhuǎn)換為字符串，或者使用它的code>prettify或renderContents。你也可以使用unicode函數(shù)以Unicode字符串的形式獲得。

prettify?方法添加了一些換行和空格以便讓文檔結(jié)構(gòu)看起來更清晰。它也將那些只包含空白符的，可能影響一個XML文檔意義的文檔節(jié)點(nodes)剔除(strips out)。?str和unicode函數(shù)不會剔除這些節(jié)點，他們也不會添加任何空白符。

看看這個例子：

from BeautifulSoup import BeautifulSoup doc = "<html><h1>Heading</h1>Text" soup = BeautifulSoup(doc) str(soup) #'<html><h1>Heading</h1>Text</html>' soup.renderContents() #'<html><h1>Heading</h1>Text</html>' soup.__str__() #'<html><h1>Heading</h1>Text</html>' unicode(soup) #u'<html><h1>Heading</h1>Text</html>' soup.prettify() #'<html>\n <h1>\n Heading\n </h1>\n \n Text\n \n</html>' print soup.prettify() #<html> # <h1> # Heading # </h1> # # Text # #</html>

可以看到使用文檔中的tag成員時?str和renderContents返回的結(jié)果是不同的。

heading = soup.h1 str(heading) #'<h1>Heading</h1>' heading.renderContents() #'Heading'

當你調(diào)用__str__,prettify或者renderContents時，你可以指定輸出的編碼。默認的編碼(str使用的)是UTF-8。下面是處理ISO-8851-1的串并以不同的編碼輸出同樣的串的例子。

from BeautifulSoup import BeautifulSoup doc = "Sacr\xe9 bleu!" soup = BeautifulSoup(doc) str(soup) #'Sacr\xc3\xa9 bleu!' # UTF-8 soup.__str__("ISO-8859-1") #'Sacr\xe9 bleu!' soup.__str__("UTF-16") #'\xff\xfeS\x00a\x00c\x00r\x00\xe9\x00 \x00b\x00l\x00e\x00u\x00!\x00' soup.__str__("EUC-JP") #'Sacr\x8f\xab\xb1 bleu!'

如果原始文檔含有編碼聲明，Beautiful Soup會將原始的編碼聲明改為新的編碼。也就是說，你載入一個HTML文檔到BeautifulSoup后，在輸出它，不僅HTML被清理過了，而且可以明顯的看到它已經(jīng)被轉(zhuǎn)換為UTF-8。

這是HTML的例子：

from BeautifulSoup import BeautifulSoup doc = """<html> <meta http-equiv="Content-type" content="text/html; charset=ISO-Latin-1" > Sacr\xe9 bleu! </html>""" print BeautifulSoup(doc).prettify() #<html> # <meta http-equiv="Content-type" content="text/html; charset=utf-8" /> # Sacré bleu! #</html>

這是XML的例子：

from BeautifulSoup import BeautifulStoneSoup doc = """<?xml version="1.0" encoding="ISO-Latin-1">Sacr\xe9 bleu!""" print BeautifulStoneSoup(doc).prettify() #<?xml version='1.0' encoding='utf-8'> #Sacré bleu!

剖析樹

到目前為止，我們只是載入文檔，然后再輸出它。現(xiàn)在看看更讓我們感興趣的剖析樹： Beautiful Soup剖析一個文檔后生成的數(shù)據(jù)結(jié)構(gòu)。

剖析對象 (BeautifulSoup或?BeautifulStoneSoup的實例)是深層嵌套(deeply-nested), 精心構(gòu)思的(well-connected)的數(shù)據(jù)結(jié)構(gòu)，可以與XML和HTML結(jié)構(gòu)相互協(xié)調(diào)。剖析對象包括2個其他類型的對象，Tag對象，用于操縱像<TITLE> ，這樣的標簽；NavigableString對象，用于操縱字符串，如"Page title"和"This is paragraph"。

NavigableString的一些子類 (CData,?Comment,?Declaration, and?ProcessingInstruction), 也處理特殊XML結(jié)構(gòu)。它們就像NavigableString一樣, 除了但他們被輸出時，他們會被添加一些額外的數(shù)據(jù)。下面是一個包含有注釋(comment)的文檔：

from BeautifulSoup import BeautifulSoup import re hello = "Hello! " commentSoup = BeautifulSoup(hello) comment = commentSoup.find(text=re.compile("nice")) comment.__class__ #<class 'BeautifulSoup.Comment'> comment #u"I've got to be nice to get what I want." comment.previousSibling #u'Hello! ' str(comment) #"" print commentSoup #Hello!

現(xiàn)在，我們深入研究一下我們開頭使用的那個文檔:

from BeautifulSoup import BeautifulSoup doc = ['<html><head><title>Page title</title></head>', '<body>This is paragraph one.', 'This is paragraph two.', '</html>'] soup = BeautifulSoup(''.join(doc)) print soup.prettify() #<html> # <head> # <title> # Page title # </title> # </head> # <body> # # This is paragraph # # one # # . # # # This is paragraph # # two # # . # # </body> #</html>

Tag的屬性

Tag和NavigableString對象有很多有用的成員，在?Navigating剖析樹和?Searching剖析樹中我們會更詳細的介紹。現(xiàn)在，我們先看看這里使用的Tag成員：屬性

SGML標簽有屬性：.例如，在上面那個HTML?中每個標簽都有"id"屬性和"align"屬性。你可以將Tag看成字典來訪問標簽的屬性：

firstPTag, secondPTag = soup.findAll('p') firstPTag['id'] #u'firstPara' secondPTag['id'] #u'secondPara'

NavigableString對象沒有屬性;只有Tag?對象有屬性。

Navigating剖析樹

Tag?對象都有如下含有所有的成員的列表(盡管，某些實際的成員值可能為None).?NavigableString對象也有下面這些成員，除了contents和?string成員。

parent

上面那個?例子中, <HEAD>?Tag的parent是<HTML>?Tag. <HTML>?Tag?的parent是BeautifulSoup?剖析對象自己。剖析對象的parent是None. 利用parent，你可以向前遍歷剖析樹。

soup.head.parent.name #u'html' soup.head.parent.parent.__class__.__name__ #'BeautifulSoup' soup.parent == None #True

使用parent向前遍歷樹。使用contents向后遍歷樹。?contents是Tag的有序列表，?NavigableString?對象包含在一個頁面元素內(nèi)。只有最高層的剖析對象和?Tag?對象有contents。NavigableString?只有strings，不能包含子元素，因此他們也沒有contents.

在上面的例子中，?contents?的第一個?Tag是個列表，包含一個?NavigableString?("This is paragraph "), 一個?Tag, 和其它的?NavigableString?(".")。而contents?的?Tag: 包含一個NavigableString?("one")的列表。

pTag = soup.p pTag.contents #[u'This is paragraph ', one, u'.'] pTag.contents[1].contents #[u'one'] pTag.contents[0].contents #AttributeError: 'NavigableString' object has no attribute 'contents'

string

為了方便，如果一個標簽只有一個子節(jié)點且是字符串類型，這個自己可以這樣訪問?tag.string，等同于tag.contents[0]的形式。在上面的例子中，?soup.b.string是個NavigableString對象，它的值是Unicode字符串"one". 這是剖析樹中Tag?的第一個string。

soup.b.string #u'one' soup.b.contents[0] #u'one'

但是soup.p.string是None, 剖析中的第一個?Tag?擁有多個子元素。soup.head.string也為None, 雖然<HEAD> Tag只有一個子節(jié)點，但是這個子節(jié)點是Tag類型 (<TITLE>?Tag)，不是NavigableString。

soup.p.string == None #True soup.head.string == None #True

nextSibling和previousSibling

使用它們你可以跳往在剖析樹中同等層次的下一個元素。在上面的文檔中， <HEAD>?Tag的nextSibling?是<BODY>?Tag，因為<BODY>?Tag是在<html>?Tag的下一層。 <BODY>標簽的nextSibling為None, 因為<HTML>下一層沒有標簽是直接的在它之后。

soup.head.nextSibling.name #u'body' soup.html.nextSibling == None #True

相應(yīng)的<BODY>?Tag的previousSibling是<HEAD>標簽， <HEAD>?Tag的previousSibling為None：

soup.body.previousSibling.name #u'head' soup.head.previousSibling == None #True

更多例子：?Tag的第一個nextSibling是第二個 ?Tag。第二個Tag里的Tag的previousSibling是?NavigableString"This is paragraph"。這個NavigableString的previousSibling是None, 不會是第一個?Tag里面的任何元素。

soup.p.nextSibling #This is paragraph two. secondBTag = soup.findAlll('b')[1] secondBTag.previousSibling #u'This is paragraph' secondBTag.previousSibling.previousSibling == None #True

next和previous

使用它們可以按照soup處理文檔的次序遍歷整個文檔，而不是它們在剖析樹中看到那種次序。例如<HEAD>?Tag的next是<TITLE>Tag, 而不是<BODY>?Tag。這是因為在原始文檔中，<TITLE> tag 直接在<HEAD>標簽之后。

soup.head.next #u'title' soup.head.nextSibling.name #u'body' soup.head.previous.name #u'html'

Where?next?and?previous?are concerned, a?Tag's?contents?come before its?nextSibling. 通常不會用到這些成員，但有時使用它們能夠非常方便地從剖析樹獲得不易找到的信息。

遍歷一個標簽(Iterating over a Tag)

你可以像遍歷list一樣遍歷一個標簽(Tag)的contents?。這非常有用。類似的，一個Tag的有多少child可以直接使用len(tag)而不必使用len(tag.contents)來獲得。以上面那個文檔中的為例：

for i in soup.body: print i #This is paragraph one. #This is paragraph two. len(soup.body) #2 len(soup.body.contents) #2

使用標簽(tag)名作為成員

像剖析對象或Tag對象的成員一樣使用Tag名可以很方便的操作剖析樹。前面一些例子我們已經(jīng)用到了這種方式。以上述文檔為例，?soup.head獲得文檔第一個<HEAD>標簽：

soup.head #<head><title>Page title</title></head>

通常，調(diào)用mytag.foo獲得的是mytag的第一個child，同時必須是一個<FOO>?標簽。如果在mytag中沒有<FOO>?標簽，mytag.foo返回一個None。你可以使用這中方法快速的讀取剖析樹：

soup.head.title #<title>Page title</title> soup.body.p.b.string #u'one'

你也可以使用這種方法快速的跳到剖析樹的某個特定位置。例如，如果你擔心<TITLE> tags會離奇的在<HEAD> tag之外，你可以使用soup.title去獲得一個HTML文檔的標題(title)，而不必使用soup.head.title:

soup.title.string #u'Page title'

soup.p跳到文檔中的第一個 tag，不論它在哪里。?soup.table.tr.td?跳到文檔總第一個table的第一列第一行。

這些成員實際上是下面first?方法的別名，這里更多介紹。這里提到是因為別名使得一個定位(zoom)一個結(jié)構(gòu)良好剖析樹變得異常容易。

獲得第一個<FOO> 標簽另一種方式是使用.fooTag?而不是?.foo。例如，soup.table.tr.td可以表示為soup.tableTag.trTag.tdTag，甚至為soup.tableTag.tr.tdTag。如果你喜歡更明確的知道表示的意義，或者你在剖析一個標簽與Beautiful Soup的方法或成員有沖突的XML文檔是，使用這種方式非常有用。

from BeautifulSoup import BeautifulStoneSoup xml = '<person name="Bob"><parent rel="mother" name="Alice">' xmlSoup = BeautifulStoneSoup(xml) xmlSoup.person.parent # A Beautiful Soup member #<person name="Bob"><parent rel="mother" name="Alice"></parent></person> xmlSoup.person.parentTag # A tag name #<parent rel="mother" name="Alice"></parent>

如果你要找的標簽名不是有效的Python標識符，(例如hyphenated-name)，你就需要使用first方法了。

搜索剖析樹

Beautiful Soup提供了許多方法用于瀏覽一個剖析樹，收集你指定的Tag和NavigableString。

有幾種方法去定義用于Beautiful Soup的匹配項。我們先用深入解釋最基本的一種搜索方法findAll。和前面一樣，我們使用下面這個文檔說明：

還有, 這里的兩個方法(findAll和?find)僅對Tag對象以及頂層剖析對象有效，但?NavigableString不可用。這兩個方法在Searching 剖析樹內(nèi)部同樣可用。

The basic find method:?findAll(name,?attrs,?recursive,?text,?limit,?**kwargs)

方法findAll?從給定的點開始遍歷整個樹，并找到滿足給定條件所有Tag以及NavigableString。?findall函數(shù)原型定義如下：

findAll(name=None, attrs={}, recursive=True, text=None, limit=None, **kwargs)

這些參數(shù)會反復(fù)的在這個文檔中出現(xiàn)。其中最重要的是name參數(shù) 和keywords參數(shù)(譯注：就是**kwargs參數(shù))。

參數(shù)name?匹配tags的名字，獲得相應(yīng)的結(jié)果集。有幾種方法去匹配name，在這個文檔中會一再的用到。

?
最簡單用法是僅僅給定一個tag name值。下面的代碼尋找文檔中所有的 ?Tag:
soup.findAll('b') #[one, two]
你可以傳一個正則表達式。下面的代碼尋找所有以b開頭的標簽:
import re tagsStartingWithB = soup.findAll(re.compile('^b')) [tag.name for tag in tagsStartingWithB] #[u'body', u'b', u'b']
你可以傳一個list或dictionary。下面兩個調(diào)用是查找所有的<TITLE>和標簽。他們獲得結(jié)果一樣，但是后一種方法更快一些:
soup.findAll(['title', 'p']) #[<title>Page title</title>, # This is paragraph one., # This is paragraph two.] soup.findAll({'title' : True, 'p' : True}) #[<title>Page title</title>, # This is paragraph one., # This is paragraph two.]
你可以傳一個True值,這樣可以匹配每個tag的name：也就是匹配每個tag。
allTags = soup.findAll(True) [tag.name for tag in allTags] [u'html', u'head', u'title', u'body', u'p', u'b', u'p', u'b']
這看起來不是很有用，但是當你限定屬性(attribute)值時候，使用True就很有用了。
你可以傳callable對象，就是一個使用Tag對象作為它唯一的參數(shù)，并返回布爾值的對象。?findAll使用的每個作為參數(shù)的Tag對象都會傳遞給這個callable對象，并且如果調(diào)用返回True，則這個tag便是匹配的。

下面是查找兩個并僅兩個屬性的標簽(tags)：
soup.findAll(lambda tag: len(tag.attrs) == 2) #[This is paragraph one., # This is paragraph two.]
下面是尋找單個字符為標簽名并且沒有屬性的標簽：
soup.findAll(lambda tag: len(tag.name) == 1 and not tag.attrs) #[one, two]
keyword參數(shù)用于篩選tag的屬性。下面這個例子是查找擁有屬性align且值為center的所有標簽：
soup.findAll(align="center") #[This is paragraph one.]
如同name參數(shù)，你也可以使用不同的keyword參數(shù)對象，從而更加靈活的指定屬性值的匹配條件。你可以向上面那樣傳遞一個字符串，來匹配屬性的值。你也可以傳遞一個正則表達式，一個列表(list)，一個哈希表(hash)，特殊值True或None，或者一個可調(diào)用的以屬性值為參數(shù)的對象(注意：這個值可能為None)。一些例子：
soup.findAll(id=re.compile("para$")) #[This is paragraph one., # This is paragraph two.] soup.findAll(align=["center", "blah"]) #[This is paragraph one., # This is paragraph two.] soup.findAll(align=lambda(value): value and len(value) < 5) #[This is paragraph two.]
特殊值True和None更讓人感興趣。?True匹配給定屬性為任意值的標簽，None匹配那些給定的屬性值為空的標簽。一些例子如下：
soup.findAll(align=True) #[This is paragraph one., # This is paragraph two.] [tag.name for tag in soup.findAll(align=None)] #[u'html', u'head', u'title', u'body', u'b', u'b']
如果你需要在標簽的屬性上添加更加復(fù)雜或相互關(guān)聯(lián)的(interlocking)匹配值，?如同上面一樣，以callable對象的傳遞參數(shù)來處理Tag對象。

在這里你也許注意到一個問題。?如果你有一個文檔，它有一個標簽定義了一個name屬性,會怎么樣？你不能使用name為keyword參數(shù)，因為Beautiful Soup 已經(jīng)定義了一個name參數(shù)使用。你也不能用一個Python的保留字例如for作為關(guān)鍵字參數(shù)。

Beautiful Soup提供了一個特殊的參數(shù)attrs，你可以使用它來應(yīng)付這些情況。?attrs是一個字典，用起來就和keyword參數(shù)一樣：
soup.findAll(id=re.compile("para$")) #[This is paragraph one., # This is paragraph two.] soup.findAll(attrs={'id' : re.compile("para$")}) #[This is paragraph one., # This is paragraph two.]
你可以使用attrs去匹配那些名字為Python保留字的屬性，例如class,?for, 以及import；或者那些不是keyword參數(shù)但是名字為Beautiful Soup搜索方法使用的參數(shù)名的屬性，例如name,?recursive,?limit,?text, 以及attrs本身。
from BeautifulSoup import BeautifulStoneSoup xml = '<person name="Bob"><parent rel="mother" name="Alice">' xmlSoup = BeautifulStoneSoup(xml) xmlSoup.findAll(name="Alice") #[] xmlSoup.findAll(attrs={"name" : "Alice"}) #[parent rel="mother" name="Alice"></parent>]
使用CSS類查找

對于CSS類attrs參數(shù)更加方便。例如class不僅是一個CSS屬性，也是Python的保留字。

你可以使用soup.find("tagName", { "class" : "cssClass" })搜索CSS class，但是由于有很多這樣的操作，你也可以只傳遞一個字符串給attrs。這個字符串默認處理為CSS的class的參數(shù)值。
from BeautifulSoup import BeautifulSoup soup = BeautifulSoup("""Bob's Bold Barbeque Sauce now available in Hickory and Lime</a>""") soup.find("b", { "class" : "lime" }) #Lime soup.find("b", "hickory") #Hickory
text?是一個用于搜索NavigableString對象的參數(shù)。它的值可以是字符串，一個正則表達式，一個list或dictionary，True或None，一個以NavigableString為參數(shù)的可調(diào)用對象：
soup.findAll(text="one") #[u'one'] soup.findAll(text=u'one') #[u'one'] soup.findAll(text=["one", "two"]) #[u'one', u'two'] soup.findAll(text=re.compile("paragraph")) #[u'This is paragraph ', u'This is paragraph '] soup.findAll(text=True) #[u'Page title', u'This is paragraph ', u'one', u'.', u'This is paragraph ', # u'two', u'.'] soup.findAll(text=lambda(x): len(x) < 12) #[u'Page title', u'one', u'.', u'two', u'.']
如果你使用text，任何指定給name?以及keyword參數(shù)的值都會被忽略。
recursive?是一個布爾參數(shù)(默認為True)，用于指定Beautiful Soup遍歷整個剖析樹，還是只查找當前的子標簽或者剖析對象。下面是這兩種方法的區(qū)別：
[tag.name for tag in soup.html.findAll()] #[u'head', u'title', u'body', u'p', u'b', u'p', u'b'] [tag.name for tag in soup.html.findAll(recursive=False)] #[u'head', u'body']
當recursive為false，只有當前的子標簽<HTML>會被搜索。如果你需要搜索樹，使用這種方法可以節(jié)省一些時間。
設(shè)置limit?參數(shù)可以讓Beautiful Soup 在找到特定個數(shù)的匹配時停止搜索。文檔中如果有上千個表格，但你只需要前四個，傳值4到limit可以讓你節(jié)省很多時間。默認是沒有限制(limit沒有指定值).
soup.findAll('p', limit=1) #[This is paragraph one.] soup.findAll('p', limit=100) #[This is paragraph one., # This is paragraph two.]

像findall一樣調(diào)用tag

一個小捷徑。如果你像函數(shù)一樣調(diào)用剖析對象或者Tag對象，這樣你調(diào)用所用參數(shù)都會傳遞給findall的參數(shù)，就和調(diào)用findall一樣。就上面那個文檔為例：

soup(text=lambda(x): len(x) < 12) #[u'Page title', u'one', u'.', u'two', u'.'] soup.body('p', limit=1) #[This is paragraph one.]

find(name,?attrs,?recursive,?text,?**kwargs)

好了，我們現(xiàn)在看看其他的搜索方法。他們都是有和?findAll?幾乎一樣的參數(shù)。

find方法是最接近findAll的函數(shù)，只是它并不會獲得所有的匹配對象，它僅僅返回找到第一個可匹配對象。也就是說，它相當于limit參數(shù)為1的結(jié)果集。以上面的?文檔為例:

soup.findAll('p', limit=1) #[This is paragraph one.] soup.find('p', limit=1) #This is paragraph one. soup.find('nosuchtag', limit=1) == None #True

通常，當你看到一個搜索方法的名字由復(fù)數(shù)構(gòu)成 (如findAll和findNextSiblings)時，這個方法就會存在limit參數(shù)，并返回一個list的結(jié)果。但你看到的方法不是復(fù)數(shù)形式(如find和findNextSibling)時，你就可以知道這函數(shù)沒有l(wèi)imit參數(shù)且返回值是單一的結(jié)果。

first哪里去了?

早期的Beautiful Soup 版本有一些first，fetch以及fetchPrevious方法。這些方法還在，但是已經(jīng)被棄用了，也許不久就不在存在了。因為這些名字有些令人迷惑。新的名字更加有意義：前面提到了，復(fù)數(shù)名稱的方法名，比如含有All的方法名，它將返回一個多對象。否則，它只會返回單個對象。

Searching Within the Parse Tree

上面說明的方法findAll及find，都是從剖析樹的某一點開始并一直往下。他們反復(fù)的遍歷對象的contents直到最低點。

也就是說你不能在?NavigableString對象上使用這些方法，因為NavigableString沒有contents：它們是剖析樹的葉子。

[這段翻譯的不太準確]但是向下搜索不是唯一的遍歷剖析樹的方法。在Navigating剖析樹?中，我們可以使用這些方法：parent,?nextSibling等。他們都有2個相應(yīng)的方法：一個類似findAll,一個類似find. 由于NavigableString對象也支持這些方法，你可以像Tag一樣使用這些方法。

為什么這個很有用?因為有些時候，你不能使用findAll或find?從Tag或NavigableString獲得你想要的。例如，下面的HTML文檔：

from BeautifulSoup import BeautifulSoup soup = BeautifulSoup('''<ul> <li>An unrelated list </ul> <h1>Heading</h1> This is the list you want: <ul><li>The data you want</ul>''')

有很多方法去定位到包含特定數(shù)據(jù)的<LI> 標簽。最明顯的方式如下：

soup('li', limit=2)[1] #<li>The data you want</li>

顯然，這樣獲得所需的<LI>標簽并不穩(wěn)定。如果，你只分析一次頁面，這沒什么影響。但是如果你需要在一段時間分析很多次這個頁面，就需要考慮一下這種方法。 If the irrelevant list grows another <LI> tag, you'll get that tag instead of the one you want, and your script will break or give the wrong data.?
因為如果列表發(fā)生變化，你可能就得不到你想要的結(jié)果。

soup('ul', limit=2)[1].li #<li>The data you want</li>

That's is a little better, because it can survive changes to the irrelevant list. But if the document grows another irrelevant list at the top, you'll get the first <LI> tag of that list instead of the one you want. A more reliable way of referring to the ul tag you want would better reflect that tag's place in the structure of the document.?
這有一點好處，因為那些不相干的列表的變更生效了。但是如果文檔增長的不相干的列表在頂部，你會獲得第一個<LI>標簽而不是你想要的標簽。一個更可靠的方式是去引用對應(yīng)的ul標簽，這樣可以更好的處理文檔的結(jié)構(gòu)。

在HTML里面，你也許認為你想要的list是<H1>標簽下的<UL>標簽。問題是那個標簽不是在<H1>下，它只是在它后面。獲得<H1>標簽很容易，但是獲得 <UL>卻沒法使用first和fetch，因為這些方法只是搜索<H1>標簽的contents。你需要使用next或nextSibling來獲得<UL>標簽。

s = soup.h1 while getattr(s, 'name', None) != 'ul': s = s.nextSibling s.li #<li>The data you want</li>

或者，你覺得這樣也許會比較穩(wěn)定：

s = soup.find(text='Heading') while getattr(s, 'name', None) != 'ul': s = s.next s.li #<li>The data you want</li>

但是還有很多困難需要你去克服。這里會介紹一下非常有用的方法。你可以在你需要的使用它們寫一些遍歷成員的方法。它們以某種方式遍歷樹，并跟蹤那些滿足條件的Tag?和NavigableString對象。代替上面那個例子的第一的循環(huán)的代碼，你可以這樣寫：

soup.h1.findNextSibling('ul').li #<li>The data you want</li>

第二循環(huán)，你可以這樣寫：

soup.find(text='Heading').findNext('ul').li #<li>The data you want</li>

這些循環(huán)代替調(diào)用findNextString和findNext。本節(jié)剩下的內(nèi)容是這種類型所用方法的參考。同時，對于遍歷總是有兩種方法：一個是返回list的findAll，一個是返回單一量的find。

下面，我們再舉一個例子來說明：

findNextSiblings(name,?attrs,?text,?limit,?kwargs)?and?findNextSibling(name,?attrs,?text,?kwargs)

這兩個方法以nextSibling的成員為依據(jù)，獲得滿足條件的Tag或NavigableText對象。以上面的文檔為例:

paraText = soup.find(text='This is paragraph ') paraText.findNextSiblings('b') #[one] paraText.findNextSibling(text = lambda(text): len(text) == 1) #u'.'

findPreviousSiblings(name,?attrs,?text,?limit,?kwargs)?and?findPreviousSibling(name,?attrs,?text,?kwargs)

這兩個方法以previousSibling成員為依據(jù)，獲得滿足條件的Tag和?NavigableText對象。以上面的文檔為例:

paraText = soup.find(text='.') paraText.findPreviousSiblings('b') #[one] paraText.findPreviousSibling(text = True) #u'This is paragraph '

findAllNext(name,?attrs,?text,?limit,?kwargs)?and?findNext(name,?attrs,?text,?kwargs)

這兩個方法以next的成員為依據(jù)，獲得滿足條件的Tag和NavigableText對象。以上面的文檔為例:

pTag = soup.find('p') pTag.findAllNext(text=True) #[u'This is paragraph ', u'one', u'.', u'This is paragraph ', u'two', u'.'] pTag.findNext('p') #This is paragraph two. pTag.findNext('b') #one

findAllPrevious(name,?attrs,?text,?limit,?kwargs)?and?findPrevious(name,?attrs,?text,?kwargs)

這兩方法以previous的成員依據(jù)，獲得滿足條件的Tag和NavigableText對象。以上面的文檔為例:

lastPTag = soup('p')[-1] lastPTag.findAllPrevious(text=True) #[u'.', u'one', u'This is paragraph ', u'Page title'] #Note the reverse order! lastPTag.findPrevious('p') #This is paragraph one. lastPTag.findPrevious('b') #one

findParents(name,?attrs,?limit,?kwargs)?and?findParent(name,?attrs,?kwargs)

這兩個方法以parent成員為依據(jù)，獲得滿足條件的Tag和NavigableText對象。他們沒有text參數(shù),因為這里的對象的parent不會有NavigableString。以上面的文檔為例:

bTag = soup.find('b') [tag.name for tag in bTag.findParents()] #[u'p', u'body', u'html', '[document]'] #NOTE: "u'[document]'" means that that the parser object itself matched. bTag.findParent('body').name #u'body'

修改剖析樹

現(xiàn)在你已經(jīng)知道如何在剖析樹中尋找東西了。但也許你想對它做些修改并輸出出來。你可以僅僅將一個元素從其父母的contents中分離，但是文檔的其他部分仍然擁有對這個元素的引用。Beautiful Soup 提供了幾種方法幫助你修改剖析樹并保持其內(nèi)部的一致性。

修改屬性值

你可以使用字典賦值來修改Tag對象的屬性值。

from BeautifulSoup import BeautifulSoup soup = BeautifulSoup("Argh!") print soup #Argh! b = soup.b b['id'] = 10 print soup #Argh! b['id'] = "ten" print soup #Argh! b['id'] = 'one "million"' print soup #Argh!

你也可以刪除一個屬性值，然后添加一個新的屬性:

del(b['id']) print soup #Argh! b['class'] = "extra bold and brassy!" print soup #Argh!

刪除元素

要是你引用了一個元素，你可以使用extract將它從樹中抽離。下面是將所有的注釋從文檔中移除的代碼：

from BeautifulSoup import BeautifulSoup, Comment soup = BeautifulSoup("""1 <a>23""") comments = soup.findAll(text=lambda text:isinstance(text, Comment)) [comment.extract() for comment in comments] print soup #1 #<a>23</a>

這段代碼是從文檔中移除一個子樹:

from BeautifulSoup import BeautifulSoup soup = BeautifulSoup("<a1></a1><a>Amazing content<c><d></a><a2></a2>") soup.a1.nextSibling #<a>Amazing content<c><d></d></c></a> soup.a2.previousSibling #<a>Amazing content<c><d></d></c></a> subtree = soup.a subtree.extract() print soup #<a1></a1><a2></a2> soup.a1.nextSibling #<a2></a2> soup.a2.previousSibling #<a1></a1>

extract方法將一個剖析樹分離為兩個不連貫的樹。naviation的成員也因此變得看起來好像這兩個樹從來不是一起的。

soup.a1.nextSibling #<a2></a2> soup.a2.previousSibling #<a1></a1> subtree.previousSibling == None #True subtree.parent == None #True

使用一個元素替換另一個元素

replaceWith方法抽出一個頁面元素并將其替換為一個不同的元素。新元素可以為一個Tag（它可能包含一個剖析樹）或者NavigableString。如果你傳一個字符串到replaceWith, 它會變?yōu)镹avigableString。這個Navigation成員會完全融入到這個剖析樹中，就像它本來就存在一樣。

下面是一個簡單的例子：

The new element can be a?Tag?(possibly with a whole parse tree beneath it) or a?NavigableString. If you pass a plain old string into?replaceWith, it gets turned into a?NavigableString. The navigation members are changed as though the document had been parsed that way in the first place.

Here's a simple example:

from BeautifulSoup import BeautifulSoup soup = BeautifulSoup("Argh!") soup.find(text="Argh!").replaceWith("Hooray!") print soup #Hooray! newText = soup.find(text="Hooray!") newText.previous #Hooray! newText.previous.next #u'Hooray!' newText.parent #Hooray! soup.b.contents #[u'Hooray!']

這里有一個更復(fù)雜點的，相互替換標簽(tag)的例子：

from BeautifulSoup import BeautifulSoup, Tag soup = BeautifulSoup("Argh!<a>Foo</a>Blah!") tag = Tag(soup, "newTag", [("id", 1)]) tag.insert(0, "Hooray!") soup.a.replaceWith(tag) print soup #Argh!<newTag id="1">Hooray!</newTag>Blah!

You can even rip out an element from one part of the document and stick it in another part:?
你也可以將一個元素抽出然后插入到文檔的其他地方：

from BeautifulSoup import BeautifulSoup text = "<html>There's no business like show business</html>" soup = BeautifulSoup(text) no, show = soup.findAll('b') show.replaceWith(no) print soup #<html>There's business like no business</html>

添加一個全新的元素

The?Tag?class and the parser classes support a method called?insert. It works just like a Python list's?insert?method: it takes an index to the tag's?contents?member, and sticks a new element in that slot.?
標簽類和剖析類有一個insert方法，它就像Python列表的insert方法：它使用索引來定位標簽的contents成員，然后在那個位置插入一個新的元素。

This was demonstrated in the previous section, when we replaced a tag in the document with a brand new tag. You can use?insert?to build up an entire parse tree from scratch:?
在前面那個小結(jié)中，我們在文檔替換一個新的標簽時有用到這個方法。你可以使用insert來重新構(gòu)建整個剖析樹：

from BeautifulSoup import BeautifulSoup, Tag, NavigableString soup = BeautifulSoup() tag1 = Tag(soup, "mytag") tag2 = Tag(soup, "myOtherTag") tag3 = Tag(soup, "myThirdTag") soup.insert(0, tag1) tag1.insert(0, tag2) tag1.insert(1, tag3) print soup #<mytag><myOtherTag></myOtherTag><myThirdTag></myThirdTag></mytag> text = NavigableString("Hello!") tag3.insert(0, text) print soup #<mytag><myOtherTag></myOtherTag><myThirdTag>Hello!</myThirdTag></mytag>

An element can occur in only one place in one parse tree. If you give?insert?an element that's already connected to a soup object, it gets disconnected (with?extract) before it gets connected elsewhere. In this example, I try to insert my?NavigableString?into a second part of the soup, but it doesn't get inserted again. It gets moved:?
一個元素可能只在剖析樹中出現(xiàn)一次。如果你給insert的元素已經(jīng)和soup對象所關(guān)聯(lián)，它會被取消關(guān)聯(lián)(使用extract)在它在被連接別的地方之前。在這個例子中，我試著插入我的NavigableString到 soup對象的第二部分，但是它并沒有被再次插入而是被移動了：

tag2.insert(0, text) print soup #<mytag><myOtherTag>Hello!</myOtherTag><myThirdTag></myThirdTag></mytag>

This happens even if the element previously belonged to a completely different soup object. An element can only have one?parent, one?nextSibling, et cetera, so it can only be in one place at a time.?
即使這個元素屬于一個完全不同的soup對象，還是會這樣。一個元素只可以有一個parent，一個nextSibling等等，也就是說一個地方只能出現(xiàn)一次。

常見問題(Troubleshooting)

This section covers common problems people have with Beautiful Soup. 這一節(jié)是使用BeautifulSoup時會遇到的一些常見問題的解決方法。

為什么Beautiful Soup不能打印我的no-ASCII字符?

If you're getting errors that say:?"'ascii' codec can't encode character 'x' in position y: ordinal not in range(128)", the problem is probably with your Python installation rather than with Beautiful Soup. Try printing out the non-ASCII characters without running them through Beautiful Soup and you should have the same problem. For instance, try running code like this:
如果你遇到這樣的錯誤：?"'ascii' codec can't encode character 'x' in position y: ordinal not in range(128)", 這個錯誤可能是Python的問題而不是BeautifulSoup。?
（譯者注：在已知文檔編碼類型的情況下，可以先將編碼轉(zhuǎn)換為unicode形式，在轉(zhuǎn)換為utf-8編碼，然后才傳遞給BeautifulSoup。例如HTML的內(nèi)容htm是GB2312編碼:?
htm=unicode(htm,'gb2312','ignore').encode('utf-8','ignore')
soup=BeautifulSoup(htm)?
如果不知道編碼的類型,可以使用chardet先檢測一下文檔的編碼類型。chardet需要自己安裝一下，在網(wǎng)上很容下到。）?
試著不用Beautiful Soup而直接打印non-ASCII 字符，你也會遇到一樣的問題。例如，試著運行以下代碼：

latin1word = 'Sacr\xe9 bleu!' unicodeword = unicode(latin1word, 'latin-1') print unicodeword

If this works but Beautiful Soup doesn't, there's probably a bug in Beautiful Soup. However, if this doesn't work, the problem's with your Python setup. Python is playing it safe and not sending non-ASCII characters to your terminal. There are two ways to override this behavior.?
如果它沒有問題而Beautiful Soup不行，這可能是BeautifulSoup的一個bug。但是，如果這個也有問題，就是Python本身的問題。Python為了安全緣故不支持發(fā)送non-ASCII 到終端。有兩種方法可以解決這個限制。

The easy way is to remap standard output to a converter that's not afraid to send ISO-Latin-1 or UTF-8 characters to the terminal.?
最簡單的方式是將標準輸出重新映射到一個轉(zhuǎn)換器，不在意發(fā)送到終端的字符類型是ISO-Latin-1還是UTF-8字符串。

import codecs import sys streamWriter = codecs.lookup('utf-8')[-1] sys.stdout = streamWriter(sys.stdout)

codecs.lookup?returns a number of bound methods and other objects related to a codec. The last one is a?StreamWriter?object capable of wrapping an output stream.?
codecs.lookup返回一些綁定的方法和其它和codec相關(guān)的對象。最后一行是一個封裝了輸出流的StreamWriter對象。

The hard way is to create a?sitecustomize.py?file in your Python installation which sets the default encoding to ISO-Latin-1 or to UTF-8. Then all your Python programs will use that encoding for standard output, without you having to do something for each program. In my installation, I have a?/usr/lib/python/sitecustomize.py?which looks like this:?
稍微困難點的方法是創(chuàng)建一個sitecustomize.py文件在你的Python安裝中，將默認編碼設(shè)置為ISO-Latin-1或UTF-8。這樣你所有的Python程序都會使用這個編碼作為標準輸出，不用在每個程序里再設(shè)置一下。在我的安裝中，我有一個?/usr/lib/python/sitecustomize.py，內(nèi)容如下：

import sys sys.setdefaultencoding("utf-8")

For more information about Python's Unicode support, look at?Unicode for Programmers?or?End to End Unicode Web Applications in Python. Recipes 1.20 and 1.21 in the Python cookbook are also very helpful.?
更多關(guān)于Python的Unicode支持的信息，參考?Unicode for Programmers?or?End to End Unicode Web Applications in Python。Python食譜的給的菜譜1.20和1.21也很有用。

Remember, even if your terminal display is restricted to ASCII, you can still use Beautiful Soup to parse, process, and write documents in UTF-8 and other encodings. You just can't print certain strings with?print.?
但是即使你的終端顯示被限制為ASCII，你也可以使用BeautifulSoup以UTF-8和其它的編碼類型來剖析，處理和修改文檔。只是對于某些字符，你不能使用print來輸出。

Beautiful Soup 弄丟了我給的數(shù)據(jù)!為什么?為什么?????

Beautiful Soup can handle poorly-structured SGML, but sometimes it loses data when it gets stuff that's not SGML at all. This is not nearly as common as poorly-structured markup, but if you're building a web crawler or something you'll surely run into it.?
Beautiful Soup可以處理結(jié)構(gòu)不太規(guī)范的SGML，但是給它的材料非常不規(guī)范，它會丟失數(shù)據(jù)。如果你是在寫一個網(wǎng)絡(luò)爬蟲之類的程序，你肯定會遇到這種，不太常見的結(jié)構(gòu)有問題的文檔。

The only solution is to?sanitize the data ahead of time?with a regular expression. Here are some examples that I and Beautiful Soup users have discovered:?
唯一的解決方法是先使用正則表達式來規(guī)范數(shù)據(jù)。?下面是一些我和一些Beautiful Soup的使用者發(fā)現(xiàn)的例子：

Beautiful Soup treats ill-formed XML definitions as data. However, it loses well-formed XML definitions that don't actually exist:?
Beautiful Soup 將不規(guī)范德XML定義處理為數(shù)據(jù)(data)。然而，它丟失了那些實際上不存在的良好的XML定義：
from BeautifulSoup import BeautifulSoup BeautifulSoup("< ! FOO @=>") #< ! FOO @=> BeautifulSoup("<!FOO>!") #!
If your document starts a declaration and never finishes it, Beautiful Soup assumes the rest of your document is part of the declaration. If the document ends in the middle of the declaration, Beautiful Soup ignores the declaration totally. A couple examples:?
如果你的文檔開始了聲明但卻沒有關(guān)閉，Beautiful Soup假定你的文檔的剩余部分都是這個聲明的一部分。如果文檔在聲明的中間結(jié)束了，Beautiful Soup會忽略這個聲明。如下面這個例子：
from BeautifulSoup import BeautifulSoup BeautifulSoup("foo<!bar") #foo soup = BeautifulSoup("<html>foo<!bar</html>") print soup.prettify() #<html> # foo<!bar</html> #</html>
There are a couple ways to fix this; one is detailed?here.?
有幾種方法來處理這種情況；其中一種在?這里有詳細介紹。

Beautiful Soup also ignores an entity reference that's not finished by the end of the document:?
Beautiful Soup 也會忽略實體引用，如果它沒有在文檔結(jié)束的時候關(guān)閉：
BeautifulSoup("<foo&gt") #<foo
I've never seen this in real web pages, but it's probably out there somewhere. 我從來沒有在實際的網(wǎng)頁中遇到這種情況，但是也許別的地方會出現(xiàn)。
A malformed comment will make Beautiful Soup ignore the rest of the document. This is covered as the example in?Sanitizing Bad Data with Regexps.?
一個畸形的注釋會是Beautiful Soup回來文檔的剩余部分。在使用正則規(guī)范數(shù)據(jù)這里有詳細的例子。

The parse tree built by the?BeautifulSoup?class offends my senses!?
BeautifulSoup類構(gòu)建的剖析樹讓我感到頭痛。

To get your markup parsed differently, check out?
嘗試一下別的剖析方法，試試?其他內(nèi)置的剖析器,或者?自定義一個剖析器.

Beautiful Soup 太慢了！

Beautiful Soup will never run as fast as ElementTree or a custom-built?SGMLParser?subclass. ElementTree is written in C, and?SGMLParser?lets you write your own mini-Beautiful Soup that only does what you want. The point of Beautiful Soup is to save programmer time, not processor time.?
Beautiful Soup 不會像ElementTree或者自定義的SGMLParser子類一樣快。 ElementTree是用C寫的，并且做那些你想要做的事。 Beautiful Soup是用來節(jié)省程序員的時間，而不是處理器的時間。

That said, you can speed up Beautiful Soup quite a lot by?only parsing the parts of the document you need, and you can make unneeded objects get garbage-collected by using?extract.?
但是你可以加快Beautiful Soup通過解析部分的文檔,

高級主題

That does it for the basic usage of Beautiful Soup. But HTML and XML are tricky, and in the real world they're even trickier. So Beautiful Soup keeps some extra tricks of its own up its sleeve.?
那些是對Beautiful Soup的基本用法。但是現(xiàn)實中的HTML和XML是非常棘手的(tricky)，即使他們不是trickier。因此Beautiful Soup也有一些額外的技巧。

產(chǎn)生器

The search methods described above are driven by generator methods. You can use these methods yourself: they're called?nextGenerator,?previousGenerator,?nextSiblingGenerator,previousSiblingGenerator, and?parentGenerator.?Tag?and parser objects also have?childGenerator?and?recursiveChildGenerator?available.?
以上的搜索方法都是由產(chǎn)生器驅(qū)動的。你也可以自己使用這些方法：他們是nextGenerator,?previousGenerator,?nextSiblingGenerator,?previousSiblingGenerator, 和parentGenerator.?Tag和剖析對象可以使用childGenerator和recursiveChildGenerator。

Here's a simple example that strips HTML tags out of a document by iterating over the document and collecting all the strings.?
下面是一個簡單的例子,將遍歷HTML的標簽并將它們從文檔中剝離，搜集所有的字符串:

from BeautifulSoup import BeautifulSoup soup = BeautifulSoup("""<div>You bet <a href="http://www.crummy.com/software/BeautifulSoup/">BeautifulSoup</a> rocks!</div>""") ''.join([e for e in soup.recursiveChildGenerator() if isinstance(e,unicode)]) #u'You bet\nBeautifulSoup\nrocks!'

Here's a more complex example that uses?recursiveChildGenerator?to iterate over the elements of a document, printing each one as it gets it. 這是一個稍微復(fù)雜點的使用recursiveChildGenerator的例子來遍歷文檔中所有元素，并打印它們。

from BeautifulSoup import BeautifulSoup soup = BeautifulSoup("1<a>23") g = soup.recursiveChildGenerator() while True: try: print g.next() except StopIteration: break #1 #<a>23</a> #2 #3 #3

其它內(nèi)置的剖析器

Beautiful Soup comes with three parser classes besides?BeautifulSoup?and?BeautifulStoneSoup:?
除了BeautifulSoup和?BeautifulStoneSoup，還有其它三個Beautiful Soup剖析器：

MinimalSoup?is a subclass of?BeautifulSoup. It knows most facts about HTML like which tags are self-closing, the special behavior of the <SCRIPT> tag, the possibility of an encoding mentioned in a <META> tag, etc. But it has no nesting heuristics at all. So it doesn't know that <LI> tags go underneath <UL> tags and not the other way around. It's useful for parsing pathologically bad markup, and for subclassing.?
MinimalSoup是BeautifulSoup的子類。對于HTML的大部分內(nèi)容都可以處理，例如自關(guān)閉的標簽，特殊的標簽<SCRIPT>,<META>中寫到的可能的編碼類型，等等。但是它沒有內(nèi)置的智能判斷能力。例如它不知道<LI>標簽應(yīng)該在<UL>下，而不是其他方式。對于處理糟糕的標記和用來被繼承還是有用的。
ICantBelieveItsBeautifulSoup?is also a subclass of?BeautifulSoup. It has HTML heuristics that conform more closely to the HTML standard, but ignore how HTML is used in the real world. For instance, it's valid HTML to nest tags, but in the real world a nested tag almost always means that the author forgot to close the first tag. If you run into someone who actually nests tags, then you can use?ICantBelieveItsBeautifulSoup.?ICantBelieveItsBeautifulSoup也是BeautifulSoup的子類。它具有HTML的智能(heuristics)判斷能力，更加符合標準的HTML，但是忽略實際使用的HTML。例如：一個嵌入標簽的HTML是有效的，但是實際上一個嵌入的通常意味著那個HTML的作者忘記了關(guān)閉第一個標簽。如果你運行某些人確實使用嵌入的標簽的HTML，這是你可以是使用ICantBelieveItsBeautifulSoup。
BeautifulSOAP?is a subclass of?BeautifulStoneSoup. It's useful for parsing documents like SOAP messages, which use a subelement when they could just use an attribute of the parent element. Here's an example:?
BeautifulSOAP是BeautifulStoneSoup的子類。對于處理那些類似SOAP消息的文檔，也就是處理那些可以將標簽的子標簽變?yōu)槠鋵傩缘奈臋n很方便。下面是一個例子：
from BeautifulSoup import BeautifulStoneSoup, BeautifulSOAP xml = "<doc><tag>subelement</tag></doc>" print BeautifulStoneSoup(xml) #<doc><tag>subelement</tag></doc> print BeautifulSOAP(xml) <doc tag="subelement"><tag>subelement</tag></doc>
With?BeautifulSOAP?you can access the contents of the <TAG> tag without descending into the tag.?
使用BeautifulSOAP,你可以直接存取<TAG>而不需要再往下解析。

定制剖析器(Parser)

When the built-in parser classes won't do the job, you need to customize. This usually means customizing the lists of nestable and self-closing tags. You can customize the list of self-closing tags by passing a?selfClosingTags?argument into the soup constructor. To customize the lists of nestable tags, though, you'll have to subclass.?
當內(nèi)置的剖析類不能做一些工作時，你需要定制它們。這通常意味著重新定義可內(nèi)嵌的標簽和自關(guān)閉的標簽列表。你可以通過傳遞參數(shù)selfClosingTags?給soup的構(gòu)造器來定制自關(guān)閉的標簽。自定義可以內(nèi)嵌的標簽的列表，你需要子類化。

The most useful classes to subclass are?MinimalSoup?(for HTML) and?BeautifulStoneSoup?(for XML). I'm going to show you how to override?RESET_NESTING_TAGS?and?NESTABLE_TAGS?in a subclass. This is the most complicated part of Beautiful Soup and I'm not going to explain it very well here, but I'll get something written and then I can improve it with feedback.?
非常有用的用來子類的類是MinimalSoup類(針對HTML)和BeautifulStoneSoup(針對XML)。我會說明如何在子類中重寫RESET_NESTING_TAGS和NESTABLE_TAGS。這是Beautiful Soup 中最為復(fù)雜的部分，所以我也不會在這里詳細的解釋，但是我會寫些東西并利用反饋來改進它。

When Beautiful Soup is parsing a document, it keeps a stack of open tags. Whenever it sees a new start tag, it tosses that tag on top of the stack. But before it does, it might close some of the open tags and remove them from the stack. Which tags it closes depends on the qualities of tag it just found, and the qualities of the tags in the stack.?
當Beautiful Soup剖析一個文檔的時候，它會保持一個打開的tag的堆棧。任何時候只要它看到一個新的開始tag，它會將這個tag拖到堆棧的頂端。但在做這步之前，它可能會關(guān)閉某些已經(jīng)打開的標簽并將它們從堆棧中移除。

The best way to explain it is through example. Let's say the stack looks like?['html', 'p', 'b'], and Beautiful Soup encounters a tag. If it just tossed another?'p'?onto the stack, this would imply that the second tag is within the first tag, not to mention the open tag. But that's not the way tags work. You can't stick a tag inside another tag. A tag isn't "nestable" at all.?
我們最好還是通過例子來解釋。我們假定堆棧如同['html','p','b']，并且Beautiful Soup遇到一個標簽。如果它僅僅將另一個'p'拖到堆棧的頂端，這意味著第二個標簽在第一個內(nèi)，而不會影響到打開的。但是這不是應(yīng)該的樣子。你不能插入一個到另一個里面去。標簽不是可內(nèi)嵌的。

So when Beautiful Soup encounters a tag, it closes and pops all the tags up to and including the previously encountered tag of the same type. This is the default behavior, and this is how?BeautifulStoneSoup?treats?every?tag. It's what you get when a tag is not mentioned in either?NESTABLE_TAGS?or?RESET_NESTING_TAGS. It's also what you get when a tag shows up inRESET_NESTING_TAGS?but has no entry in?NESTABLE_TAGS, the way the tag does.?
因此當Beautiful Soup 遇到一個時，它先關(guān)閉并彈出所有的標簽，包括前面遇到的同類型的標簽。這是默認的操作，這也是Beautiful Soup對待每個標簽的方式。當一個標簽不在NESTABLE_TAGS?或RESET_NESTING_TAGS中時，你會遇到的處理方式。這也是當一個標簽在RESET_NESTING_TAGS?中而不在NESTABLE_TAGS中時的處理方式，就像處理一樣。

from BeautifulSoup import BeautifulSoup BeautifulSoup.RESET_NESTING_TAGS['p'] == None #True BeautifulSoup.NESTABLE_TAGS.has_key('p') #False print BeautifulSoup("<html>ParaonePara two") #<html>ParaonePara two</html> # ^---^--The second tag made those two tags get closed

Let's say the stack looks like?['html', 'span', 'b'], and Beautiful Soup encounters a tag. Now, tags can contain other tags without limit, so there's no need to pop up to the previous tag when you encounter one. This is represented by mapping the tag name to an empty list in?NESTABLE_TAGS. This kind of tag should not be mentioned in?RESET_NESTING_TAGS: there are no circumstances when encountering a tag would cause any tags to be popped.?
我們假定堆棧如同['html','span','b']，并且Beautiful Soup 遇到一個標簽。現(xiàn)在，可以無限制包含其他的，因此當再次遇到標簽時沒有必要彈出前面的標簽。這是通過映射標簽名到NESTABLE_TAGS中的一個空列表里。這樣的標簽也需要在RESET_NESTING_TAGS里設(shè)置：當再次遇到是不會再導致任何標簽被彈出并關(guān)閉。

from BeautifulSoup import BeautifulSoup BeautifulSoup.NESTABLE_TAGS['span'] #[] BeautifulSoup.RESET_NESTING_TAGS.has_key('span') #False print BeautifulSoup("<html>SpanoneSpan two") #<html>SpanoneSpan two</html>

Third example: suppose the stack looks like?['ol','li','ul']: that is, we've got an ordered list, the first element of which contains an unordered list. Now suppose Beautiful Soup encounters a <LI> tag. It shouldn't pop up to the first <LI> tag, because this new <LI> tag is part of the unordered sublist. It's okay for an <LI> tag to be inside another <LI> tag, so long as there's a <UL> or <OL> tag in the way.?
第三個例子:假定堆棧如同['ol','li','ul']: 也就是，我們有一個有序的list，且列表的第一個元素包含一個無序的list。現(xiàn)在假設(shè)，Beautiful Soup 遇到一個<LI>標簽。它不會彈出第一個<LI>，因為這個新的<LI>是無序的子list一部分。 <LI>中內(nèi)嵌一個<LI>是可以的，同樣的<UL>和<OL>標簽也可以這樣。

from BeautifulSoup import BeautifulSoup print BeautifulSoup("<ol><li>1<ul><li>A").prettify() #<ol> # <li> # 1 # <ul> # <li> # A # </li> # </ul> # </li> #</ol>

But if there is no intervening <UL> or <OL>, then one <LI> tag can't be underneath another:?
如果<UL>和<OL>沒有被干擾，這時一個<LI>標簽也不能在另一個之下。[bad]

print BeautifulSoup("<ol><li>1<li>A").prettify() #<ol> # <li> # 1 # </li> # <li> # A # </li> #</ol>

We tell Beautiful Soup to treat <LI> tags this way by putting "li" in?RESET_NESTING_TAGS, and by giving "li" a?NESTABLE_TAGS?entry showing list of tags under which it can nest.?
Beautiful Soup這樣對待<LI>是通過將"li"放入RESET_NESTING_TAGS，并給在NESTABLE_TAGS中給"li"一個可以內(nèi)嵌接口。

BeautifulSoup.RESET_NESTING_TAGS.has_key('li') #True BeautifulSoup.NESTABLE_TAGS['li'] #['ul', 'ol']

This is also how we handle the nesting of table tags:?
這也是處理內(nèi)嵌的table標簽的方式：

BeautifulSoup.NESTABLE_TAGS['td'] #['tr'] BeautifulSoup.NESTABLE_TAGS['tr'] #['table', 'tbody', 'tfoot', 'thead'] BeautifulSoup.NESTABLE_TAGS['tbody'] #['table'] BeautifulSoup.NESTABLE_TAGS['thead'] #['table'] BeautifulSoup.NESTABLE_TAGS['tfoot'] #['table'] BeautifulSoup.NESTABLE_TAGS['table'] #[]

That is: <TD> tags can be nested within <TR> tags. <TR> tags can be nested within <TABLE>, <TBODY>, <TFOOT>, and <THEAD> tags. <TBODY>, <TFOOT>, and <THEAD> tags can be nested in <TABLE> tags, and <TABLE> tags can be nested in other <TABLE> tags. If you know about HTML tables, these rules should already make sense to you.?
也就是<TD>標簽可以嵌入到<TR>中。 <TR>可以被嵌入到<TABLE>, <TBODY>, <TFOOT>, 以及 <THEAD> 中。 <TBODY>,<TFOOT>, and <THEAD>標簽可以嵌入到 <TABLE> 標簽中, 而 <TABLE> 嵌入到其它的<TABLE> 標簽中. 如果你對HTML有所了解，這些規(guī)則對你而言應(yīng)該很熟悉。

One more example. Say the stack looks like?['html', 'p', 'table']?and Beautiful Soup encounters a tag.?
再舉一個例子，假設(shè)堆棧如同['html','p','table']，并且Beautiful Soup遇到一個標簽。

At first glance, this looks just like the example where the stack is?['html', 'p', 'b']?and Beautiful Soup encounters a tag. In that example, we closed the and tags, because you can't have one paragraph inside another. 首先，這看起來像前面的同樣是Beautiful Soup遇到了堆棧['html','p','b']。在那個例子中，我們關(guān)閉了和標簽，因為你不能在一個段落里內(nèi)嵌另一個段落。

Except... you?can?have a paragraph that contains a table, and then the table contains a paragraph. So the right thing to do is to not close any of these tags. Beautiful Soup does the right thing: 除非，你的段落里包含了一個table，然后這table包含了一個段落。因此，這種情況下正確的處理是不關(guān)閉任何標簽。Beautiful Soup就是這樣做的：

from BeautifulSoup import BeautifulSoup print BeautifulSoup("Para 1Para 2") # # Para 1 # # # # # Para 2 # print BeautifulSoup("Para 1<table>Para 2").prettify() # # Para 1 # <table> # # Para 2 # # </table> #

What's the difference? The difference is that <TABLE> is in?RESET_NESTING_TAGS?and is not. A tag that's in?RESET_NESTING_TAGS?doesn't get popped off the stack as easily as a tag that's not.?
有什么不同？不同是<TABLE>標簽在RESET_NESTING_TAGS中，而不在。一個在RESET_NESTING_TAGS中標簽不會像不在其里面的標簽?zāi)菢?#xff0c;會是堆棧中標簽被彈出。

Okay, hopefully you get the idea. Here's the?NESTABLE_TAGS?for the?BeautifulSoup?class. Correlate this with what you know about HTML, and you should be able to create your own?NESTABLE_TAGS?for bizarre HTML documents that don't follow the normal rules, and for other XML dialects that have different nesting rules.?
好了，希望你明白了(我被弄有點暈，有些地方翻譯的不清，還請見諒)。?NESTABLE_TAGS用于BeautifulSoup類。依據(jù)你所知道的HTML，你可以創(chuàng)建你自己NESTABLE_TAGS來處理那些不遵循標準規(guī)則的HTML文檔。以及那些使用不同嵌入規(guī)則XML的方言。

from BeautifulSoup import BeautifulSoup nestKeys = BeautifulSoup.NESTABLE_TAGS.keys() nestKeys.sort() for key in nestKeys: print "%s: %s" % (key, BeautifulSoup.NESTABLE_TAGS[key]) #bdo: [] #blockquote: [] #center: [] #dd: ['dl'] #del: [] #div: [] #dl: [] #dt: ['dl'] #fieldset: [] #font: [] #ins: [] #li: ['ul', 'ol'] #object: [] #ol: [] #q: [] #span: [] #sub: [] #sup: [] #table: [] #tbody: ['table'] #td: ['tr'] #tfoot: ['table'] #th: ['tr'] #thead: ['table'] #tr: ['table', 'tbody', 'tfoot', 'thead'] #ul: []

And here's?BeautifulSoup's?RESET_NESTING_TAGS. Only the keys are important:?RESET_NESTING_TAGS?is actually a list, put into the form of a dictionary for quick random access.?
這是BeautifulSoup的RESET_NESTING_TAGS。只有鍵(keys)是重要的：?RESET_NESTING_TAGS實際是一個list，以字典的形式可以快速隨機存取。

from BeautifulSoup import BeautifulSoup resetKeys = BeautifulSoup.RESET_NESTING_TAGS.keys() resetKeys.sort() resetKeys #['address', 'blockquote', 'dd', 'del', 'div', 'dl', 'dt', 'fieldset', # 'form', 'ins', 'li', 'noscript', 'ol', 'p', 'pre', 'table', 'tbody', # 'td', 'tfoot', 'th', 'thead', 'tr', 'ul']

Since you're subclassing anyway, you might as well override?SELF_CLOSING_TAGS?while you're at it. It's a dictionary that maps self-closing tag names to any values at all (likeRESET_NESTING_TAGS, it's actually a list in the form of a dictionary). Then you won't have to pass that list in to the constructor (as?selfClosingTags) every time you instantiate your subclass.?
因為無論如何都有使用繼承，你最好還是在需要的時候重寫SELF_CLOSING_TAGS。這是一個映射自關(guān)閉標簽名的字典(如同RESET_NESTING_TAGS,它實際是字典形式的list)。這樣每次實例化你的子類時，你就不用傳list給構(gòu)造器(如selfClosingTags)。

實體轉(zhuǎn)換

When you parse a document, you can convert HTML or XML entity references to the corresponding Unicode characters. This code converts the HTML entity "é" to the Unicode character LATIN SMALL LETTER E WITH ACUTE, and the numeric entity "e" to the Unicode character LATIN SMALL LETTER E.?
當你剖析一個文檔是，你可以轉(zhuǎn)換HTML或者XML實體引用到可表達Unicode的字符。這個代碼轉(zhuǎn)換HTML實體"é"到Unicode字符 LATIN SMALL LETTER E WITH ACUTE,以及將數(shù)量實體"e"轉(zhuǎn)換到Unicode字符LATIN SMALL LETTER E.

from BeautifulSoup import BeautifulStoneSoup BeautifulStoneSoup("Sacré bleu!", convertEntities=BeautifulStoneSoup.HTML_ENTITIES).contents[0] #u'Sacr\xe9 bleu!'

That's if you use?HTML_ENTITIES?(which is just the string "html"). If you use?XML_ENTITIES?(or the string "xml"), then only numeric entities and the five XML entities (""", "'", ">", "<", and "&") get converted. If you use?ALL_ENTITIES?(or the list?["xml", "html"]), then both kinds of entities will be converted. This last one is neccessary because ' is an XML entity but not an HTML entity.?
這是針對使用HTML_ENTITIES(也就是字符串"html")。如果你使用XML_ENTITIES(或字符串"xml")，這是只有數(shù)字實體和五個XML實體((""","'", ">", "<", 和 "&") 會被轉(zhuǎn)換。如果你使用ALL_ENTITIES(或者列表["xml","html"])，兩種實體都會被轉(zhuǎn)換。最后一種方式是必要的，因為'是一個XML的實體而不是HTML的。

BeautifulStoneSoup("Sacré bleu!", convertEntities=BeautifulStoneSoup.XML_ENTITIES) #Sacré bleu! from BeautifulSoup import BeautifulStoneSoup BeautifulStoneSoup("Il a dit, <<Sacré bleu!>>", convertEntities=BeautifulStoneSoup.XML_ENTITIES) #Il a dit, <<Sacré bleu!>>

If you tell Beautiful Soup to convert XML or HTML entities into the corresponding Unicode characters, then Windows-1252 characters (like Microsoft smart quotes) also get transformed into Unicode characters. This happens even if you told Beautiful Soup to convert those characters to entities.?
如果你指定Beautiful Soup轉(zhuǎn)換XML或HTML實體到可通信的Unicode字符時，Windows-1252(微軟的smart quotes)也會被轉(zhuǎn)換為Unicode字符。即使你指定Beautiful Soup轉(zhuǎn)換這些字符到實體是，也還是這樣。

from BeautifulSoup import BeautifulStoneSoup smartQuotesAndEntities = "Il a dit, \x8BSacré bleu!\x9b" BeautifulStoneSoup(smartQuotesAndEntities, smartQuotesTo="html").contents[0] #u'Il a dit, &lsaquo;Sacré bleu!&rsaquo;' BeautifulStoneSoup(smartQuotesAndEntities, convertEntities="html", smartQuotesTo="html").contents[0] #u'Il a dit, \u2039Sacr\xe9 bleu!\u203a' BeautifulStoneSoup(smartQuotesAndEntities, convertEntities="xml", smartQuotesTo="xml").contents[0] #u'Il a dit, \u2039Sacré bleu!\u203a'

It doesn't make sense to create new HTML/XML entities while you're busy turning all the existing entities into Unicode characters.?
將所有存在的實體轉(zhuǎn)換為Unicode時，不會影響創(chuàng)建新的HTML/XML實體。

使用正則式處理糟糕的數(shù)據(jù)

Beautiful Soup does pretty well at handling bad markup when "bad markup" means tags in the wrong places. But sometimes the markup is just malformed, and the underlying parser can't handle it. So Beautiful Soup runs regular expressions against an input document before trying to parse it.?
對于那些在錯誤的位置的"壞標簽"，Beautiful Soup處理的還不錯。但有時有些非常不正常的標簽，底層的剖析器也不能處理。這時Beautiful Soup會在剖析之前運用正則表達式來處理輸入的文檔。

By default, Beautiful Soup uses regular expressions and replacement functions to do search-and-replace on input documents. It finds self-closing tags that look like , and changes them to look like . It finds declarations that have extraneous whitespace, like <! --Comment-->, and removes the whitespace: .?
默認情況下，Beautiful Soup使用正則式和替換函數(shù)對輸入文檔進行搜索替換操作。它可以發(fā)現(xiàn)自關(guān)閉的標簽如 ，轉(zhuǎn)換它們?nèi)缤?lt;BR />（譯注：多加了一個空格）。它可以找到有多余空格的聲明，如<! --Comment-->，移除空格:.

If you have bad markup that needs fixing in some other way, you can pass your own list of?(regular expression, replacement function)?tuples into the soup constructor, as the?markupMassage?argument.?
如果你的壞標簽需要以其他的方式修復(fù)，你也可以傳遞你自己的以(regular expression, replacement function)?元組的list到soup對象構(gòu)造器，作為markupMassage參數(shù)。

Let's take an example: a page that has a malformed comment. The underlying SGML parser can't cope with this, and ignores the comment and everything afterwards: 我們舉個例子：有一個頁面的注釋很糟糕。底層的SGML不能解析它，并會忽略注釋以及它后面的所有內(nèi)容。

from BeautifulSoup import BeautifulSoup badString = "Foo<!-This comment is malformed.-->Bar Baz" BeautifulSoup(badString) #Foo

Let's fix it up with a regular expression and a function:?
讓我們使用正則式和一個函數(shù)來解決這個問題：

import re myMassage = [(re.compile('<!-([^-])'), lambda match: 'Bar

Oops, we're still missing the tag. Our?markupMassage?overrides the parser's default massage, so the default search-and-replace functions don't get run. The parser makes it past the comment, but it dies at the malformed self-closing tag. Let's add our new massage function to the default list, so we run all the functions.?
哦呃呃，我們還是漏掉了 標簽。我們的markupMassage?重載了剖析默認的message，因此默認的搜索替換函數(shù)不會運行。剖析器讓它來處理注釋，但是它在壞的自關(guān)閉標簽?zāi)抢锿Ｖ沽?。讓我加一些新的message函數(shù)到默認的list中去，并讓這些函數(shù)都運行起來。

import copy myNewMassage = copy.copy(BeautifulSoup.MARKUP_MASSAGE) myNewMassage.extend(myMassage) BeautifulSoup(badString, markupMassage=myNewMassage) #FooBar Baz

Now we've got it all.?
這樣我們就搞定了。

If you know for a fact that your markup doesn't need any regular expressions run on it, you can get a faster startup time by passing in?False?for?markupMassage.?
如果你已經(jīng)知道你的標簽不需要任何正則式，你可以通過傳遞一個False給markupMassage.

玩玩SoupStrainer

Recall that all the search methods take more or less?the same arguments. Behind the scenes, your arguments to a search method get transformed into a?SoupStrainer?object. If you call one of the methods that returns a list (like?findAll), the?SoupStrainer?object is made available as the?source?property of the resulting list.?
回憶起所有的搜索方法都是或多或少使用了一些一樣的參數(shù)。在后臺，你傳遞給搜索函數(shù)的參數(shù)都會傳給SoupStrainer對象。如果你所使用的函數(shù)返回一個list(如findAll)，那是SoupStrainer對象使結(jié)果列表的source屬性變的可用。

from BeautifulSoup import BeautifulStoneSoup xml = '<person name="Bob"><parent rel="mother" name="Alice">' xmlSoup = BeautifulStoneSoup(xml) results = xmlSoup.findAll(rel='mother') results.source #<BeautifulSoup.SoupStrainer instance at 0xb7e0158c> str(results.source) #"None|{'rel': 'mother'}"

The?SoupStrainer?constructor takes most of the same arguments as?find:?name,?attrs,?text, and?**kwargs. You can pass in a?SoupStrainer?as the?name?argument to any search method:?
SoupStrainer的構(gòu)造器幾乎使用和find一樣的參數(shù)：?name,?attrs,?text, 和**kwargs. 你可以在一個SoupStrainer中傳遞和其他搜索方法一樣的name參數(shù)：

xmlSoup.findAll(results.source) == results #True customStrainer = BeautifulSoup.SoupStrainer(rel='mother') xmlSoup.findAll(customStrainer) == results # True

Yeah, who cares, right? You can carry around a method call's arguments in many other ways. But another thing you can do with?SoupStrainer?is pass it into the soup constructor to restrict the parts of the document that actually get parsed. That brings us to the next section:?
耶,誰會在意，對不對？你可以把一個方法的參數(shù)用在很多其他地方。還有一件你可以用SoupStrainer做的事是，將它傳遞給soup的構(gòu)建器，來部分的解析文檔。下一節(jié)，我們就談這個。

通過剖析部分文檔來提升效率

Beautiful Soup turns every element of a document into a Python object and connects it to a bunch of other Python objects. If you only need a subset of the document, this is really slow. But you can pass in a?SoupStrainer?as the?parseOnlyThese?argument to the soup constructor. Beautiful Soup checks each element against the?SoupStrainer, and only if it matches is the element turned into a?Tag?or?NavigableText, and added to the tree.?
Beautiful Soup 將一個文檔的每個元素都轉(zhuǎn)換為Python對象并將文檔轉(zhuǎn)換為一些Python對象的集合。如果你只需要這個文檔的子集，全部轉(zhuǎn)換確實非常慢。但是你可以傳遞SoupStrainer作為parseOnlyThese參數(shù)的值給 soup的構(gòu)造器。Beautiful Soup檢查每一個元素是否滿足SoupStrainer條件，只有那些滿足條件的元素會轉(zhuǎn)換為Tag標簽或NavigableText，并被添加到剖析樹中。

If an element is added to to the tree, then so are its children—even if they wouldn't have matched the?SoupStrainer?on their own. This lets you parse only the chunks of a document that contain the data you want.?
如果一個元素被加到剖析樹中，那么的子元素即使不滿足SoupStrainer也會被加入到樹中。這可以讓你只剖析文檔中那些你想要的數(shù)據(jù)塊。

Here's a pretty varied document:?
看看下面這個有意思的例子：

doc = '''Bob reports <a href="http://www.bob.com/">success</a> with his plasma breeding <a href="http://www.bob.com/plasma">experiments</a>. Don't get any on us, Bob! Ever hear of annular fusion? The folks at <a href="http://www.boogabooga.net/">BoogaBooga</a> sure seem obsessed with it. Secret project, or WEB MADNESS? You decide!'''

Here are several different ways of parsing the document into soup, depending on which parts you want. All of these are faster and use less memory than parsing the whole document and then using the same?SoupStrainer?to pick out the parts you want.?
有幾種不同的方法可以根據(jù)你的需求來剖析部分文檔.比起剖析全部文檔，他們都更快并占用更少的內(nèi)存，他們都是使用相同的?SoupStrainer來挑選文檔中你想要的部分。

from BeautifulSoup import BeautifulSoup, SoupStrainer import re links = SoupStrainer('a') [tag for tag in BeautifulSoup(doc, parseOnlyThese=links)] #[<a href="http://www.bob.com/">success</a>, # <a href="http://www.bob.com/plasma">experiments</a>, # <a href="http://www.boogabooga.net/">BoogaBooga</a>] linksToBob = SoupStrainer('a', href=re.compile('bob.com/')) [tag for tag in BeautifulSoup(doc, parseOnlyThese=linksToBob)] #[<a href="http://www.bob.com/">success</a>, # <a href="http://www.bob.com/plasma">experiments</a>] mentionsOfBob = SoupStrainer(text=re.compile("Bob")) [text for text in BeautifulSoup(doc, parseOnlyThese=mentionsOfBob)] #[u'Bob reports ', u"Don't get any on\nus, Bob!"] allCaps = SoupStrainer(text=lambda(t):t.upper()==t) [text for text in BeautifulSoup(doc, parseOnlyThese=allCaps)] #[u'. ', u'\n', u'WEB MADNESS?']

There is one major difference between the?SoupStrainer?you pass into a search method and the one you pass into a soup constructor. Recall that the?name?argument can take?a function whose argument is a?Tag?object. You can't do this for a?SoupStrainer's?name, because the?SoupStrainer?is used to decide whether or not a?Tag?object should be created in the first place. You can pass in a function for a?SoupStrainer's?name, but it can't take a?Tag?object: it can only take the tag name and a map of arguments.?
把SoupStrainer傳遞給搜索方法和soup構(gòu)造器有一個很大的不同。回憶一下，name參數(shù)可以使用以Tag對象為參數(shù)的函數(shù)。但是你不能對SoupStrainer的name使用這招，因為SoupStrainer被用于決定一個Tag對象是否可以在第一個地方被創(chuàng)建。你可以傳遞一個函數(shù)給SoupStrainer的name，但是不能是使用Tag對象的函數(shù)：只能使用tag的名字和一個參數(shù)映射。

shortWithNoAttrs = SoupStrainer(lambda name, attrs: \ len(name) == 1 and not attrs) [tag for tag in BeautifulSoup(doc, parseOnlyThese=shortWithNoAttrs)] #[Don't get any on us, Bob!, # WEB MADNESS?]

使用extract改進內(nèi)存使用

When Beautiful Soup parses a document, it loads into memory a large, densely connected data structure. If you just need a string from that data structure, you might think that you can grab the string and leave the rest of it to be garbage collected. Not so. That string is a?NavigableString?object. It's got a?parent?member that points to a?Tag?object, which points to other?Tag?objects, and so on. So long as you hold on to any part of the tree, you're keeping the whole thing in memory.?
但Beautiful Soup剖析一個文檔的時候，它會將整個文檔以一個很大很密集的數(shù)據(jù)結(jié)構(gòu)中載入內(nèi)存。如果你僅僅需要從這個數(shù)據(jù)結(jié)構(gòu)中獲得一個字符串，你可能覺得為了這個字符串而弄了那么一堆要被當垃圾收集的數(shù)據(jù)會很不劃算。而且，那個字符串還是NavigableString對象。也就是要獲得一個指向Tag對象的parent的成員，而這個Tag又會指向其他的Tag對象，等等。因此，你不得不保持一顆剖析樹所有部分，也就是把整個東西放在內(nèi)存里。

The?extract?method breaks those connections. If you call?extract?on the string you need, it gets disconnected from the rest of the parse tree. The rest of the tree can then go out of scope and be garbage collected, while you use the string for something else. If you just need a small part of the tree, you can call?extract?on its top-level?Tag?and let the rest of the tree get garbage collected.?
extrace方法可以破壞這些鏈接。如果你調(diào)用extract來獲得你需要字符串，它將會從樹的其他部分中鏈接中斷開。當你使用這個字符串做什么時,樹的剩下部分可以離開作用域而被垃圾收集器捕獲。如果你即使需要一個樹的一部分，你也可以講extract使用在頂層的Tag上，讓其它部分被垃圾收集器收集。

This works the other way, too. If there's a big chunk of the document you?don't?need, you can call?extract?to rip it out of the tree, then abandon it to be garbage collected while retaining control of the (smaller) tree. 也可以使用extract實現(xiàn)些別的功能。如果文檔中有一大塊不是你需要，你也可以使用extract來將它弄出剖析樹，再把它丟給垃圾收集器同時對(較小的那個)剖析樹的控制。

If you find yourself destroying big chunks of the tree, you might have been able to save time by?not parsing that part of the tree in the first place.?
如果你覺得你正在破壞樹的大塊頭，你應(yīng)該看看?通過剖析部分文檔來提升效率來省省時間。

其它

Applications that use Beautiful Soup

Lots of real-world applications use Beautiful Soup. Here are the publicly visible applications that I know about:?
很多實際的應(yīng)用程序已經(jīng)使用Beautiful Soup。這里是一些我了解的公布的應(yīng)用程序：

Scrape 'N' Feed?is designed to work with Beautiful Soup to build RSS feeds for sites that don't have them.?
htmlatex?uses Beautiful Soup to find LaTeX equations and render them as graphics.
chmtopdf?converts CHM files to PDF format. Who am I to argue with that?
Duncan Gough's?Fotopic backup?uses Beautiful Soup to scrape the Fotopic website.
I?igo Serna's?googlenews.py?uses Beautiful Soup to scrape Google News (it's in the parse_entry and parse_category functions)
The?Weather Office Screen Scraper?uses Beautiful Soup to scrape the Canadian government's weather office site.
News Clues?uses Beautiful Soup to parse RSS feeds.
BlinkFlash?uses Beautiful Soup to automate form submission for an online service.
The?linky?link checker uses Beautiful Soup to find a page's links and images that need checking.
Matt Croydon?got Beautiful Soup 1.x to work on his Nokia Series 60 smartphone.?C.R. Sandeep?wrote a real-time currency converter for the Series 60 using Beautiful Soup, but he won't show us how he did it.
Here's?a short script?from jacobian.org to fix the metadata on music files downloaded from allofmp3.com.
The?Python Community Server?uses Beautiful Soup in its spam detector.

類似的庫

I've found several other parsers for various languages that can handle bad markup, do tree traversal for you, or are otherwise more useful than your average parser.?
我已經(jīng)找了幾個其他的用于不同語言的可以處理爛標記的剖析器。簡單介紹一下，也許對你有所幫助。

I've ported Beautiful Soup to Ruby. The result is?Rubyful Soup.
Hpricot?is giving Rubyful Soup a run for its money.
ElementTree?is a fast Python XML parser with a bad attitude. I love it.
Tag Soup?is an XML/HTML parser written in Java which rewrites bad HTML into parseable HTML.
HtmlPrag?is a Scheme library for parsing bad HTML.
xmltramp?is a nice take on a 'standard' XML/XHTML parser. Like most parsers, it makes you traverse the tree yourself, but it's easy to use.
pullparser?includes a tree-traversal method.
Mike Foord didn't like the way Beautiful Soup can change HTML if you write the tree back out, so he wrote?HTML Scraper. It's basically a version of HTMLParser that can handle bad HTML. It might be obsolete with the release of Beautiful Soup 3.0, though; I'm not sure.
Ka-Ping Yee's?scrape.py?combines page scraping with URL opening.

小結(jié)

That's it! Have fun! I wrote Beautiful Soup to save everybody time. Once you get used to it, you should be able to wrangle data out of poorly-designed websites in just a few minutes. Send me email if you have any comments, run into problems, or want me to know about your project that uses Beautiful Soup.?
就這樣了！玩的開心！我寫的Beautiful Soup是為了幫助每個人節(jié)省時間。一旦你習慣上用它之后，只要幾分鐘就能整好那些有些糟糕的站點?？梢园l(fā)郵件給我，如果你有什么建議，或者遇到什么問題，或者讓我知道你的項目再用Beautiful Soup。

--Leonard

This document (source) is part of Crummy, the webspace of?Leonard Richardson?(contact information). It was last modified on Thursday, February 02 2012, 13:11:38 Nowhere Standard Time and last built on Saturday, July 01 2017, 10:00:01 Nowhere Standard Time.

Crummy is ? 1996-2017 Leonard Richardson. Unless otherwise noted, all text licensed under a?Creative Commons License.

Document tree:

http://www.crummy.com/

software/

BeautifulSoup/

bs3/

documentation.zh.html

Site Search:

轉(zhuǎn)載于:https://www.cnblogs.com/fengshuihuan/p/7103089.html

總結(jié)

以上是生活随笔為你收集整理的Beautifulsoup官方文档的全部內(nèi)容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網(wǎng)站內(nèi)容還不錯，歡迎將生活随笔推薦給好友。

上一篇： es实战-Monitoring原理讲解及
下一篇： JAVA开发运维（CI/CD）