如果爬蟲沒有異常處理，那么爬行中一旦出現(xiàn)錯(cuò)誤，程序?qū)⒈罎⑼Ｖ构ぷ?#xff0c;有異常處理即使出現(xiàn)錯(cuò)誤也能繼續(xù)執(zhí)行下去

?

1.常見狀態(tài)嗎

301：重定向到新的URL，永久性
302：重定向到臨時(shí)URL，非永久性
304：請(qǐng)求的資源未更新
400：非法請(qǐng)求
401：請(qǐng)求未經(jīng)授權(quán)
403：禁止訪問
404：沒找到對(duì)應(yīng)頁(yè)面
500：服務(wù)器內(nèi)部出現(xiàn)錯(cuò)誤
501：服務(wù)器不支持實(shí)現(xiàn)請(qǐng)求所需要的功能

?

2.異常處理

URLError捕獲異常信息

#!/usr/bin/env python # -*- coding: utf-8 -*-import urllib.request import urllib.errortry: #嘗試執(zhí)行里面的內(nèi)容html = urllib.request.urlopen('http://www.xiaohuar.com/').read().decode("utf-8")print(html)except urllib.error.URLError as e: #如果出現(xiàn)錯(cuò)誤if hasattr(e,"code"): #如果有錯(cuò)誤代碼print(e.code) #打印錯(cuò)誤代碼if hasattr(e,"reason"): #如果有錯(cuò)誤信息print(e.reason) #打印錯(cuò)誤信息#返回 說明網(wǎng)站禁止了爬蟲訪問 # 403 # Forbidden

?

瀏覽器偽裝技術(shù)

很多網(wǎng)站，做了反爬技術(shù)，一般在后臺(tái)檢測(cè)請(qǐng)求頭信息里是否有User-Agent瀏覽器信息，如果沒有說明不是瀏覽器訪問，就屏蔽了這次請(qǐng)求

所以，我們需要偽裝瀏覽器報(bào)頭來(lái)請(qǐng)求

?

#!/usr/bin/env python # -*- coding: utf-8 -*-import urllib.request url = 'https://www.qiushibaike.com/' #抓取頁(yè)面URL tou = ('User-Agent','Mozilla/5.0 (Windows NT 10.0; WOW64; rv:54.0) Gecko/20100101 Firefox/54.0') #設(shè)置模擬瀏覽器報(bào)頭 b_tou = urllib.request.build_opener() #創(chuàng)建請(qǐng)求對(duì)象 b_tou.addheaders=[tou] #添加報(bào)頭 html = b_tou.open(url).read().decode("utf-8") #開始抓取頁(yè)面 print(html)

?

注意：我們可以看到這次請(qǐng)求并不是用urlopen()方法請(qǐng)求的，此時(shí)用urlopen()無(wú)法請(qǐng)求，但是我們就會(huì)感覺到這樣很費(fèi)勁，難道每次請(qǐng)求都要?jiǎng)?chuàng)建build_opener()，所以我們需要設(shè)置使用urlopen()方法請(qǐng)求自動(dòng)報(bào)頭

?

設(shè)置使用urlopen()方法請(qǐng)求自動(dòng)報(bào)頭，也就是設(shè)置用戶代理

install_opener()將報(bào)頭信息設(shè)置為全局，urlopen()方法請(qǐng)求時(shí)也會(huì)自動(dòng)添加報(bào)頭

#!/usr/bin/env python # -*- coding: utf-8 -*-import urllib.request #設(shè)置報(bào)頭信息 tou = ('User-Agent','Mozilla/5.0 (Windows NT 10.0; WOW64; rv:54.0) Gecko/20100101 Firefox/54.0') #設(shè)置模擬瀏覽器報(bào)頭 b_tou = urllib.request.build_opener() #創(chuàng)建請(qǐng)求對(duì)象 b_tou.addheaders=[tou] #添加報(bào)頭到請(qǐng)求對(duì)象 #將報(bào)頭信息設(shè)置為全局，urlopen()方法請(qǐng)求時(shí)也會(huì)自動(dòng)添加報(bào)頭 urllib.request.install_opener(b_tou)#請(qǐng)求 url = 'https://www.qiushibaike.com/' html = urllib.request.urlopen(url).read().decode("utf-8") print(html)

?

創(chuàng)建用戶代理池

?

#!/usr/bin/env python # -*- coding: utf-8 -*-import urllib.request import random #引入隨機(jī)模塊文件def yh_dl(): #創(chuàng)建用戶代理池yhdl = ['Mozilla/5.0 (Windows; U; Windows NT 6.1; en-us) AppleWebKit/534.50 (KHTML, like Gecko) Version/5.1 Safari/534.50','Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0','Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; Trident/4.0)','Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:2.0.1) Gecko/20100101 Firefox/4.0.1','Mozilla/5.0 (Windows NT 6.1; rv:2.0.1) Gecko/20100101 Firefox/4.0.1','Opera/9.80 (Macintosh; Intel Mac OS X 10.6.8; U; en) Presto/2.8.131 Version/11.11','Opera/9.80 (Windows NT 6.1; U; en) Presto/2.8.131 Version/11.11','Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Maxthon 2.0)','Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; TencentTraveler 4.0)','Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)','Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; The World)','Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; 360SE)','Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Avant Browser)','Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)','Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_3_3 like Mac OS X; en-us) AppleWebKit/533.17.9 (KHTML, like Gecko) Version/5.0.2 Mobile/8J2 Safari/6533.18.5','User-Agent:Mozilla/5.0 (iPod; U; CPU iPhone OS 4_3_3 like Mac OS X; en-us) AppleWebKit/533.17.9 (KHTML, like Gecko) Version/5.0.2 Mobile/8J2 Safari/6533.18.5','Mozilla/5.0 (iPad; U; CPU OS 4_3_3 like Mac OS X; en-us) AppleWebKit/533.17.9 (KHTML, like Gecko) Version/5.0.2 Mobile/8J2 Safari/6533.18.5','Mozilla/5.0 (Linux; U; Android 2.3.7; en-us; Nexus One Build/FRF91) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1','Opera/9.80 (Android 2.3.4; Linux; Opera Mobi/build-1107180945; U; en-GB) Presto/2.8.149 Version/11.10','Mozilla/5.0 (Linux; U; Android 3.0; en-us; Xoom Build/HRI39) AppleWebKit/534.13 (KHTML, like Gecko) Version/4.0 Safari/534.13','Mozilla/5.0 (BlackBerry; U; BlackBerry 9800; en) AppleWebKit/534.1+ (KHTML, like Gecko) Version/6.0.0.337 Mobile Safari/534.1+','Mozilla/5.0 (compatible; MSIE 9.0; Windows Phone OS 7.5; Trident/5.0; IEMobile/9.0; HTC; Titan)','UCWEB7.0.2.37/28/999','NOKIA5700/ UCWEB7.0.2.37/28/999','Openwave/ UCWEB7.0.2.37/28/999','Mozilla/4.0 (compatible; MSIE 6.0; ) Opera/UCWEB7.0.2.37/28/999']thisua = random.choice(yhdl) #隨機(jī)獲取代理信息headers = ("User-Agent",thisua) #拼接報(bào)頭信息opener = urllib.request.build_opener() #創(chuàng)建請(qǐng)求對(duì)象opener.addheaders=[headers] #添加報(bào)頭到請(qǐng)求對(duì)象urllib.request.install_opener(opener) #將報(bào)頭信息設(shè)置為全局，urlopen()方法請(qǐng)求時(shí)也會(huì)自動(dòng)添加報(bào)頭#請(qǐng)求 yh_dl() #執(zhí)行用戶代理池函數(shù) url = 'https://www.qiushibaike.com/' html = urllib.request.urlopen(url).read().decode("utf-8") print(html)

?

這樣爬蟲會(huì)隨機(jī)調(diào)用，用戶代理，也就是隨機(jī)報(bào)頭，保證每次報(bào)頭信息不一樣

轉(zhuǎn)載于:https://www.cnblogs.com/meng-wei-zhi/p/8182538.html

總結(jié)

以上是生活随笔為你收集整理的七 web爬虫讲解2—urllib库爬虫—状态吗—异常处理—浏览器伪装技术、设置用户代理...的全部?jī)?nèi)容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網(wǎng)站內(nèi)容還不錯(cuò)，歡迎將生活随笔推薦給好友。