python 多功能下载网页
生活随笔
收集整理的這篇文章主要介紹了
python 多功能下载网页
小編覺(jué)得挺不錯(cuò)的,現(xiàn)在分享給大家,幫大家做個(gè)參考.
#下載網(wǎng)頁(yè)
#具有功能：捕獲異常，重試下載并設(shè)置用戶代理
import urllib.request
import urllib.error
#下載網(wǎng)頁(yè)
#wscp:默認(rèn)用戶代理 web scraping with python 縮寫(xiě)
def download(url, user_agent='wscp',num_retries=2):print('Downloading:',url) #打印下載路徑headers={'USer-Agent':user_agent}request=urllib.request.Request(url,headers=headers)try:html=urllib.request.urlopen(request).read()except urllib.error.URLError as e:print('download error:',e.reason)html=Noneif num_retries>0:#下載遇到錯(cuò)誤時(shí)嘗試下載if hasattr(e,'code') and 500 <=e.code <600:#404 notfound 這種錯(cuò)誤，說(shuō)明網(wǎng)頁(yè)不存在，故不需要重新下載print(user_agent)return download(url,user_agent,num_retries-1)return html
download('http://example.webscraping.com/')
download('http://httpstat.us/500')#測(cè)試錯(cuò)誤500
# print(dir(urllib)) Downloading: http://example.webscraping.com/
Downloading: http://httpstat.us/500
download error: Internal Server Error
wscp
Downloading: http://httpstat.us/500
download error: Internal Server Error
wscp
Downloading: http://httpstat.us/500
download error: Internal Server Error
?
轉(zhuǎn)載于:https://www.cnblogs.com/liangliangzz/p/10160482.html
總結(jié)
以上是生活随笔為你收集整理的python 多功能下载网页的全部?jī)?nèi)容，希望文章能夠幫你解決所遇到的問(wèn)題。
