python3.6.5 + ?pycharm

注意：

一、python3里的 urllib2 已經沒有了，改為了?urllbi.request，因此，直接導入?import urllib.request?即可。

二、必須對正則表達式里的引用變量進行格式變換??.decode('utf-8')，否則會報錯說 不能在一個字節類的對象上使用字符串格式。

?

如下代碼所示。

### 一、 網站地圖爬蟲，控制用戶代理的設定，可捕獲異常、重試下載并設置用戶代理。wswp: web scraping with python import urllib.request ## -- written by LiSongbo def Rocky_dnload(url,user_agent='wswp',num_retries = 2):print('Downloading:',url)LiSongbo_he={'User-agent':user_agent}request = urllib.request.Request(url, headers=LiSongbo_he)try: ## -- written by LiSongbo html = urllib.request.urlopen(request).read()except urllib.request.URLError as e: ## -- written by LiSongbo print('Download error:',e.reason)html = Noneif num_retries > 0: ## -- written by LiSongbo if hasattr(e,'code') and 500 <= e.code < 600:return Rocky_dnload(url,user_agent,num_retries-1) ## retry 5xx HTTP errors return htmlimport re ## -- written by LiSongbo def Rocky_crawl_sitemap(url): ## -- written by LiSongbo sitemap = Rocky_dnload(url) ## download the sitmap file # sitemap = sitemap.decode('utf-8') ## must add this . links = re.findall('<loc>(.*?)</loc>', sitemap) ## extract the sitemap links from flag loc for link in links: ## download each link html = Rocky_dnload(link) ## crape html here Rocky_crawl_sitemap('http://example.webscraping.com/sitemap.xml') 運行結果報錯：

Downloading: http://example.webscraping.com/sitemap.xml
Traceback (most recent call last):
? File "C:/Users/klooa/my_env/book9/test.py", line 25, in <module>
? ? Rocky_crawl_sitemap('http://example.webscraping.com/sitemap.xml')
? File "C:/Users/klooa/my_env/book9/test.py", line 22, in Rocky_crawl_sitemap
? ? links = re.findall('<loc>(.*?)</loc>', sitemap) ?## extract the sitemap links from flag loc
? File "C:\Users\klooa\AppData\Local\Programs\Python\Python36\lib\re.py", line 222, in findall
? ? return _compile(pattern, flags).findall(string)
TypeError: cannot use a string pattern on a bytes-like object

必須在 sitemap的下一行加上 ?

sitemap = sitemap.decode('utf-8')

修改后的運行結果為：

Downloading: http://example.webscraping.com/sitemap.xml
Downloading: http://example.webscraping.com/places/default/view/Afghanistan-1
Downloading: http://example.webscraping.com/places/default/view/Aland-Islands-2
Downloading: http://example.webscraping.com/places/default/view/Albania-3
Downloading: http://example.webscraping.com/places/default/view/Algeria-4
Downloading: http://example.webscraping.com/places/default/view/American-Samoa-5
Downloading: http://example.webscraping.com/places/default/view/Andorra-6
Downloading: http://example.webscraping.com/places/default/view/Angola-7
Downloading: http://example.webscraping.com/places/default/view/Anguilla-8
Downloading: http://example.webscraping.com/places/default/view/Antarctica-9
Downloading: http://example.webscraping.com/places/default/view/Antigua-and-Barbuda-10
Downloading: http://example.webscraping.com/places/default/view/Argentina-11
Downloading: http://example.webscraping.com/places/default/view/Armenia-12
Download error: TOO MANY REQUESTS
Downloading: http://example.webscraping.com/places/default/view/Aruba-13

Download error: TOO MANY REQUESTS

……

## -- written by LiSongbo

轉載于:https://www.cnblogs.com/LiSongbo/p/9245449.html

總結

以上是生活随笔為你收集整理的python3 学习（2）：在网站地图爬虫时的cannot use a string pattern on a bytes-like object 问题的解决方法...的全部內容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網站內容還不錯，歡迎將生活随笔推薦給好友。