系統(tǒng)環(huán)境：

操作系統(tǒng)：Windows8.1專業(yè)版 64bit Python：anaconda、Python2.7 Python modules:requests、random、json

Background:

對(duì)于靜態(tài)網(wǎng)頁(yè)，我們只需要把網(wǎng)頁(yè)地址欄中的url傳到get請(qǐng)求中就可以輕松地獲取到網(wǎng)頁(yè)的數(shù)據(jù)。但是，我們經(jīng)常會(huì)遇到直接把網(wǎng)頁(yè)地址欄中的url傳到get請(qǐng)求中無(wú)法直接獲取到網(wǎng)頁(yè)的數(shù)據(jù)的情況，而且右鍵查看網(wǎng)頁(yè)源代碼也無(wú)法看到網(wǎng)頁(yè)的數(shù)據(jù)，同時(shí)點(diǎn)擊第二頁(yè)、第三頁(yè)等進(jìn)行翻頁(yè)的時(shí)候，網(wǎng)頁(yè)地址欄中的url也沒(méi)變，這些就是動(dòng)態(tài)網(wǎng)頁(yè)，例如：

自律監(jiān)管措施 - 全國(guó)中小企業(yè)股份轉(zhuǎn)讓系統(tǒng)?www.neeq.com.cn

解決辦法：

對(duì)于動(dòng)態(tài)網(wǎng)頁(yè)抓取的關(guān)鍵是先分析網(wǎng)頁(yè)數(shù)據(jù)獲取和跳轉(zhuǎn)的邏輯，再去寫(xiě)代碼。接下來(lái)，將以上面的那個(gè)網(wǎng)頁(yè)為例，介紹如何利用Python來(lái)爬取動(dòng)態(tài)網(wǎng)頁(yè)的數(shù)據(jù)。

1、分析網(wǎng)頁(yè)數(shù)據(jù)請(qǐng)求和跳轉(zhuǎn)的邏輯：

如上圖所示，我們打開(kāi)網(wǎng)頁(yè)之后，按“F12”進(jìn)入chrome瀏覽器的開(kāi)發(fā)工具，點(diǎn)擊“Network”->XHR（有時(shí)候是JS），然后我們點(diǎn)擊上面的頁(yè)面跳轉(zhuǎn)欄的“2”跳轉(zhuǎn)到第二頁(yè)，然后我們可以看到開(kāi)發(fā)工具左邊的框里出現(xiàn)了一個(gè)新的請(qǐng)求，即左下圖的最下面那一行（藍(lán)色那條），我們用鼠標(biāo)點(diǎn)擊它，就可以在右邊顯示出該請(qǐng)求的headers的相關(guān)信息。在Headers中我們可以知道：Requests URL就是該網(wǎng)頁(yè)真正請(qǐng)求的URL，而且由Request Method可以知道這是一個(gè)post請(qǐng)求，而下面的Request Headers就是該請(qǐng)求所需要設(shè)置的headers參數(shù)。因?yàn)檫@是一個(gè)post請(qǐng)求，所以我們要查看一下post請(qǐng)求提交了那些數(shù)據(jù)，所以我們可以在右邊的Headers中繼續(xù)往下拉來(lái)查看。

所以由上圖的Form Data我們可以知道，post請(qǐng)求上傳了兩個(gè)關(guān)鍵的數(shù)據(jù)：disclosureType和page，到此我們就成功地分析了該動(dòng)態(tài)網(wǎng)頁(yè)數(shù)據(jù)請(qǐng)求和跳轉(zhuǎn)的邏輯，接下來(lái)通過(guò)編程來(lái)實(shí)現(xiàn)爬取該網(wǎng)頁(yè)的數(shù)據(jù)。

2、Coding:

# -*- coding: utf-8 -*- """ Created on Tue May 01 18:52:49 2018 @author: gmn """ #導(dǎo)入requests module import requests #導(dǎo)入random module import random #導(dǎo)入json module import json# ============================================================================= # 應(yīng)對(duì)網(wǎng)站反爬蟲(chóng)的相關(guān)設(shè)置 # ============================================================================= #User-Agent列表，這個(gè)可以自己在網(wǎng)上搜到，用于偽裝瀏覽器的User Agent USER_AGENTS = ["Mozilla/5.0 (Windows; U; Windows NT 5.2) Gecko/2008070208 Firefox/3.0.1""Mozilla/5.0 (Windows; U; Windows NT 6.1; en-us) AppleWebKit/534.50 (KHTML, like Gecko) Version/5.1 Safari/534.50","Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:2.0.1) Gecko/20100101 Firefox/4.0.1","Mozilla/5.0 (Windows NT 6.1; rv:2.0.1) Gecko/20100101 Firefox/4.0.1","Opera/9.80 (Macintosh; Intel Mac OS X 10.6.8; U; en) Presto/2.8.131 Version/11.11","Opera/9.80 (Windows NT 6.1; U; en) Presto/2.8.131 Version/11.11","Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_0) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.56 Safari/535.11","Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E)","Opera/9.80 (Windows NT 5.1; U; zh-cn) Presto/2.9.168 Version/11.50","Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0","Mozilla/5.0 (Windows NT 5.2) AppleWebKit/534.30 (KHTML, like Gecko) Chrome/12.0.742.122 Safari/534.30","Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.11 (KHTML, like Gecko) Chrome/20.0.1132.11 TaoBrowser/2.0 Safari/536.11","Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.71 Safari/537.1 LBBROWSER","Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E; LBBROWSER)","Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Trident/4.0; SV1; QQDownload 732; .NET4.0C; .NET4.0E; 360SE)","Mozilla/5.0 (Windows NT 5.1) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.84 Safari/535.11 SE 2.X MetaSr 1.0","Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0)","Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.2)","Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)","Mozilla/4.0 (compatible; MSIE 5.0; Windows NT)","Mozilla/5.0 (Windows; U; Windows NT 5.2) Gecko/2008070208 Firefox/3.0.1","Mozilla/5.0 (Windows; U; Windows NT 5.1) Gecko/20070309 Firefox/2.0.0.3","Mozilla/5.0 (Windows; U; Windows NT 5.1) Gecko/20070803 Firefox/1.5.0.12 "] #IP地址列表，用于設(shè)置IP代理 IP_AGENTS = ["http://58.240.53.196:8080", "http://219.135.99.185:8088","http://117.127.0.198:8080","http://58.240.53.194:8080" ]#設(shè)置IP代理 proxies={"http":random.choice(IP_AGENTS)} # ============================================================================= # 上面的設(shè)置是為了應(yīng)對(duì)網(wǎng)站的反爬蟲(chóng)，與具體的網(wǎng)頁(yè)爬取無(wú)關(guān) # =============================================================================# ============================================================================= # 下面這些是根據(jù)剛才第一步的分析來(lái)設(shè)置的，所以下面需要按照第一步的分析來(lái)設(shè)置對(duì)應(yīng)的參數(shù)。 # 根據(jù)第一步圖片的右下角部分來(lái)設(shè)置Cookie、url、headers和post參數(shù) # ============================================================================= #設(shè)置cookie Cookie = "Hm_lvt_b58fe8237d8d72ce286e1dbd2fc8308c=1525162758; BIGipServerNEEQ_8000-NEW=83952564.16415.0000; JSESSIONID=E50D2B8270D728502754D4330CB0E275; Hm_lpvt_b58fe8237d8d72ce286e1dbd2fc8308c=1525165761" #設(shè)置動(dòng)態(tài)js的url url = 'http://www.neeq.com.cn/disclosureInfoController/infoResult.do?callback=jQuery18307528463705200819_1525173495230' #設(shè)置requests請(qǐng)求的 headers headers = {'User-agent': random.choice(USER_AGENTS), #設(shè)置get請(qǐng)求的User-Agent，用于偽裝瀏覽器UA 'Cookie': Cookie,'Connection': 'keep-alive','Accept': 'text/javascript, application/javascript, application/ecmascript, application/x-ecmascript, */*; q=0.01','Accept-Encoding': 'gzip, deflate','Accept-Language': 'zh-CN,zh;q=0.9','Host': 'www.neeq.com.cn','Referer': 'http://www.neeq.com.cn/disclosure/supervise.html' } #設(shè)置頁(yè)面索引 pageIndex=0 #設(shè)置url post請(qǐng)求的參數(shù) data={'page':pageIndex,'disclosureType':8}#requests post請(qǐng)求 req=requests.post(url,data=data,headers=headers,proxies=proxies) print(req.content) #通過(guò)打印req.content，我們可以知道post請(qǐng)求返回的是json數(shù)據(jù)，而且該數(shù)據(jù)是一個(gè)字符串類型的 #獲取包含json數(shù)據(jù)的字符串 #str_data=req.content ##獲取json字符串?dāng)?shù)據(jù) #str_json=str_data[8:-2] #print(str_json) ##把json數(shù)據(jù)轉(zhuǎn)成dict類型 #json_Info=json.loads(str_json)

運(yùn)行結(jié)果如下：

我們可以看到返回的數(shù)據(jù)req.content為json格式的數(shù)據(jù)，但是json數(shù)據(jù)的前面和后面分別是"jQuery18307528463705200819_1525173495230(["和"])"，所以我們要去掉這兩部分，留下中間的json格式的數(shù)據(jù)。在此之前，我們可以發(fā)現(xiàn)“jQuery18307528463705200819_1525173495230”就是我們的url參數(shù)“callback”的值，所以為了去掉jQuery后面的一大串?dāng)?shù)字，我們可以把“callback”的值改成“jQuery”（當(dāng)然你也可以改成其他的值），所以u(píng)rl變?yōu)?#39;

http://www.neeq.com.cn/disclosureInfoController/infoResult.do?callback=jQuery?www.neeq.com.cn

在此運(yùn)行代碼，可以得到：

而且我們發(fā)現(xiàn)req.content是一個(gè)字符串類型的數(shù)據(jù)，所以我們可以用：

#獲取json字符串?dāng)?shù)據(jù) str_json=str_data[8:-2]

來(lái)獲取我們需要的中間的那部分json數(shù)據(jù)，此時(shí)代碼如下：

# -*- coding: utf-8 -*- """ Created on Tue May 01 18:52:49 2018 @author: gmn """ #導(dǎo)入requests module import requests #導(dǎo)入random module import random #導(dǎo)入json module import json# ============================================================================= # 應(yīng)對(duì)網(wǎng)站反爬蟲(chóng)的相關(guān)設(shè)置 # ============================================================================= #User-Agent列表，這個(gè)可以自己在網(wǎng)上搜到，用于偽裝瀏覽器的User Agent USER_AGENTS = ["Mozilla/5.0 (Windows; U; Windows NT 5.2) Gecko/2008070208 Firefox/3.0.1""Mozilla/5.0 (Windows; U; Windows NT 6.1; en-us) AppleWebKit/534.50 (KHTML, like Gecko) Version/5.1 Safari/534.50","Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:2.0.1) Gecko/20100101 Firefox/4.0.1","Mozilla/5.0 (Windows NT 6.1; rv:2.0.1) Gecko/20100101 Firefox/4.0.1","Opera/9.80 (Macintosh; Intel Mac OS X 10.6.8; U; en) Presto/2.8.131 Version/11.11","Opera/9.80 (Windows NT 6.1; U; en) Presto/2.8.131 Version/11.11","Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_0) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.56 Safari/535.11","Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E)","Opera/9.80 (Windows NT 5.1; U; zh-cn) Presto/2.9.168 Version/11.50","Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0","Mozilla/5.0 (Windows NT 5.2) AppleWebKit/534.30 (KHTML, like Gecko) Chrome/12.0.742.122 Safari/534.30","Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.11 (KHTML, like Gecko) Chrome/20.0.1132.11 TaoBrowser/2.0 Safari/536.11","Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.71 Safari/537.1 LBBROWSER","Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E; LBBROWSER)","Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Trident/4.0; SV1; QQDownload 732; .NET4.0C; .NET4.0E; 360SE)","Mozilla/5.0 (Windows NT 5.1) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.84 Safari/535.11 SE 2.X MetaSr 1.0","Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0)","Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.2)","Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)","Mozilla/4.0 (compatible; MSIE 5.0; Windows NT)","Mozilla/5.0 (Windows; U; Windows NT 5.2) Gecko/2008070208 Firefox/3.0.1","Mozilla/5.0 (Windows; U; Windows NT 5.1) Gecko/20070309 Firefox/2.0.0.3","Mozilla/5.0 (Windows; U; Windows NT 5.1) Gecko/20070803 Firefox/1.5.0.12 "] #IP地址列表，用于設(shè)置IP代理 IP_AGENTS = ["http://58.240.53.196:8080", "http://219.135.99.185:8088","http://117.127.0.198:8080","http://58.240.53.194:8080" ]#設(shè)置IP代理 proxies={"http":random.choice(IP_AGENTS)} # ============================================================================= # 上面的設(shè)置是為了應(yīng)對(duì)網(wǎng)站的反爬蟲(chóng)，與具體的網(wǎng)頁(yè)爬取無(wú)關(guān) # =============================================================================# ============================================================================= # 下面這些是根據(jù)剛才第一步的分析來(lái)設(shè)置的，所以下面需要按照第一步的分析來(lái)設(shè)置對(duì)應(yīng)的參數(shù)。 # 根據(jù)第一步圖片的右下角部分來(lái)設(shè)置Cookie、url、headers和post參數(shù) # ============================================================================= #設(shè)置cookie Cookie = "Hm_lvt_b58fe8237d8d72ce286e1dbd2fc8308c=1525162758; BIGipServerNEEQ_8000-NEW=83952564.16415.0000; JSESSIONID=E50D2B8270D728502754D4330CB0E275; Hm_lpvt_b58fe8237d8d72ce286e1dbd2fc8308c=1525165761" #設(shè)置動(dòng)態(tài)js的url url = 'http://www.neeq.com.cn/disclosureInfoController/infoResult.do?callback=jQuery' #設(shè)置requests請(qǐng)求的 headers headers = {'User-agent': random.choice(USER_AGENTS), #設(shè)置get請(qǐng)求的User-Agent，用于偽裝瀏覽器UA 'Cookie': Cookie,'Connection': 'keep-alive','Accept': 'text/javascript, application/javascript, application/ecmascript, application/x-ecmascript, */*; q=0.01','Accept-Encoding': 'gzip, deflate','Accept-Language': 'zh-CN,zh;q=0.9','Host': 'www.neeq.com.cn','Referer': 'http://www.neeq.com.cn/disclosure/supervise.html' } #設(shè)置頁(yè)面索引 pageIndex=0 #設(shè)置url post請(qǐng)求的參數(shù) data={'page':pageIndex,'disclosureType':8}#requests post請(qǐng)求 req=requests.post(url,data=data,headers=headers,proxies=proxies) #print(req.content) #通過(guò)打印req.content，我們可以知道post請(qǐng)求返回的是json數(shù)據(jù)，而且該數(shù)據(jù)是一個(gè)字符串類型的 #獲取包含json數(shù)據(jù)的字符串 str_data=req.content #獲取json字符串?dāng)?shù)據(jù) str_json=str_data[8:-2] print(str_json) #把json數(shù)據(jù)轉(zhuǎn)成dict類型 #json_Info=json.loads(str_json)

運(yùn)行結(jié)果如下：

我們把str_json打印出來(lái)的字符串復(fù)制粘貼到網(wǎng)上的json在線解析工具來(lái)分析該數(shù)據(jù)的規(guī)律，結(jié)果如下：

由右圖，我們可以發(fā)現(xiàn)json數(shù)據(jù)的規(guī)律。接下來(lái)，我們先把str_json轉(zhuǎn)成dict字典類型的數(shù)據(jù)：

#把json數(shù)據(jù)轉(zhuǎn)成dict類型 json_Info=json.loads(str_json)

然后就可以通過(guò)字典數(shù)據(jù)的相關(guān)操作來(lái)獲取網(wǎng)頁(yè)的相關(guān)數(shù)據(jù)了。

完整代碼如下：

# -*- coding: utf-8 -*- """ Created on Tue May 01 18:52:49 2018 @author: gmn """ #導(dǎo)入requests module import requests #導(dǎo)入random module import random #導(dǎo)入json module import json# ============================================================================= # 應(yīng)對(duì)網(wǎng)站反爬蟲(chóng)的相關(guān)設(shè)置 # ============================================================================= #User-Agent列表，這個(gè)可以自己在網(wǎng)上搜到，用于偽裝瀏覽器的User Agent USER_AGENTS = ["Mozilla/5.0 (Windows; U; Windows NT 5.2) Gecko/2008070208 Firefox/3.0.1""Mozilla/5.0 (Windows; U; Windows NT 6.1; en-us) AppleWebKit/534.50 (KHTML, like Gecko) Version/5.1 Safari/534.50","Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:2.0.1) Gecko/20100101 Firefox/4.0.1","Mozilla/5.0 (Windows NT 6.1; rv:2.0.1) Gecko/20100101 Firefox/4.0.1","Opera/9.80 (Macintosh; Intel Mac OS X 10.6.8; U; en) Presto/2.8.131 Version/11.11","Opera/9.80 (Windows NT 6.1; U; en) Presto/2.8.131 Version/11.11","Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_0) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.56 Safari/535.11","Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E)","Opera/9.80 (Windows NT 5.1; U; zh-cn) Presto/2.9.168 Version/11.50","Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0","Mozilla/5.0 (Windows NT 5.2) AppleWebKit/534.30 (KHTML, like Gecko) Chrome/12.0.742.122 Safari/534.30","Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.11 (KHTML, like Gecko) Chrome/20.0.1132.11 TaoBrowser/2.0 Safari/536.11","Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.71 Safari/537.1 LBBROWSER","Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E; LBBROWSER)","Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Trident/4.0; SV1; QQDownload 732; .NET4.0C; .NET4.0E; 360SE)","Mozilla/5.0 (Windows NT 5.1) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.84 Safari/535.11 SE 2.X MetaSr 1.0","Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0)","Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.2)","Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)","Mozilla/4.0 (compatible; MSIE 5.0; Windows NT)","Mozilla/5.0 (Windows; U; Windows NT 5.2) Gecko/2008070208 Firefox/3.0.1","Mozilla/5.0 (Windows; U; Windows NT 5.1) Gecko/20070309 Firefox/2.0.0.3","Mozilla/5.0 (Windows; U; Windows NT 5.1) Gecko/20070803 Firefox/1.5.0.12 "] #IP地址列表，用于設(shè)置IP代理 IP_AGENTS = ["http://58.240.53.196:8080", "http://219.135.99.185:8088","http://117.127.0.198:8080","http://58.240.53.194:8080" ]#設(shè)置IP代理 proxies={"http":random.choice(IP_AGENTS)} # ============================================================================= # 上面的設(shè)置是為了應(yīng)對(duì)網(wǎng)站的反爬蟲(chóng)，與具體的網(wǎng)頁(yè)爬取無(wú)關(guān) # =============================================================================# ============================================================================= # 下面這些是根據(jù)剛才第一步的分析來(lái)設(shè)置的，所以下面需要按照第一步的分析來(lái)設(shè)置對(duì)應(yīng)的參數(shù)。 # 根據(jù)第一步圖片的右下角部分來(lái)設(shè)置Cookie、url、headers和post參數(shù) # ============================================================================= #設(shè)置cookie Cookie = "Hm_lvt_b58fe8237d8d72ce286e1dbd2fc8308c=1525162758; BIGipServerNEEQ_8000-NEW=83952564.16415.0000; JSESSIONID=E50D2B8270D728502754D4330CB0E275; Hm_lpvt_b58fe8237d8d72ce286e1dbd2fc8308c=1525165761" #設(shè)置動(dòng)態(tài)js的url url = 'http://www.neeq.com.cn/disclosureInfoController/infoResult.do?callback=jQuery' #設(shè)置requests請(qǐng)求的 headers headers = {'User-agent': random.choice(USER_AGENTS), #設(shè)置get請(qǐng)求的User-Agent，用于偽裝瀏覽器UA 'Cookie': Cookie,'Connection': 'keep-alive','Accept': 'text/javascript, application/javascript, application/ecmascript, application/x-ecmascript, */*; q=0.01','Accept-Encoding': 'gzip, deflate','Accept-Language': 'zh-CN,zh;q=0.9','Host': 'www.neeq.com.cn','Referer': 'http://www.neeq.com.cn/disclosure/supervise.html' } #設(shè)置頁(yè)面索引 pageIndex=0 #設(shè)置url post請(qǐng)求的參數(shù) data={'page':pageIndex,'disclosureType':8}#requests post請(qǐng)求 req=requests.post(url,data=data,headers=headers,proxies=proxies) #print(req.content) #通過(guò)打印req.content，我們可以知道post請(qǐng)求返回的是json數(shù)據(jù)，而且該數(shù)據(jù)是一個(gè)字符串類型的 #獲取包含json數(shù)據(jù)的字符串 str_data=req.content #獲取json字符串?dāng)?shù)據(jù) str_json=str_data[8:-2] #print(str_json) #把json數(shù)據(jù)轉(zhuǎn)成dict類型 json_Info=json.loads(str_json)

注意事項(xiàng)：

有時(shí)候我們按照以上步驟，仍然難以準(zhǔn)確的找到數(shù)據(jù)訪問(wèn)的URL的時(shí)候，可以考慮使用selenium + 瀏覽器driver (如：chromedriver)的方式來(lái)爬取動(dòng)態(tài)網(wǎng)頁(yè)，不過(guò)這種方式比較耗時(shí)間。

總結(jié)

以上是生活随笔為你收集整理的chrome动态ip python_用Python爬虫爬取动态网页，附带完整代码，有错误欢迎指出！...的全部?jī)?nèi)容，希望文章能夠幫你解決所遇到的問(wèn)題。

如果覺(jué)得生活随笔網(wǎng)站內(nèi)容還不錯(cuò)，歡迎將生活随笔推薦給好友。