python獲取百度遷徙大數(shù)據(jù)

在疫情期間，百度遷徙大數(shù)據(jù)大放異彩，很多研究都是基于百度遷徙數(shù)據(jù)來(lái)進(jìn)行的，由于本次研究也依賴與百度遷徙數(shù)據(jù)，所以小試牛刀。

首先，打開(kāi)目標(biāo)網(wǎng)站 http://qianxi.baidu.com/，觀察網(wǎng)頁(yè)布局

其次，F12打開(kāi)源網(wǎng)頁(yè)，Ctrl+R刷新，找到數(shù)據(jù)對(duì)應(yīng)item

接著，切換源網(wǎng)頁(yè)的header，找到請(qǐng)求url

然后，復(fù)制url到瀏覽器打開(kāi)

發(fā)現(xiàn)兩者一毛一樣，接下來(lái)就是分析數(shù)據(jù)結(jié)構(gòu)和寫(xiě)代碼提取數(shù)據(jù)，先來(lái)一個(gè)按照省份級(jí)別遷入，并寫(xiě)入到csv里面

def get_country_province(): #獲取全國(guó)省份級(jí)別流入url='http://huiyan.baidu.com/migration/provincerank.jsonp?dt=country&id=0&type=move_in&date=20200218'response=requests.get(url, timeout=10) # #發(fā)出請(qǐng)求并json化處理time.sleep(1) #掛起一秒r=response.text[3:-1] #去頭去尾data_dict=json.loads(r) #字典化if data_dict['errmsg']=='SUCCESS':data_list=data_dict['data']['list']with open("全國(guó)省份級(jí)別流入.csv", "w+", newline="") as csv_file:writer=csv.writer(csv_file)header=["province_name", "value"] #表頭"city_name",writer.writerow(header) #把表頭寫(xiě)入for i in range(len(data_list)):province_name=data_list[i]['province_name'] #省份名value=data_list[i]['value']writer.writerow([ province_name, value]) #city_name

有進(jìn)就有出，回?fù)粼W(wǎng)頁(yè)點(diǎn)擊熱門(mén)遷出地，然后會(huì)發(fā)現(xiàn)源網(wǎng)頁(yè)多了一條item

獲取相應(yīng)的url，寫(xiě)代碼如下獲取全國(guó)城市級(jí)別遷出數(shù)據(jù)

def get_country_city(): #獲取全國(guó)城市級(jí)別流出url='http://huiyan.baidu.com/migration/cityrank.jsonp?dt=country&id=0&type=move_out&date=20200218'response=requests.get(url, timeout=10) # #發(fā)出請(qǐng)求并json化處理time.sleep(1) #掛起一秒r=response.text[3:-1] #去頭去尾data_dict=json.loads(r) #字典化if data_dict['errmsg']=='SUCCESS':data_list=data_dict['data']['list']with open("全國(guó)城市級(jí)別流出.csv", "w+", newline="") as csv_file:writer=csv.writer(csv_file)header=["city_name", "province_name", "value"] #表頭city_namewriter.writerow(header) #把表頭寫(xiě)入for i in range(len(data_list)):city_name=data_list[i]['city_name'] #城市名province_name=data_list[i]['province_name'] #省份名value=data_list[i]['value']writer.writerow([city_name, province_name, value]) #city_name

代碼成功運(yùn)行后，文件夾下會(huì)多出兩個(gè)csv文件，截圖如下

獲取兩個(gè)全國(guó)層面的數(shù)據(jù)，你或許會(huì)有所感悟，那就是不同日期，不同城市，不同遷徙方向，不同層級(jí)（省份，城市）的請(qǐng)求都體現(xiàn)在url的參數(shù)里面

• cityrank 表示城市級(jí)別
• provincerank表示省份級(jí)別
• id表示中華人民共和國(guó)縣以上行政區(qū)劃代碼，這個(gè)可以通過(guò)如下網(wǎng)站獲取 http://www.mca.gov.cn/article/sj/xzqh/2019/2019/201912251506.html
• move_out表示遷出
• move_in表示遷入
• date表示日期
  下面給出如何循環(huán)構(gòu)造不同組合的url代碼

def generate_url(): #定義生成不同時(shí)期，不同城市，不同遷徙方向，不同層級(jí)的請(qǐng)求urldate_list=['20200129', '20200130', '20200131', '20200201', '20200202', '20200203','20200204', '20200205', '20200206', '20200207', '20200208', '20200209'] #初五到初十六時(shí)間段directions=['in', 'out'] #遷徙方向level_list=['province','city'] #數(shù)據(jù)級(jí)別city_list={'北京':'110000', '上海': '310000', '廣州':'440100', '深圳': '440300','杭州': '330100', '南京': '320100', '天津': '120000', '成都': '510100', '武漢': '420100', '蘇州': '320500'} #構(gòu)造重點(diǎn)10城字典urls=[] #用來(lái)存放urlfor city_id in list(city_list.values()): #城市idfor direction in directions: #遷入還是遷出for date in date_list: #對(duì)日期循環(huán)for level in level_list: url='http://huiyan.baidu.com/migration/{}rank.jsonp?dt=country&id={}&type=move_{}&&date={}'.format(level, city_id, direction, date) #請(qǐng)求urlprint(url)urls.append(url) #追加進(jìn)去return urls

最后貼出完整代碼

# -*- coding: utf-8 -*- """ Created on Tue Feb 18 14:25:58 2020 project name:百度遷徙 @author: 帥帥de三叔 """ import requests #導(dǎo)入請(qǐng)求模塊 import json #導(dǎo)入json模塊 import csv #導(dǎo)入csv模塊 import time #導(dǎo)入時(shí)間模塊def get_country_province(): #獲取全國(guó)省份級(jí)別流入url='http://huiyan.baidu.com/migration/provincerank.jsonp?dt=country&id=0&type=move_in&date=20200218'response=requests.get(url, timeout=10) # #發(fā)出請(qǐng)求并json化處理time.sleep(1) #掛起一秒r=response.text[3:-1] #去頭去尾data_dict=json.loads(r) #字典化if data_dict['errmsg']=='SUCCESS':data_list=data_dict['data']['list']with open("全國(guó)省份級(jí)別流入.csv", "w+", newline="") as csv_file:writer=csv.writer(csv_file)header=["province_name", "value"] #表頭"city_name",writer.writerow(header) #把表頭寫(xiě)入for i in range(len(data_list)):province_name=data_list[i]['province_name'] #省份名value=data_list[i]['value']writer.writerow([ province_name, value]) #city_namedef get_country_city(): #獲取全國(guó)城市級(jí)別流出url='http://huiyan.baidu.com/migration/cityrank.jsonp?dt=country&id=0&type=move_out&date=20200218'response=requests.get(url, timeout=10) # #發(fā)出請(qǐng)求并json化處理time.sleep(1) #掛起一秒r=response.text[3:-1] #去頭去尾data_dict=json.loads(r) #字典化if data_dict['errmsg']=='SUCCESS':data_list=data_dict['data']['list']with open("全國(guó)城市級(jí)別流出.csv", "w+", newline="") as csv_file:writer=csv.writer(csv_file)header=["city_name", "province_name", "value"] #表頭city_namewriter.writerow(header) #把表頭寫(xiě)入for i in range(len(data_list)):city_name=data_list[i]['city_name'] #城市名province_name=data_list[i]['province_name'] #省份名value=data_list[i]['value']writer.writerow([city_name, province_name, value]) #city_namedef generate_url(): #定義生成不同時(shí)期，不同城市，不同遷徙方向，不同層級(jí)的請(qǐng)求urldate_list=['20200129', '20200130', '20200131', '20200201', '20200202', '20200203','20200204', '20200205', '20200206', '20200207', '20200208', '20200209'] #初五到初十六時(shí)間段directions=['in', 'out'] #遷徙方向level_list=['province','city'] #數(shù)據(jù)級(jí)別city_list={'北京':'110000', '上海': '310000', '廣州':'440100', '深圳': '440300','杭州': '330100', '南京': '320100', '天津': '120000', '成都': '510100', '武漢': '420100', '蘇州': '320500'} #構(gòu)造重點(diǎn)10城字典urls=[] #用來(lái)存放urlfor city_id in list(city_list.values()): #城市idfor direction in directions: #遷入還是遷出for date in date_list: #對(duì)日期循環(huán)for level in level_list: url='http://huiyan.baidu.com/migration/{}rank.jsonp?dt=country&id={}&type=move_{}&&date={}'.format(level, city_id, direction, date) #請(qǐng)求urlprint(url)urls.append(url) #追加進(jìn)去return urlsif __name__=="__main__":get_country_province()get_country_city()generate_url()

Python爬蟲(chóng)僅為學(xué)習(xí)交流，如有冒犯，請(qǐng)告知?jiǎng)h。

總結(jié)

以上是生活随笔為你收集整理的python获取百度迁徙大数据的全部?jī)?nèi)容，希望文章能夠幫你解決所遇到的問(wèn)題。

如果覺(jué)得生活随笔網(wǎng)站內(nèi)容還不錯(cuò)，歡迎將生活随笔推薦給好友。