App中的數(shù)據(jù)可以用網(wǎng)絡(luò)爬蟲抓取么

答案是完全肯定的：凡是可以看到的APP數(shù)據(jù)都可以抓取。

下面我就介紹下自己的學(xué)習(xí)經(jīng)驗和一些方法吧 本篇適合有過web爬蟲基礎(chǔ)的程序猿看

沒有的的話學(xué)的可能會吃力一些

App接口爬取數(shù)據(jù)過程

• 使用抓包工具
• 手機(jī)使用代理，app所有請求通過抓包工具
• 獲得接口，分析接口
• 反編譯apk獲取key
• 突破反爬限制

需要的工具：

• 夜神模擬器
• Fiddler
• Pycharm

實現(xiàn)過程

首先下載夜神模擬器模擬手機(jī)也可以用真機(jī)，然后下載Fiddler抓取手機(jī)APP數(shù)據(jù)包，分析接口完成以后使用Python實現(xiàn)爬蟲程序

Fiddler安裝配置過程

第一步：下載神器Fiddler

Fiddler下載完成之后，傻瓜式的安裝一下！

第二步：設(shè)置Fiddler

打開Fiddler, Tools-> Fiddler Options (配置完后記得要重啟Fiddler)

選中"Decrpt HTTPS traffic", Fiddler就可以截獲HTTPS請求

選中"Allow remote computers to connect". 是允許別的機(jī)器把HTTP/HTTPS請求發(fā)送到Fiddler上來

記住這個端口號是:8888

夜神模擬器安裝配置過程

第一步:下載安裝

• 夜神模擬器下載完成之后，傻瓜式的安裝一下！

第二步:配置橋接 實現(xiàn)互通

• 首先將當(dāng)前手機(jī)網(wǎng)絡(luò)橋接到本電腦網(wǎng)絡(luò) 實現(xiàn)互通

安裝完成橋接驅(qū)動后配置IP地址，要配成和本機(jī)互通的網(wǎng)段，配置完成后打開主機(jī)cmd終端ping通ok

第三步:配置代理

打開主機(jī)cmd

2、輸入ipconfig查看本機(jī)IP

3、配置代理

進(jìn)入夜神模擬器–打開設(shè)置–打開WLAN

點擊修改網(wǎng)絡(luò)–配置代理 如下圖：

配置完后保存

到這里我們就設(shè)置好所有的值，下面就來測試一下，打開手機(jī)的超級課程表APP

4、在夜神模擬器上下載你想爬取得App使用Fiddler抓包分析api后使用python進(jìn)行爬取就可以了

爬取充電網(wǎng)APP實例

爬取部分內(nèi)容截圖：

部分python代碼分享：

import requests import city import json import jsonpath import re ''' 更多Python學(xué)習(xí)資料以及源碼教程資料，可以加群821460695 免費獲取 ''' city_list = city.jsons tags_list = city.Tagdef city_func(city_id):try:city = jsonpath.jsonpath(city_list, '$..sub[?(@.code=={})]'.format(int(city_id)))[0]["name"]except:city = jsonpath.jsonpath(city_list, '$[?(@.code=={})]'.format(int(city_id)))[0]["name"]return citydef tags_func(tags_id):tags_join = []if tags_id:for tags in tags_id:t = jsonpath.jsonpath(tags_list,'$..spotFilterTags[?(@.id=={})]'.format(int(tags)))tags_join.append(t[0]["title"])return ('-'.join(tags_join))def split_n(ags):return re.sub('n',' ',ags)def request(page):print('開始下載第%d頁'%page)url = 'https://app-api.chargerlink.com/spot/searchSpot'two_url = "https://app-api.chargerlink.com/spot/getSpotDetail?spotId=ozvdkddzhkzd"head = {"device": "client=android&cityName=%E5%8C%97%E4%BA%AC%E5%B8%82&cityCode=110106&lng=116.32154281224254&device_id=8A261C9D60ACEBDED7CD3706C92DD68E&ver=3.7.7&lat=39.895024107858724&network=WIFI&os_version=19","appId": "20171010","timestamp": "1532342711477","signature": "36daaa33e7b0d5d29ac9c64a2ce6c4cf","forcecheck": "1","Content-Type": "application/x-www-form-urlencoded","Content-Length": "68","Host": "app-api.chargerlink.com","Connection": "Keep-Alive","User-Agent": "okhttp/3.2.0"}data = {"userFilter[operateType]": 2,"cityCode": 110000,"sort": 1,"page": page,"limit": 10,}response = requests.post(url,data=data,headers=head)#獲取數(shù)據(jù)data = response.json()for i in data['data']:c = []id = i['id']name = i["name"] #充電樁名phone = i["phone"] #手機(jī)號num = i['quantity'] #有幾個充電樁city = city_func(i["provinceCode"]) #城市tags =tags_func(i["tags"].split(','))#標(biāo)簽message = c + [id,name,phone,num,city,tags]parse_info(two_url.format(d=id),message)def parse_info(url,message):#打開文件with open('car.csv','a',encoding='utf-8')as c:head = {"device": "client=android&cityName=&cityCode=&lng=116.32154281224254&device_id=8A261C9D60ACEBDED7CD3706C92DD68E&ver=3.7.7&lat=39.895024107858724&network=WIFI&os_version=19","TOKEN": "036c8e24266c9089db50899287a99e65dc3bf95f","appId": "20171010","timestamp": "1532357165598","signature": "734ecec249f86193d6e54449ec5e8ff6","forcecheck": "1","Host": "app-api.chargerlink.com","Connection": "Keep-Alive","User-Agent": "okhttp/3.2.0",}#發(fā)起詳情請求res = requests.get(url,headers=head)price = split_n(jsonpath.jsonpath(json.loads(res.text),'$..chargingFeeDesc')[0]) #價錢payType = jsonpath.jsonpath(json.loads(res.text),'$..payTypeDesc')[0] #支付方式businessTime =split_n(jsonpath.jsonpath(json.loads(res.text),'$..businessTime')[0]) #營業(yè)時間result = (message + [price,payType,businessTime])r = ','.join([str(i) for i in result])+',n'c.write(r)def get_page():url = 'https://app-api.chargerlink.com/spot/searchSpot'head = {"device": "client=android&cityName=%E5%8C%97%E4%BA%AC%E5%B8%82&cityCode=110106&lng=116.32154281224254&device_id=8A261C9D60ACEBDED7CD3706C92DD68E&ver=3.7.7&lat=39.895024107858724&network=WIFI&os_version=19","appId": "20171010","timestamp": "1532342711477","signature": "36daaa33e7b0d5d29ac9c64a2ce6c4cf","forcecheck": "1","Content-Type": "application/x-www-form-urlencoded","Content-Length": "68","Host": "app-api.chargerlink.com","Connection": "Keep-Alive","User-Agent": "okhttp/3.2.0"}data = {"userFilter[operateType]": 2,"cityCode": 110000,"sort": 1,"page": 1,"limit": 10,}response = requests.post(url, data=data, headers=head)# 獲取數(shù)據(jù)data = response.json()total = (data["pager"]["total"])page_Size = (data["pager"]["pageSize"])totalPage = (data['pager']["totalPage"])print('當(dāng)前共有{total}個充電樁，每頁展示{page_Size}個，共{totalPage}頁'.format(total=total,page_Size=page_Size,totalPage=totalPage)) if __name__ == '__main__':get_page()start = int(input("親,請輸入您要獲取的開始頁:"))end = int(input("親,請輸入您要獲取的結(jié)束頁:"))for i in range(start,end+1):request(i)

總結(jié)

以上是生活随笔為你收集整理的okhttp post json 数据_使用python抓取App数据的全部內(nèi)容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網(wǎng)站內(nèi)容還不錯，歡迎將生活随笔推薦給好友。