python之requests爬蟲(chóng)Boss數(shù)據(jù)

需要用到的庫(kù)：reqeusts、lxml

沒(méi)有的可以用直接下載

pip install requests

pip install lxm

這里以python崗位，地點(diǎn)北京為例

爬取的數(shù)據(jù)就是崗位名稱、薪資、地點(diǎn)

首先導(dǎo)入需要用到的模塊

import requests

from lxml import etree

崗位可以通過(guò)input提前輸入好，傳參給url

job = input('輸入職位')

將需要訪問(wèn)的url賦給一個(gè)變量

url = 'https://www.zhipin.com/job_detail/?query=%s&city=101010100&industry=&position='%job

query=%s(%s是一個(gè)占位)在引號(hào)后面的%job就是占的值

訪問(wèn)這個(gè)頁(yè)面需要加一個(gè)頭部(headers)降低被識(shí)別爬蟲(chóng)的概率

在當(dāng)前頁(yè)面按f12點(diǎn)擊Network，如果沒(méi)有東西的話可以刷新一下頁(yè)面

這里只用到兩個(gè)參數(shù)，一個(gè)user-agent，一個(gè)cookie

headers = {

'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36',

'cookie': '_uab_collina=157853739340991408682799; Hm_lvt_194df3105ad7148dcf2b98a91b5e727a=1578537393,1578554153; __c=1578554153; __g=-; Hm_lpvt_194df3105ad7148dcf2b98a91b5e727a=1578554168; __zp_stoken__=d0e7eq77rh2ql3R%2F5VwP4mPjHKu%2BjYVQMbIFSPnpEWipSXfKaWf%2FM%2FxBRat22vE%2FR4PdiD%2BDhDiSNaW%2FTjVMpYOEMTTUmxg7WSFqYfpdWi5SSIMEcHuwoKbmd%2B6tlv5ONmSF; __l=l=%2Fwww.zhipin.com%2Fjob_detail%2F%3Fquery%3D%25E4%25BA%25BA%25E5%25B7%25A5%25E6%2599%25BA%25E8%2583%25BD%26city%3D101010100%26industry%3D%26position%3D&r=&friend_source=0&friend_source=0; __a=32343010.1578537387.1578537387.1578554153.9.2.2.9'

}

請(qǐng)求數(shù)據(jù)并且返回值

res = requests.get(url,headers=headers).text

利用etree中的HTML解析數(shù)據(jù)

html = etree.HTML(res)

#崗位名稱

job_name = html.xpath('//*[@id="main"]/div/div[2]/ul/li/div/div[1]/h3/a/div[1]/text()')

#薪資

salary = html.xpath('//*[@id="main"]//ul/li//h3/a/span/text()')

#地點(diǎn)、工作經(jīng)驗(yàn)、學(xué)歷

site = html.xpath('//*[@id="main"]/div/div[2]/ul/li/div/div[1]/p')

print('工作崗位:',job)

print('薪資:',salary)

print('地點(diǎn)：',site)

輸出的數(shù)據(jù)如下

整體代碼

import requests

from lxml import etree

import json

job = input('輸入職位')

url = 'https://www.zhipin.com/job_detail/?query=%s&city=101010100&industry=&position='%job

headers = {

'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36',

'cookie': '_uab_collina=157853739340991408682799; Hm_lvt_194df3105ad7148dcf2b98a91b5e727a=1578537393,1578554153; __c=1578554153; __g=-; __l=l=%2Fwww.zhipin.com%2Fjob_detail%2F%3Fquery%3D%25E4%25BA%25BA%25E5%25B7%25A5%25E6%2599%25BA%25E8%2583%25BD%26city%3D101010100%26industry%3D%26position%3D&r=&friend_source=0&friend_source=0; lastCity=101010100; Hm_lpvt_194df3105ad7148dcf2b98a91b5e727a=1578556532; __zp_stoken__=d0e7eq77rh2ql3R%2F5VwP4mPjHOT%2BY0u%2F2GMG6hriOPZlx6iA6NPb%2FycP1M1RRJxkLq%2FdiD%2BDhDiSNaW%2FTjVMpYOEMScFTSjVVO31G%2B8%2Bwf%2Bxs7gEcHuwoKbmd%2B6tlv5ONmSF; __a=32343010.1578537387.1578537387.1578554153.29.2.22.29'

}

res = requests.get(url,headers=headers).text

html = etree.HTML(res)

job_name = html.xpath('//*[@id="main"]/div/div[2]/ul/li/div/div[1]/h3/a/div[1]/text()')

salary = html.xpath('//*[@id="main"]//ul/li//h3/a/span/text()')

site = html.xpath('//*[@id="main"]/div/div[2]/ul/li/div/div[1]/p/text()')

print('工作崗位:',job_name)

print('薪資:',salary)

print('地點(diǎn)：',site)

最后需要注意網(wǎng)站中的cookie是實(shí)時(shí)更新，如果數(shù)據(jù)沒(méi)有出來(lái)再去網(wǎng)頁(yè)中查看cookie值

總結(jié)

以上是生活随笔為你收集整理的python爬boss网站_python之requests爬虫Boss数据的全部?jī)?nèi)容，希望文章能夠幫你解決所遇到的問(wèn)題。

如果覺(jué)得生活随笔網(wǎng)站內(nèi)容還不錯(cuò)，歡迎將生活随笔推薦給好友。