拉勾因其結(jié)構(gòu)化的數(shù)據(jù)比較多因此過(guò)去常常被爬，所以在其多次改版之下變得難爬。不過(guò)只要清楚它的原理，依然比較好爬。其機(jī)制主要就是AJAX異步加載JSON數(shù)據(jù)，所以至少在搜索頁(yè)面里翻頁(yè)url不會(huì)變化，而且數(shù)據(jù)也不會(huì)出現(xiàn)在源代碼里。

數(shù)據(jù)解析

這是深圳地區(qū)的數(shù)據(jù)分析師頁(yè)面，用Chrome檢查打開(kāi)。在XHR中可以看到一個(gè)以postionAjax.json開(kāi)頭的腳本，打開(kāi)Preview看一下，可以看到：

可以發(fā)現(xiàn)這些數(shù)據(jù)與前端的數(shù)據(jù)一致，此時(shí)我們已經(jīng)找到了數(shù)據(jù)入口，就可以開(kāi)始爬了。

數(shù)據(jù)爬取

在Headers里可以查看請(qǐng)求方式：

Request Header:

Request URL:https://www.lagou.com/jobs/positionAjax.json?city=深圳&needAddtionalResult=false

Request Method:POST

Status Code:200 OK

Remote Address:106.75.72.62:443

從Request Header中可以看到，是用POST提交表單方式查詢的（所以如果你直接點(diǎn)開(kāi)Request URL你會(huì)發(fā)現(xiàn)數(shù)據(jù)不對(duì)，因?yàn)闆](méi)有提交表單數(shù)據(jù)）。

那么我們就可以在Python中構(gòu)造請(qǐng)求頭以及提交表單數(shù)據(jù)來(lái)訪問(wèn)：

import requests

import time

from sqlalchemy import create_engine

import pandas as pd

from random import choice

import json

import numpy

engine=create_engine(#這里填你自己數(shù)據(jù)庫(kù)的參數(shù)#) # 連接數(shù)據(jù)庫(kù)

dl = pd.read_sql("proxys",engine)

def get_proxy(dl):

    n = choice(range(1, len(dl.index)))

    proxy = {"http":"http://%s:%s" %(dl["ip"][n],dl["port"][n]),

             "https": "http://%s:%s" % (dl["ip"][n], dl["port"][n])}

    return(proxy)

def get_header():

    headers = {

        "User-Agent": ""Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; AcooBrowser; .NET CLR 1.1.4322; .NET CLR 2.0.50727)"",

        "Accept": "application/json, text/javascript, */*; q=0.01",

        "Content-Type": "application/x-www-form-urlencoded; charset=UTF-8",

        "Referer": "https://www.lagou.com/jobs/list_%E6%95%B0%E6%8D%AE%E5%88%86%E6%9E%90%E5%B8%88?px=default&city=%E6%B7%B1%E5%9C%B3&district=%E5%8D%97%E5%B1%B1%E5%8C%BA",

        "X-Requested-With": "XMLHttpRequest",

        "Host": "www.lagou.com",

        "Connection":"keep-alive",

        "Cookie":"user_trace_token=20160214102121-0be42521e365477ba08bd330fd2c9c72; LGUID=20160214102122-a3b749ae-d2c1-11e5-8a48-525400f775ce; tencentSig=9579373568; pgv_pvi=3712577536; index_location_city=%E5%85%A8%E5%9B%BD; SEARCH_ID=c684c55390a84fe5bd7b62bf1754b900; JSESSIONID=8C779B1311176D4D6B74AF3CE40CE5F2; TG-TRACK-CODE=index_hotjob; Hm_lvt_4233e74dff0ae5bd0a3d81c6ccf756e6=1485318435,1485338972,1485393674,1485423558; Hm_lpvt_4233e74dff0ae5bd0a3d81c6ccf756e6=1485423598; _ga=GA1.2.1996921784.1455416480; LGRID=20170126174002-691cb0a5-e3ab-11e6-bdc0-525400f775ce",

        "Origin": "https://www.lagou.com",

        "Upgrade-Insecure-Requests":"1",

        "X-Anit-Forge-Code": "0",

        "X-Anit-Forge-Token": "None",

        "Accept-Encoding": "gzip, deflate, br",

        "Accept-Language": "zh-CN,zh;q=0.8"

        }

    return(headers)

def get_form(i):

    data={"first":"false","pn":i,"kd":"數(shù)據(jù)分析師"}

    return(data)

districts = ["南山區(qū)","福田區(qū)","寶安區(qū)","龍崗區(qū)","龍華新區(qū)","羅湖區(qū)","鹽田區(qū)","大鵬新區(qū)"]

pagenos = [22,10,1,4,1,2,1,1]

url_lists = ["https://www.lagou.com/jobs/positionAjax.json?px=default&city=深圳&district=%s&needAddtionalResult=false"%area for area in districts]

s = requests.Session()

s.keep_alive = False

s.adapters.DEFAULT_RETRIES = 10

def get_jobinfo(i,j): # i表區(qū)號(hào)，j表頁(yè)數(shù)

    if i >= 8 or j > pagenos[i]:

        return("索引超標(biāo)！")

    resp=s.post(url_lists[i], data=get_form(j), headers=get_header())

    resp.encoding="utf-8"

    max_num = len(json.loads(resp.text)["content"]["positionResult"]["result"])

    for k in range(max_num):

        try:

            json_data=json.loads(resp.text)["content"]["positionResult"]["result"][k]

            df = pd.DataFrame(dict(

                approve=json_data["approve"],

        #        businessZones=json_data["businessZones"],

                companyId=json_data["companyId"],

        #        companyLabelList=json_data["companyLabelList"],

                companyShortName=json_data["companyShortName"],

                companySize=json_data["companySize"],

                createTime=json_data["createTime"],

                education=json_data["education"],

                financeStage=json_data["financeStage"],

                firstType=json_data["firstType"],

                industryField=json_data["industryField"],

                jobNature=json_data["jobNature"],

                positionAdvantage=json_data["positionAdvantage"],

                positionId=json_data["positionId"],

                positionName=json_data["positionName"],

                salary=json_data["salary"],

                secondType=json_data["secondType"],

                workYear=json_data["workYear"],

                scrapy_time=time.strftime('%Y-%m-%d %H:%M:%S', time.localtime(time.time()))),index=[0])

            df.to_sql(con = engine, name = "job_info", if_exists = 'append', flavor = "mysql",index=False)

        except:

            print("第%d區(qū)，第%d頁(yè)，第%d個(gè)出錯(cuò)了！"%(i,j,k))

以上這個(gè)函數(shù)就可以通過(guò)提交區(qū)和頁(yè)數(shù)，返回當(dāng)前頁(yè)的職位數(shù)。

其實(shí)AJAX返回JSON數(shù)據(jù)的方式也有好處，數(shù)據(jù)都是規(guī)整的，不必花太多時(shí)間精力在數(shù)據(jù)清洗上。

不過(guò)注意要加延時(shí)，拉勾的反爬蟲(chóng)措施還是比較嚴(yán)的，不加延時(shí)爬一小會(huì)兒就會(huì)被封IP。

總結(jié)

以上是生活随笔為你收集整理的Python拉勾爬虫——以深圳地区数据分析师为例的全部?jī)?nèi)容，希望文章能夠幫你解決所遇到的問(wèn)題。

如果覺(jué)得生活随笔網(wǎng)站內(nèi)容還不錯(cuò)，歡迎將生活随笔推薦給好友。