1 定義抓取內容

在items.py中添加如下代碼：

class Job51Item(scrapy.Item):title = scrapy.Field()salary = scrapy.Field()job_city = scrapy.Field()requirement = scrapy.Field() # 經驗、學歷、人數company_name = scrapy.Field()company_size = scrapy.Field()publish_date = scrapy.Field()job_advantage_tags = scrapy.Field() # 福利url = scrapy.Field()

2 編寫爬蟲

由于51job的反爬蟲機制，其招聘信息數據并不是如游覽器一般直接展示出來，而是隱藏在script標簽內

在spiders文件夾新建文件job_51.py，并編寫如下代碼：

# -*- coding: utf-8 -*- import json import re import scrapy from scrapy.http import Requestfrom job.items import Job51Itemclass Job51Spider(scrapy.Spider):name = 'job51'allowed_domains = ['jobs.51job.com', 'search.51job.com']start_urls = ['https://search.51job.com/list/090200,000000,0000,00,9,99,python,2,''3.html?lang=c&postchannel=0000&workyear=99&cotype=99&degreefrom=99&jobterm=99&companysize=99&ord_field=0''&dibiaoid=0&line=&welfare=']def parse(self, response):body = response.body.decode("gbk")with open("b.html","w",encoding="utf-8") as f:f.write(body)data = re.findall('window.__SEARCH_RESULT__ =(.+)}</script>', str(body))[0] + "}"data = json.loads(data)item = Job51Item()for result in data["engine_search_result"]:item["requirement"] = result["attribute_text"]item["url"] = result["job_href"]item["title"] = result["job_name"]item["salary"] = result["issuedate"]item["job_city"] = result["workarea_text"]item["publish_date"] = result["issuedate"]item["job_advantage_tags"] = result["jobwelf"]item["company_name"] = result["company_name"]item["company_size"] = result["companysize_text"]yield itemfor i in range(2, 10):url = f"https://search.51job.com/list/090200,000000,0000,00,9,99,python,2,{i}.html?lang=c&postchannel" f"=0000&workyear=99&cotype=99&degreefrom=99&jobterm=99&companysize=99&ord_field=0&dibiaoid=0&line" f"=&welfare= "yield Request(url=url, callback=self.parse, )

3 定義管道

3.1 啟用pipelines

在settings.py中添加如下代碼：

ITEM_PIPELINES = {'job.pipelines.JobPipeline': 300, }

注：或者取消該部分內容注釋

3.2 編寫管道代碼

在pipelines.py編寫如下代碼：

import jsonfrom itemadapter import ItemAdapterclass JobPipeline:def __init__(self):self.fp = open('result.json', 'w', encoding='utf-8') def open_spider(self, spider):self.fp.write("[")def process_item(self, item, spider):data = json.dumps(dict(item), ensure_ascii=False)self.fp.write(data+',n')return itemdef close_spider(self, spider):self.fp.write("]")self.fp.close()print("spider end")

與50位技術專家面對面20年技術見證，附贈技術全景圖

總結

以上是生活随笔為你收集整理的怎么防爬虫爬取信息_scrapy爬取51job职位信息（针对新的反爬虫机制）！的全部內容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網站內容還不錯，歡迎將生活随笔推薦給好友。