單獨爬蟲配置 custom_settings = {'SOME_SETTING': 'some value',} 不同爬蟲pipeline設置 custom_settings = {'ITEM_PIPELINES': {'video.pipelines.VideoPipeline': 301,} }

cookie設置

custom_settings = {'COOKIES_ENABLED':True, # 在配置文件settings中可以設置成False, 在這個spider中這樣設置就可以開啟cookies了，其他的配置一樣適用}

?

settings/在settings同目錄下新建custom_settings.py
?

# -*- coding: utf-8 -*- custom_settings_for_spider1 = { 'LOG_LEVEL': 'INFO', 'DOWNLOAD_DELAY': 0, 'COOKIES_ENABLED': False, # enabled by default 'DOWNLOADER_MIDDLEWARES': { 'video_spider.middlewares.ProxiesMiddleware': 400, 'video_spider.middlewares.SeleniumMiddleware': 543, # 將scrapy默認的user-agent中間件關閉 12 'scrapy.downloadmiddlewares.useragent.UserAgentMiddleware': None, }, }

在spider文件中引入custom_settings
?

import scrapy from scrapy import Request from scrapy.utils.project import get_project_settings from scrapy import signals from pydispatch import dispatcher # setting class ShanbaySpider(scrapy.Spider): name = 'shanbay' allowed_domains = ['shanbay.com'] start_urls = ['http://shanbay.com/'] custom_settings = custom_settings_for_spider1

?

總結

以上是生活随笔為你收集整理的scrapy custom_settings的全部內(nèi)容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網(wǎng)站內(nèi)容還不錯，歡迎將生活随笔推薦給好友。