我有蜘蛛使用meta處理鏈中的請求,以產生具有來自多個請求的數據的項目.

我用來生成請求的方式是在首次調用parse函數時啟動所有請求,但是,如果我有太多鏈接無法請求,則并不是所有請求都已計劃好,最終我也無法獲得所需的一切.

為了解決這個問題,我正在嘗試使Spider一次請求5個產品,然后在Spider空閑時再次請求(通過在from_crawler中連接信號).

問題是,由于我的代碼現在正確,spider_idle沒有運行請求函數,并且蜘蛛立即關閉.好像蜘蛛不會閑置.

這是一些代碼：

class ProductSpider(scrapy.Spider):

def __init__(self, *args, **kwargs):

super(ProductSpider, self).__init__(*args, **kwargs)

self.parsed_data = []

self.header = {}

f = open('file.csv', 'r')

f_data = [[x.strip()] for x in f]

count=1

first = 'smth'

for product in f_data:

if first != '':

header = product[0].split(';')

for each in range(len(header[1:])):

self.header[header[each+1]] = each+1

first = ''

else:

product = product[0].split(';')

product.append(count)

count+=1

self.parsed_data.append(product)

f.close()

@classmethod

def from_crawler(cls, crawler, *args, **kwargs):

spider = super(ProductSpider, cls).from_crawler(crawler, *args, **kwargs)

crawler.signals.connect(spider.request, signal=signals.spider_idle)

return spider

name = 'products'

allowed_domains = [domains]

handle_httpstatus_list = [400, 404, 403, 503, 504]

start_urls = [start]

def next_link(self,response):

product = response.meta['product']

there_is_next = False

for each in range(response.meta['each']+1, len(product)-1):

if product[each] != '':

there_is_next = True

yield scrapy.Request(product[each], callback=response.meta['func_dict'][each], meta={'func_dict': response.meta['func_dict'],'product':product,'each':each,'price_dict':response.meta['price_dict'], 'item':response.meta['item']}, dont_filter=True)

break

if not there_is_next:

item = response.meta['item']

item['prices'] = response.meta['price_dict']

yield item

#[...] chain parsing functions for each request

def get_products(self):

products = []

data = self.parsed_data

for each in range(5):

if data:

products.append(data.pop())

return products

def request(self):

item = Header()

item['first'] = True

item['sellers'] = self.header

yield item

func_dict = {parsing_functions_for_every_site}

products = self.get_products()

if not products:

return

for product in products:

item = Product()

price_dict = {1:product[1]}

item['name'] = product[0]

item['order'] = product[-1]

for each in range(2, len(product)-1):

if product[each] != '':

#print each, func_dict, product[each]

yield scrapy.Request(product[each], callback=func_dict[each],

meta={'func_dict': func_dict,'product':product,

'each':each,'price_dict':price_dict, 'item':item})

break

raise DontCloseSpider

def parse(self, response=None):

pass

總結

以上是生活随笔為你收集整理的python绿色版spider的idle_Python-Scrapy Spider没有收到spider_idle信号的全部內容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網站內容還不錯，歡迎將生活随笔推薦給好友。