我需要從命令行中獲取一個自變量(-a FILE_NAME =“ stuff”),并將其應(yīng)用于在pipeline.py文件中由我的CSVWriterPipeLine創(chuàng)建的文件. (我之所以使用pipeline.py是因為內(nèi)置的導(dǎo)出器正在重復(fù)數(shù)據(jù)并在輸出文件中重復(fù)標(biāo)題.相同的代碼,但是在管道中進行寫入修復(fù)了它.)

我嘗試從scrapy.utils.project導(dǎo)入get_project_settings中看到

但是我無法從命令行更改文件名.

我還嘗試實現(xiàn)頁面上的@avaleske解決方案,因為它專門解決了這個問題,但是我不知道他談?wù)摰拇a在我的scrapy文件夾中的位置.

救命？

settings.py：

BOT_NAME = 'internal_links'

SPIDER_MODULES = ['internal_links.spiders']

NEWSPIDER_MODULE = 'internal_links.spiders'

CLOSESPIDER_PAGECOUNT = 100

ITEM_PIPELINES = ['internal_links.pipelines.CsvWriterPipeline']

# Crawl responsibly by identifying yourself (and your website) on the user-agent

USER_AGENT = 'internal_links (+http://www.mycompany.com)'

FILE_NAME = "mytestfilename"

pipelines.py：

import csv

class CsvWriterPipeline(object):

def __init__(self, file_name):

header = ["URL"]

self.file_name = file_name

self.csvwriter = csv.writer(open(self.file_name, 'wb'))

self.csvwriter.writerow(header)

def process_item(self, item, internallinkspider):

# build your row to export, then export the row

row = [item['url']]

self.csvwriter.writerow(row)

return item

spider.py：

from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor

from scrapy.contrib.spiders import CrawlSpider, Rule

from internal_links.items import MyItem

class MySpider(CrawlSpider):

name = 'internallinkspider'

allowed_domains = ['angieslist.com']

start_urls = ['http://www.angieslist.com']

rules = (Rule(SgmlLinkExtractor(), callback='parse_url', follow=True), )

def parse_url(self, response):

item = MyItem()

item['url'] = response.url

return item

總結(jié)

以上是生活随笔為你收集整理的scrapy的name变量_python-将file_name参数传递给管道以在scrapy中...的全部內(nèi)容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網(wǎng)站內(nèi)容還不錯，歡迎將生活随笔推薦給好友。