當(dāng)前位置：首頁 > 编程资源 > 编程问答 >内容正文

编程问答

Scrapy爬取豆瓣小组图片

發(fā)布時(shí)間：2024/1/1 编程问答 39 豆豆

生活随笔收集整理的這篇文章主要介紹了 Scrapy爬取豆瓣小组图片小編覺得挺不錯(cuò)的,現(xiàn)在分享給大家,幫大家做個(gè)參考.

2019獨(dú)角獸企業(yè)重金招聘Python工程師標(biāo)準(zhǔn)>>>

狗年開工毫無工作心情，胡思亂想后決定爬取豆瓣上的一下信息打發(fā)時(shí)間，畢竟之前基本沒接觸過爬蟲，還是挺感興趣的。

Scrapy簡介

首先簡單介紹一下Scrapy爬蟲框架，主要是架構(gòu)方面，這方面能快速理解scrapy是如何工作的。

Scrapy的數(shù)據(jù)流由執(zhí)行引擎（Engine）控制，其基本過程如下：

引擎從Spider中獲取到初始Requests。

引擎將該Requests放入調(diào)度器，并請(qǐng)求下一個(gè)要爬取的Requests。

調(diào)度器返回下一個(gè)要爬取的Requests給引擎

引擎將Requests通過下載器中間件轉(zhuǎn)發(fā)給下載器(Downloader)。

一旦頁面下載完畢，下載器生成一個(gè)該頁面的Response，并將其通過下載中間件(返回(response)方向)發(fā)送給引擎。

引擎從下載器中接收到Response并通過Spider中間件(輸入方向)發(fā)送給Spider處理。

Spider處理Response并返回爬取到的Item及(跟進(jìn)的)新的Request給引擎。

引擎將(Spider返回的)爬取到的Item交給ItemPipeline處理，將(Spider返回的)Request交給調(diào)度器，并請(qǐng)求下一個(gè)Requests（如果存在的話）。

(從第一步)重復(fù)直到調(diào)度器中沒有更多地Request。

使用Scrapy創(chuàng)建的項(xiàng)目架構(gòu)如下

其中：

spider 文件夾編寫自己的爬蟲；

settings.py 配置爬蟲的默認(rèn)信息，功能開關(guān)，中間件執(zhí)行順序等；

middlewares.py 中間件，主要是對(duì)功能的拓展，添加自定義功能，比如user-agent和proxy

item.py 定義抓取處理的字段

piplines.py 管道文件，處理item

爬取豆瓣小組

豆瓣小組的帖子主要核心內(nèi)容是圖片，因此要按不同的帖子分類下載。

settings.py

設(shè)置了user-agent，指定了中間件和piplines

BOT_NAME = 'douban'SPIDER_MODULES = ['douban.spiders'] NEWSPIDER_MODULE = 'douban.spiders'# Crawl responsibly by identifying yourself (and your website) on the user-agent #USER_AGENT = 'douban (+http://www.yourdomain.com)'# Obey robots.txt rules ROBOTSTXT_OBEY = FalseMY_USER_AGENT = ["Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; AcooBrowser; .NET CLR 1.1.4322; .NET CLR 2.0.50727)","Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; Acoo Browser; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.0.04506)","Mozilla/4.0 (compatible; MSIE 7.0; AOL 9.5; AOLBuild 4337.35; Windows NT 5.1; .NET CLR 1.1.4322; .NET CLR 2.0.50727)","Mozilla/5.0 (Windows; U; MSIE 9.0; Windows NT 9.0; en-US)","Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Win64; x64; Trident/5.0; .NET CLR 3.5.30729; .NET CLR 3.0.30729; .NET CLR 2.0.50727; Media Center PC 6.0)","Mozilla/5.0 (compatible; MSIE 8.0; Windows NT 6.0; Trident/4.0; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; .NET CLR 1.0.3705; .NET CLR 1.1.4322)","Mozilla/4.0 (compatible; MSIE 7.0b; Windows NT 5.2; .NET CLR 1.1.4322; .NET CLR 2.0.50727; InfoPath.2; .NET CLR 3.0.04506.30)","Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN) AppleWebKit/523.15 (KHTML, like Gecko, Safari/419.3) Arora/0.3 (Change: 287 c9dfb30)","Mozilla/5.0 (X11; U; Linux; en-US) AppleWebKit/527+ (KHTML, like Gecko, Safari/419.3) Arora/0.6","Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.2pre) Gecko/20070215 K-Ninja/2.1.1","Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; rv:1.9) Gecko/20080705 Firefox/3.0 Kapiko/3.0","Mozilla/5.0 (X11; Linux i686; U;) Gecko/20070322 Kazehakase/0.4.5","Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.8) Gecko Fedora/1.9.0.8-1.fc10 Kazehakase/0.5.6","Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.56 Safari/535.11","Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_3) AppleWebKit/535.20 (KHTML, like Gecko) Chrome/19.0.1036.7 Safari/535.20","Opera/9.80 (Macintosh; Intel Mac OS X 10.6.8; U; fr) Presto/2.9.168 Version/11.52","Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.11 (KHTML, like Gecko) Chrome/20.0.1132.11 TaoBrowser/2.0 Safari/536.11","Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.71 Safari/537.1 LBBROWSER","Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E; LBBROWSER)","Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; QQDownload 732; .NET4.0C; .NET4.0E; LBBROWSER)","Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.84 Safari/535.11 LBBROWSER","Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; WOW64; Trident/5.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E)","Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E; QQBrowser/7.0.3698.400)","Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; QQDownload 732; .NET4.0C; .NET4.0E)","Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Trident/4.0; SV1; QQDownload 732; .NET4.0C; .NET4.0E; 360SE)","Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; QQDownload 732; .NET4.0C; .NET4.0E)","Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; WOW64; Trident/5.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E)","Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1","Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1","Mozilla/5.0 (iPad; U; CPU OS 4_2_1 like Mac OS X; zh-cn) AppleWebKit/533.17.9 (KHTML, like Gecko) Version/5.0.2 Mobile/8C148 Safari/6533.18.5","Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:2.0b13pre) Gecko/20110307 Firefox/4.0b13pre","Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:16.0) Gecko/20100101 Firefox/16.0","Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.64 Safari/537.11","Mozilla/5.0 (X11; U; Linux x86_64; zh-CN; rv:1.9.2.10) Gecko/20100922 Ubuntu/10.10 (maverick) Firefox/3.6.10","Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36",] DOWNLOADER_MIDDLEWARES = {'scrapy.downloadermiddleware.useragent.UserAgentMiddleware': None,'douban.middlewares.MyUserAgentMiddleware': 400, } COOKIES_ENABLES = True DOWNLOAD_DELAY=1 ITEM_PIPELINES = {'douban.pipelines.DoubanPipeline': 1, }

item.py

定義字段，包括作者，帖子名稱，作者主頁地址，圖片地址

class DoubanItem(scrapy.Item):# define the fields for your item here like:# name = scrapy.Field()title=scrapy.Field()author=scrapy.Field()author_homepage=scrapy.Field()img_url=scrapy.Field()pass

middlewares.py

設(shè)置user-agent

class MyUserAgentMiddleware(UserAgentMiddleware):'''設(shè)置User-Agent'''def __init__(self, user_agent, ip):self.user_agent = user_agentself.ip=ip@classmethoddef from_crawler(cls, crawler):return cls(user_agent=crawler.settings.get('MY_USER_AGENT'), ip=crawler.settings.get('PROXIES'))def process_request(self, request, spider):agent = random.choice(self.user_agent)request.headers['User-Agent'] = agent

spiders/douban_spider.py

爬蟲的處理代碼，先登錄然后爬取，如果有驗(yàn)證碼，下載圖片然后輸入驗(yàn)證碼

import urllibimport scrapy from scrapy import Request, FormRequestfrom douban.items import DoubanItem import jsonclass DoubanSpider(scrapy.Spider):name = 'douban'allowed_domains = ['douban.com']start_urls = []def start_requests(self):yield Request("https://www.douban.com/login", callback=self.parse, meta={"cookiejar":1})def parse(self, response):captcha = response.xpath('//img[@id="captcha_image"]/@src').extract()if len(captcha)>0:print("此時(shí)有驗(yàn)證碼")localpath = "E:/spider/douban/captchar.jpg"urllib.request.urlretrieve(captcha[0],filename=localpath)print("請(qǐng)查看本地驗(yàn)證碼圖片并輸入驗(yàn)證碼")captcha_value=input()data = {"form_email": "*******@126.com","form_password": "*******","captcha-solution": str(captcha_value),"redir": "https://www.douban.com/group/haixiuzu/discussion?start=0" # 登錄后要返回的頁面}else:print("此時(shí)沒有驗(yàn)證碼")data = {"form_email": "nofree1990@126.com","form_password": "8296926",# "redir": "https://www.douban.com/group/haixiuzu/discussion?start=0" # 登錄后要返回的頁面}print("登陸中...")yield FormRequest.from_response(response,meta={"cookiejar": response.meta["cookiejar"]}, formdata=data, callback=self.parse_redirect)def parse_redirect(self, response):print("已登錄豆瓣")title = response.xpath('//title//text()').extract()baseurl='https://www.douban.com/group/haixiuzu/discussion?start='for i in range(0, 625, 25):pageUrl=baseurl+str(i)yield Request(url=pageUrl, callback=self.parse_process,dont_filter = True)def parse_process(self, response):title = response.xpath('//title//text()').extract()items = response.xpath('//td//a/@href').extract()for item in items:if 'topic' in item:url=itemyield Request(url=item,callback=self.parse_img)def parse_img(self,response):img = DoubanItem()title=response.xpath('//title//text()').extract()img['title']=titleauthor=response.xpath('//div[@class="topic-doc"]//h3//a//text()').extract()img['author']=authorauthor_homepage = response.xpath('//div[@class="topic-doc"]//h3//a/@href').extract()img['author_homepage'] = author_homepageimg_url = response.xpath('//div[@class="image-wrapper"]//img/@src').extract()img['img_url'] = img_urlyield img

piplines.py

在此保存帖子信息，沒有使用自帶的保存圖片的類主要原因是不夠靈活。

class DoubanPipeline(object):def process_item(self, item, spider):author=item["author"][0]title=item["title"][0].replace('\n','').strip()author_homepage=item["author_homepage"][0]#路徑dir="E:/spider/douban/img/"if not os.path.exists(dir):os.mkdir(dir)author_dir=dir+titleif not os.path.exists(author_dir):os.mkdir(author_dir)#用戶信息txtinfo=open(author_dir+"/用戶信息.txt", "w")info.write(author+'\n'+author_homepage)info.close()#保存圖片count=1for url in item["img_url"]:path=author_dir+"/"+str(count)+".jpg"urllib.request.urlretrieve(url, filename=path)count += 1return item

爬蟲結(jié)果

遇到的問題

主要問題就是爬取太頻繁而被禁止，登錄豆瓣也是想減少被禁止概率，但是發(fā)現(xiàn)沒什么用。網(wǎng)上有很多解決方案，還是要偽造一些user-agent，使用proxy代理。也爬取過一些proxy存到數(shù)據(jù)庫中，但是proxy比較慢，遂放棄。

轉(zhuǎn)載于:https://my.oschina.net/M97hQfT2SJQ/blog/1629257

總結(jié)

以上是生活随笔為你收集整理的Scrapy爬取豆瓣小组图片的全部內(nèi)容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網(wǎng)站內(nèi)容還不錯(cuò)，歡迎將生活随笔推薦給好友。