python爬蟲之爬取起點中文網小說

hello大家好，這篇文章帶大家來制作一個python爬蟲爬取閱文集團旗下產品起點中文網的程序，這篇文章的靈感來源于本人制作的一個項目：電腦助手 啟帆助手

?是項目的部分源碼

準備工作

用到的庫有：

• urllib.request
• lxml.etree

代碼分析

第一步：導入要用到的庫

from urllib import request from lxml import etree

2.第二步:設置請求頭及小說網址(這里的網址以作者寫的一本為例)

header = {'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36'} url="https://book.qidian.com/info/1020546097"

3.第三步：爬取每個章節的鏈接、標題，并解析

req = request.Request(url, headers=header) html = request.urlopen(req).read().decode('utf-8') html = etree.HTML(html) Lit_tit_list = html.xpath('//ul[@class="cf"]/li/a/text()') #爬取每個章節名字 Lit_href_list = html.xpath('//ul[@class="cf"]/li/a/@href') #每個章節鏈接 # print(Lit_tit_list) # print(Lit_href_list)

4.第四步:抓取文章并用text保存至電腦

for tit,src in zip(Lit_tit_list,Lit_href_list):url = "http:" + srcreq = request.Request(url, headers=header)html = request.urlopen(req).read().decode('utf-8')html = etree.HTML(html)text_list = html.xpath('//div[@class="read-content j_readContent"]/p/text()')text = "\n".join(text_list)file_name = tit + ".txt"print("正在抓取文章：" + file_name)with open(file_name, 'a', encoding="utf-8") as f:f.write("\t" + tit + '\n' + text)

完整代碼

from urllib import request from lxml import etree header = {'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36'} url="https://book.qidian.com/info/1020546097" req = request.Request(url, headers=header) html = request.urlopen(req).read().decode('utf-8') html = etree.HTML(html) Lit_tit_list = html.xpath('//ul[@class="cf"]/li/a/text()') #爬取每個章節名字 Lit_href_list = html.xpath('//ul[@class="cf"]/li/a/@href') #每個章節鏈接 # print(Lit_tit_list) # print(Lit_href_list) for tit,src in zip(Lit_tit_list,Lit_href_list):url = "http:" + srcreq = request.Request(url, headers=header)html = request.urlopen(req).read().decode('utf-8')html = etree.HTML(html)text_list = html.xpath('//div[@class="read-content j_readContent"]/p/text()')text = "\n".join(text_list)file_name = tit + ".txt"print("正在抓取文章：" + file_name)with open(file_name, 'a', encoding="utf-8") as f:f.write("\t" + tit + '\n' + text)

效果展示

以下就是爬取的txt文件啦：

好啦，這篇文章就到這里吧，白······

總結

以上是生活随笔為你收集整理的python爬虫之爬取起点中文网小说的全部內容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網站內容還不錯，歡迎將生活随笔推薦給好友。