1. 小說網(wǎng)站為?

https://m.wfxnews.com/

2. 分析網(wǎng)頁結(jié)構(gòu)?

下載小說的API如下：

https://www.wfxnews.com/modules/article/txtarticle.php?id=112451

通過以下網(wǎng)址，可獲得書籍信息

https://m.wfxnews.com/book/112451.shtml

?112451為這本小說的ID。 ID最小為1，最大為199959。

遍歷加多線程完成小說資源下載。

3. 源代碼

# -*- coding:utf-8 -*-import requests import time import os from threading import Threaddef get_one_page(_url):response = requests.get(_url)response.raise_for_status()response.encoding = 'gbk'return response.textdef get_txt_save_path(_url):idx = _url.rindex('=')return 'txt/' + _url[idx + 1:] + '.txt'def save_one_txt(_url):txt_save_path = get_txt_save_path(_url)# 下載TXTone_txt = get_one_page(_url)with open(txt_save_path, 'w') as f:f.write(one_txt)website = {'last_id': 199959,'first_id': 1,'download_prefix': 'https://www.wfxnews.com/modules/article/txtarticle.php?id=','info_prefix': 'https://m.wfxnews.com/book/','info_suffix': '.shtml' }first_id = website['first_id'] last_id = website['last_id'] download_prefix = website['download_prefix']cnt = 0 queue = [] for i in range(first_id, last_id + 1):# 判斷TXT是否已經(jīng)下載if os.path.exists('txt/' + str(i) + '.txt'):continueurl = download_prefix + str(i)try:th = Thread(target=save_one_txt, args=(url,))th.start()queue.append(th)cnt += 1if cnt % 16 == 0:for q in queue:q.join()queue = []except:print(url)print('cnt =', cnt)

４. 運行結(jié)果

?

總結(jié)

以上是生活随笔為你收集整理的Python爬取wfxnews 小说网站，实现批量下载小说的全部內(nèi)容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網(wǎng)站內(nèi)容還不錯，歡迎將生活随笔推薦給好友。