1. 目標(biāo)網(wǎng)站：89文學(xué)網(wǎng)（https://www.89wxw.com）
2. python環(huán)境：IDLE

#基本思路：采用requests庫(kù)和BeautifulSoup庫(kù)爬取龍王傳說(shuō)小說(shuō)import requests from bs4 import BeautifulSoupdef get_novel_chapters():#獲得標(biāo)簽名稱及鏈接data_1 = [] root_url = 'https://www.89wxw.com/read/3440/'headers = {'user-agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36'}#用headers對(duì)python進(jìn)行偽裝r = requests.get(url = root_url, headers = headers)soup = BeautifulSoup(r.text,'html.parser')infos = soup.find_all('dd')for info in infos:link = info.find('a')['href']link =str( 'https://www.89wxw.com'+link)chapter = str(info.get_text())data_1.append((link,chapter)) #print(data_1)#寫代碼階段用于檢查程序return data_1def get_chapter_content(url):#進(jìn)入標(biāo)簽背后的鏈接，進(jìn)行爬取data_2 = []headers = {'user-agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36'}#用headers對(duì)python進(jìn)行偽裝r = requests.get(url = url, headers = headers)soup = BeautifulSoup(r.text,'html.parser')contents = soup.find('div', id = 'content').find_all('p')for content in contents:#獲取全部文本內(nèi)容content = content.get_text()data_2.append(content)return data_2 def download_txt(name,content):#下載函數(shù)f = open('%s.txt'%name,'w')#文件以標(biāo)簽名命名f.write(content)f.close()for item in get_novel_chapters():link,name = itemcontent = str(get_chapter_content(link))download_txt(name,content)

附上成功圖

總結(jié)

以上是生活随笔為你收集整理的用python抓取小说，以龙王传说为例的全部?jī)?nèi)容，希望文章能夠幫你解決所遇到的問(wèn)題。

如果覺得生活随笔網(wǎng)站內(nèi)容還不錯(cuò)，歡迎將生活随笔推薦給好友。