1. 簡介

去年用C#給自己博客寫的每日新聞爬蟲突然就不能用了，最近閑下來看了一下，原來是百度熱搜榜的前端頁面改版了，那難怪。這次索性人生苦短，我選Python吧。

2. 百度熱搜榜源碼觀察

百度熱搜榜的網(wǎng)址如下：
百度熱搜榜
去了點開源碼一看，我樂了。
百度很貼心的在最前面用注釋寫好了熱搜榜內容的數(shù)據(jù)字典，也不知道是后端程序員生成出來忘記刪了，還是真就方便大家爬呢。

那么接下來就好辦了。

3. 獲取網(wǎng)頁html源碼

使用python的urllib.request包，我就直接上代碼了，大家看吧

#獲取網(wǎng)頁源碼 def askURL():print('#>>>獲取網(wǎng)頁HTML源碼中...>>>')# 模擬瀏覽器頭部信息，向服務器發(fā)送消息head={"User-Agent": "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36"}# 用戶代理，表示告訴豆瓣服務器，我們是什么類型的機器、瀏覽器（本質上是告訴瀏覽器，我們可以接收什么水平的文件內容）request=urllib.request.Request(newsUrl,headers=head)html=""try:response = urllib.request.urlopen(request)html = response.read().decode("utf-8")print("#SUCCESS>HTML源碼爬取完畢！\n")except urllib.error.URLError as e:print("#ERROR>HTML源碼爬取失敗！\n")if hasattr(e, "code"):print("#" + str(e.code))if hasattr(e, "reason"):print("#" + str(e.reason))return html

4. 整理源碼中的數(shù)據(jù)

打算將html中的字符串直接轉換為dict，然后再將原本的dict簡化成我們需要的dict(只留新聞標題，新聞詳情，新聞圖片與新聞鏈接)
代碼如下：

#解析數(shù)據(jù) def getData(html):print('#>>>解析數(shù)據(jù)中...>>>')newsDictListStr = Nonetry:findNewsDictList = re.compile(r',"content":\[\{([\s\S]*)\}\],"moreUrl":')newsDictListStr = re.findall(findNewsDictList,html)[0]print("#SUCCESS>字典串截取成功")except IndexError:print("#ERROR>字典串截取失敗，目標網(wǎng)站源碼可能發(fā)生改變!")return newsDictListStrdataStrList = newsDictListStr.split('},{')print('#SUCCESS>原始數(shù)據(jù)字典分析完畢')newsList = []try:for dataStr in dataStrList:Str ='{' + dataStr + '}'oldDict = ast.literal_eval(Str)newDict = dict()newDict["index"] = oldDict["index"] + 1newDict["article"] = oldDict["word"]newDict["info"] = oldDict["desc"]newDict["link"] = oldDict["url"]newDict["img"] = oldDict["img"]newsList.append(newDict)except KeyError:print("#ERROR>未找到數(shù)據(jù)字典中的鍵，原網(wǎng)站數(shù)據(jù)字典可能已經(jīng)改變")return Noneprint("#SUCCESS>新數(shù)據(jù)字典已提取") return newsList

5. 輸出保存整理好的新聞

我打算將得到的數(shù)據(jù)整理成markdown的格式保存下來，方便導出成圖片。
代碼如下：

# 生成MD文件字符串 def getMD(newsList):print('#>>>MD文件生成中...>>>')nowTime = datetime.datetime.now()nowTimeStr1 = nowTime.strftime('%Y/%m/%d %H:%M:%S')nowTimeStr2 = nowTime.strftime('%Y年%m月%d日 %H:%M:%S')colorList = ["red", "green", "blue"]newsMDStr = "---\ntitle: 每日新聞\ndate: {}\n---\n".format(nowTimeStr1)newsMDStr += '\n# ' + nowTimeStr2 + '\n'for i in range(len(newsList)):if i < 3:newsMDStr += "\n ## [<font color= {} >{} .{}</font>]({})<br><br>".format(colorList[i], str(newsList[i]["index"]), newsList[i]["article"], newsList[i]["link"])else:newsMDStr += "\n ### {} .[{}]({}) <br><br>".format(str(newsList[i]["index"]), newsList[i]["article"], newsList[i]["link"])newsMDStr += r'<img src="{}" width = 60% height = 60% align= "middle"/><br><br>'.format(newsList[i]["img"])if newsList[i]["info"] == "":newsMDStr += "> 暫無詳細消息<br><br>"else:newsMDStr += "> {}<br><br>".format(newsList[i]["info"])newsMDStr += '**信息來源來自百度熱榜，本網(wǎng)站不對真實性負責**\n'return newsMDStr# 保存MD文件 def writeMD(newsMDStr):f = open(savePath,"w",encoding="utf-8")f.write(newsMDStr)print("#SUCCESS>MD文件已生成")

這樣以來，就大功告成了，可以每天輕松看新聞啦！

6. 結果與全部代碼

整體代碼如下：

import re import ast import datetime import urllib.request,urllib.errornewsUrl = "https://top.baidu.com/board?tab=realtime" savePath = "news.md"# 主邏輯 def runNewsSpider():print('#>>>每日新聞爬取中...>>>')print("#目標網(wǎng)站:" + newsUrl)print("#存儲路徑:" + savePath)htmlCode = askURL()if htmlCode is None:print("#ERROR>數(shù)據(jù)字典爬取失敗")return FalsenewsList = getData(htmlCode)if newsList is None:print("#ERROR>數(shù)據(jù)字典生成失敗")return FalsenewsMDStr = getMD(newsList)writeMD(newsMDStr)print('#SUCCESS>新聞爬取完畢')return True#獲取網(wǎng)頁源碼 def askURL():print('#>>>獲取網(wǎng)頁HTML源碼中...>>>')# 模擬瀏覽器頭部信息，向服務器發(fā)送消息head={"User-Agent": "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36"}# 用戶代理，表示告訴豆瓣服務器，我們是什么類型的機器、瀏覽器（本質上是告訴瀏覽器，我們可以接收什么水平的文件內容）request=urllib.request.Request(newsUrl,headers=head)html=""try:response = urllib.request.urlopen(request)html = response.read().decode("utf-8")print("#SUCCESS>HTML源碼爬取完畢！\n")except urllib.error.URLError as e:print("#ERROR>HTML源碼爬取失敗！\n")if hasattr(e, "code"):print("#" + str(e.code))if hasattr(e, "reason"):print("#" + str(e.reason))return html#解析數(shù)據(jù) def getData(html):print('#>>>解析數(shù)據(jù)中...>>>')newsDictListStr = Nonetry:findNewsDictList = re.compile(r',"content":\[\{([\s\S]*)\}\],"moreUrl":')newsDictListStr = re.findall(findNewsDictList,html)[0]print("#SUCCESS>字典串截取成功")except IndexError:print("#ERROR>字典串截取失敗，目標網(wǎng)站源碼可能發(fā)生改變!")return newsDictListStrdataStrList = newsDictListStr.split('},{')print('#SUCCESS>原始數(shù)據(jù)字典分析完畢')newsList = []try:for dataStr in dataStrList:Str ='{' + dataStr + '}'oldDict = ast.literal_eval(Str)newDict = dict()newDict["index"] = oldDict["index"] + 1newDict["article"] = oldDict["word"]newDict["info"] = oldDict["desc"]newDict["link"] = oldDict["url"]newDict["img"] = oldDict["img"]newsList.append(newDict)except KeyError:print("#ERROR>未找到數(shù)據(jù)字典中的鍵，原網(wǎng)站數(shù)據(jù)字典可能已經(jīng)改變")return Noneprint("#SUCCESS>新數(shù)據(jù)字典已提取") return newsList# 生成MD文件字符串 def getMD(newsList):print('#>>>MD文件生成中...>>>')nowTime = datetime.datetime.now()nowTimeStr1 = nowTime.strftime('%Y/%m/%d %H:%M:%S')nowTimeStr2 = nowTime.strftime('%Y年%m月%d日 %H:%M:%S')colorList = ["red", "green", "blue"]newsMDStr = "---\ntitle: 每日新聞\ndate: {}\n---\n".format(nowTimeStr1)newsMDStr += '\n# ' + nowTimeStr2 + '\n'for i in range(len(newsList)):if i < 3:newsMDStr += "\n ## [<font color= {} >{} .{}</font>]({})<br><br>".format(colorList[i], str(newsList[i]["index"]), newsList[i]["article"], newsList[i]["link"])else:newsMDStr += "\n ### {} .[{}]({}) <br><br>".format(str(newsList[i]["index"]), newsList[i]["article"], newsList[i]["link"])newsMDStr += r'<img src="{}" width = 60% height = 60% align= "middle"/><br><br>'.format(newsList[i]["img"])if newsList[i]["info"] == "":newsMDStr += "> 暫無詳細消息<br><br>"else:newsMDStr += "> {}<br><br>".format(newsList[i]["info"])newsMDStr += '**信息來源來自百度熱榜，本網(wǎng)站不對真實性負責**\n'return newsMDStr# 保存MD文件 def writeMD(newsMDStr):f = open(savePath,"w",encoding="utf-8")f.write(newsMDStr)print("#SUCCESS>MD文件已生成")

導出md再導出圖片如下：

好兄弟萌，關注我，下次一起爬爬天氣預報~！

總結

以上是生活随笔為你收集整理的使用Python爬取百度热搜榜的全部內容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網(wǎng)站內容還不錯，歡迎將生活随笔推薦給好友。