前言

python在進程間通信會比較麻煩~
要注意，這里我們想到的時候多個進程一起維護一個字典。
所以，我加了鎖。這里還是存在一個可能的改進方法，就是考察你的鎖加在哪個位置來判斷。
舊的單進程版本

代碼一：

用時: 10.798075914382935秒
看了三千個網頁的樣子。還算不錯的啦。但是還是想要跟好的優化~

import requests from bs4 import BeautifulSoup import re from multiprocessing import Manager, Lock, Processdef getUrl(url='https://gz.lianjia.com/ershoufang/pg{}/'):for i in range(100):yield url.format(i + 1)def findAInUrl(url, lock, tag):res = requests.get(url)soup = BeautifulSoup(res.text, 'lxml')lis = soup.find_all('li', attrs={'class': "clear"})for li in lis:info = li.find('div', attrs={'class': 'info clear'})tags = info.find('div', attrs={'class': 'tag'})for t in tags:ans = re.findall('(?<=\>).+(?=\<)', str(t))if '距離' in ans[0]:ans[0] = '地鐵'lock.acquire()if ans[0] not in tag:tag[ans[0]] = 1else:tag[ans[0]] += 1lock.release()if __name__ == '__main__':tag = Manager().dict()pList = []lock = Lock()for url in getUrl():p = Process(target=findAInUrl, args=(url, lock, tag))pList.append(p)p.start()for p in pList:p.join()print(tag)

代碼二

用時：11.187129020690918秒

import requests from bs4 import BeautifulSoup import re from multiprocessing import Manager, Lock, Process import timedef getUrl(url='https://gz.lianjia.com/ershoufang/pg{}/'):for i in range(100):yield url.format(i + 1)def findAInUrl(url, lock, tag):res = requests.get(url)soup = BeautifulSoup(res.text, 'lxml')lis = soup.find_all('li', attrs={'class': "clear"})for li in lis:info = li.find('div', attrs={'class': 'info clear'})tags = info.find('div', attrs={'class': 'tag'})lock.acquire()for t in tags:ans = re.findall('(?<=\>).+(?=\<)', str(t))if '距離' in ans[0]:ans[0] = '地鐵'if ans[0] not in tag:tag[ans[0]] = 1else:tag[ans[0]] += 1lock.release()if __name__ == '__main__':st = time.time()tag = Manager().dict()pList = []lock = Lock()for url in getUrl():p = Process(target=findAInUrl, args=(url, lock, tag))pList.append(p)p.start()for p in pList:p.join()print(tag)et = time.time()print(et- st)

代碼三

用時：11.184273481369019秒

import requests from bs4 import BeautifulSoup import re from multiprocessing import Manager, Lock, Process import timedef getUrl(url='https://gz.lianjia.com/ershoufang/pg{}/'):for i in range(100):yield url.format(i + 1)def findAInUrl(url, lock, tag):res = requests.get(url)soup = BeautifulSoup(res.text, 'lxml')lis = soup.find_all('li', attrs={'class': "clear"})lock.acquire()for li in lis:info = li.find('div', attrs={'class': 'info clear'})tags = info.find('div', attrs={'class': 'tag'})for t in tags:ans = re.findall('(?<=\>).+(?=\<)', str(t))if '距離' in ans[0]:ans[0] = '地鐵'if ans[0] not in tag:tag[ans[0]] = 1else:tag[ans[0]] += 1lock.release()if __name__ == '__main__':st = time.time()tag = Manager().dict()pList = []lock = Lock()for url in getUrl():p = Process(target=findAInUrl, args=(url, lock, tag))pList.append(p)p.start()for p in pList:p.join()print(tag)et = time.time()print(et- st)

---

前三個版本的代碼區別在于鎖的位置的不同，導致了時間的不一樣。

總結

以上是生活随笔為你收集整理的【多进程并行版本】爬取链家二手房前100页标签，进行统计的全部內容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網站內容還不錯，歡迎將生活随笔推薦給好友。