本文我們將嘗試下載如下所示的京東商品曬單圖。

來到京東首頁，在搜索框中輸入關(guān)鍵字，比如：手機(jī)。打開瀏覽器的開發(fā)者工具(Ctrl+Shift+I)，切換到Network選項(xiàng)。

點(diǎn)擊按鈕搜索之前輸入的關(guān)鍵詞，可以看到有如下請求：

請求帶到服務(wù)器的參數(shù)主要有：

keyword: 手機(jī)

enc: utf-8

響應(yīng)內(nèi)容為一個HTML文件，包含了搜索到的商品的URL。

訪問搜索到的某個商品，同樣打開瀏覽器的開發(fā)者工具，切換到Network選項(xiàng)，可以看到有如下請求(沒有看到的話刷新一下頁面)：

返回的數(shù)據(jù)包含了評論相關(guān)的數(shù)據(jù)，曬單圖也在其中。

至此，我們將整個過程分析完畢，用代碼實(shí)現(xiàn)上述過程即可。

import requests

from bs4 import BeautifulSoup

from urllib.parse import urljoin

import re

import os

import json

def parse_goods_url(keyword):

headers = {

"User-Agent":

"Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.90 Safari/537.36"

}

params = {"keyword": keyword, "enc": "utf-8"}

base_url = "https://search.jd.com/Search"

response = requests.get(base_url, params=params, headers=headers)

soup = BeautifulSoup(response.text, "lxml")

goods_url_list = soup.select("#J_goodsList .gl-item .p-name a")

for good in goods_url_list:

yield urljoin("https://item.jd.com", good.get("href"))

def get_good_comment_image_url(good_url, page_num=1):

headers = {

"User-Agent":

"Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.90 Safari/537.36",

"Referer":

good_url

}

productId = re.findall("(\d+)", good_url)[0]

base_url = "https://sclub.jd.com/comment/productPageComments.action"

for page in range(page_num):

params = {

"callback": "fetchJSON_comment98vv3487",

"productId": productId,

"score": 0,

"sortType": 5,

"page": page,

"pageSize": 10,

"isShadowSku": 0,

"fold": 1

}

response = requests.get(base_url, params=params, headers=headers)

origin_data = re.findall("(\{.*\})", response.text)[0]

comments = json.loads(origin_data)["comments"]

for comment in comments:

for image in comment.get("images"):

yield urljoin(base_url, image.get("imgUrl"))

def download_image(image_url, good_url):

file_name = image_url.split('/')[-1]

fold_name = re.findall("(\d+)", good_url)[0]

if not os.path.exists(fold_name):

os.makedirs(fold_name)

headers = {

"User-Agent":

"Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.90 Safari/537.36",

"Referer":

good_url

}

response = requests.get(image_url, headers=headers)

with open("{}/{}".format(fold_name, file_name), "wb") as f_obj:

f_obj.write(response.content)

if __name__ == '__main__':

keyword = "手機(jī)"

for url in parse_goods_url(keyword):

print("商品：{}".format(url))

for image_url in get_good_comment_image_url(url):

print("下載：{}".format(image_url))

download_image(image_url, url)

運(yùn)行結(jié)果如下：

總結(jié)

以上是生活随笔為你收集整理的python老王卖西瓜_使用Python爬取京东商品晒单图的全部內(nèi)容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網(wǎng)站內(nèi)容還不錯，歡迎將生活随笔推薦給好友。