PHP方式寫的一團(tuán)糟所以就用python3重寫了一遍，所以因?yàn)榈诙螌懥?#xff0c;思路也更清晰了些。

提醒：可能會(huì)有502的錯(cuò)誤，所以做了異常以及數(shù)據(jù)庫(kù)事務(wù)處理，暫時(shí)沒有想到更好的優(yōu)化方法，所以就先這樣吧。待更懂python再進(jìn)一步優(yōu)化哈

歡迎留言賜教~

#!C:\Users\12550\AppData\Local\Programs\Python\Python37\python.exe

# -*- coding: utf-8 -*-

from urllib.request import urlopen

from bs4 import BeautifulSoup

import pymysql

import urllib.request

import re

from urllib.error import URLError, HTTPError

conn = pymysql.connect(host='127.0.0.1', user='root', passwd='root', db='test', charset='utf8')

db = conn.cursor()

curr_url = ''

# 請(qǐng)求網(wǎng)頁(yè)

def get_html(url):

global curr_url

user_agent = 'Mozilla/6.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.6796.99 Safari/537.36'

response = urllib.request.Request(url)

response.add_header('User-Agent', user_agent)

response = urllib.request.urlopen(response)

html = BeautifulSoup(response.read(), "html.parser", from_encoding='gbk')

return html

def get_list(url, level=1, pid=0, get_level=2):

data = [];

level_arr = {'1': 'provincetr', '2': 'citytr', '3': 'countytr', '4': 'towntr', '5': 'villagetr'}

try:

print(url)

html = get_html(url)

c_url = url

tr_list = html.findAll('tr', {'class': level_arr[str(level)]})

for tr in tr_list:

region_name, href, page = '', '', ''

td_list = tr.findAll('td')

for td in td_list:

region_name = td.get_text();

# 判斷是否存在該省份

if (level == 1):

sql = "select * from region where region_name='" + region_name + "'"

db.execute(sql)

exist = db.fetchone()

if(exist):

continue

# 判斷是否全數(shù)字-非法則跳過

if (region_name.isdigit()):

continue

if (region_name):

sql = "insert into region(region_name,pid,level,url) value('" + region_name + "','" + str(

pid) + "','" + str(level) + "','" + url + "')"

db.execute(sql)

db.execute('SELECT LAST_INSERT_ID();')

last_id = db.fetchone()[0]

if (td.a):

page = td.a.attrs['href']

pattern = re.compile(r'\w*.html')

url = re.sub(pattern, page, c_url)

if (level <= get_level):

get_list(url, level + 1, last_id)

# 每個(gè)省份執(zhí)行完成，則提交

if (level == 1):

conn.commit()

return data;

except HTTPError as e:

# 如果有出錯(cuò)，則回滾

conn.rollback()

print(e) # HTTP Error 502: Proxy Error

url = 'http://www.stats.gov.cn/tjsj/tjbz/tjyqhdmhcxhfdm/2017/index.html'

get_list(url)

print('執(zhí)行完成')

總結(jié)

以上就是這篇文章的全部?jī)?nèi)容了，希望本文的內(nèi)容對(duì)大家的學(xué)習(xí)或者工作具有一定的參考學(xué)習(xí)價(jià)值，謝謝大家對(duì)腳本之家的支持。如果你想了解更多相關(guān)內(nèi)容請(qǐng)查看下面相關(guān)鏈接

總結(jié)

以上是生活随笔為你收集整理的python爬取全国真实地址_Python3爬虫全国地址信息的全部?jī)?nèi)容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網(wǎng)站內(nèi)容還不錯(cuò)，歡迎將生活随笔推薦給好友。