我在寫爬蟲的時候，經常會獲取大量的數據。這個時候大量數據的數據庫操作如果不做一定的優化的話，將會耗費大量的時間。通過實踐我發現，批量的數據操作將會極大的提高數據庫操作的效率。同時，很多時候存在數據如果存在做更新操作，如果數據不存在，做插入插入操作的情況。于是我造了一個只需傳入一個數組，就會自動更新或插入數據的工具。
實測以下方法操作幾十萬條數據只需要幾秒到十幾秒
以下為代碼：

#說明： #使用時直接調用saveAll方法，其他方法將被saveAll調用，你可以無視 #saveAll參數說明： # table 表名 # datas 數據的數組 例 ：[{"key1":"value1","key2":"value2"},{"key1":"value1","key2":"value2"}] 建議數組大小不要超過一千。 # searchKeys 用于確定唯一行的鍵的數組，如用戶表的用戶名，選課表的課程ID與學生ID等 例 ["user_id","class_id"] # ifIgnoreSearchKey 是否忽略searchKey 如果你的searchKeys 是自增長的ID 你肯定不希望插入的時候插入這個字段 "1"是，"0"否 # ifNotUpdate 是否不做更新操作 如果這個設為 "0" ,datas中數據如果已在數據庫中，將不會做更新操作 #getConnection 方法中的DB 是我從我的配置文件中導入的，你可以換成你的 def getConnection():conn = MySQLdb.connect(host=DB["host"],user=DB["user"],passwd=DB["passwd"],db=DB["db"],charset=DB["charset"])return conndef saveAll(table,datas,searchKeys,ifIgnoreSearchKey,ifNotUpdate):print datasconn = getConnection()cursor = conn.cursor()where = []#轉義數據,避免sql發生錯誤for data in datas:for key in data:data[key] = MySQLdb.escape_string(str(data[key]))for searchKey in searchKeys:searchKeyDatas = []for data in datas:searchKeyDatas.append(data[searchKey])searchKeyDatasString = "`"+searchKey+"` in ('"+"','".join(searchKeyDatas)+"')"where.append(searchKeyDatasString)whereString = " AND ".join(where)selectSql = "SELECT `"+"`,`".join(searchKeys)+"` from "+table+" WHERE " + whereStringcursor.execute(selectSql)conn.commit()results = cursor.fetchall()updateData = []insertData = []existKey = []for result in results:keyValue = []for value in result:keyValue.append(str(value))existKey.append(",".join(keyValue))for data in datas:keyValue = []for key in searchKeys:keyValue.append(data[key])currentKey = ",".join(keyValue)if currentKey in existKey:updateData.append(data)else:insertData.append(data)if ifNotUpdate == "0":updateAll(updateData,table,searchKeys)insertAll(insertData,table,searchKeys,ifIgnoreSearchKey)conn.close()passdef updateAll(datas,table,searchKeys):#同時更新多條數據if len(datas) == 0:returnconn = getConnection()cursor = conn.cursor()sets = {}updateSql = "UPDATE `"+table+"` set "whereValues = []whereKey = "WHERE CONCAT(`"+"`,',',`".join(searchKeys)+"`) IN "for data in datas:whereValue = []for searchKey in searchKeys:whereValue.append(data[searchKey])whereValueString = ",".join(whereValue)whereValues.append(whereValueString)for key in data:if key in searchKeys:passelse:searchValue = []for searchKey in searchKeys:searchValue.append(str(data[searchKey]))searchValueString = ",".join(searchValue)try:sets[key][searchValueString] = data[key]except KeyError as e:sets[key] = {}sets[key][searchValueString] = data[key]searchKeysString = "(`"+"`,',',`".join(searchKeys)+"`)"whereValuesString = "('"+"','".join(whereValues)+"')"setStringArray = []for key1 in sets:setString = ""for key2 in sets[key1]:if setString == "":setString = "`"+key1+"` = CASE WHEN (CONCAT"+searchKeysString+"='"+key2+"') THEN '"+sets[key1][key2]+"'"else:setString = setString + " WHEN (CONCAT"+searchKeysString+"='"+key2+"') THEN '"+sets[key1][key2]+"'"setString += " END "setStringArray.append(setString)setStrings = ",".join(setStringArray)whereStrings = whereKey + whereValuesStringupdateSql += setStringsupdateSql += whereStringsprint updateSqltry:cursor.execute(updateSql)result = cursor.fetchall()except Exception as e:print e.messageprint updateSqlconn.commit()conn.close()def insertAll(datas,table,searchKeys,ifIgnoreSearchKey):#多條數據同時添加if len(datas) == 0:returnconn = getConnection()cursor = conn.cursor()keys=[]for key in datas[0]:if key not in searchKeys or ifIgnoreSearchKey!= "1":keys.append(key)insertSql = "INSERT INTO "+table+" (`"+"`,`".join(keys)+"`) VALUES "valueStrings = []for data in datas:value = []for key in keys:value.append(data[key])valueString = "('" + "','".join(value) + "')"valueStrings.append(valueString)insertSql += ",".join(valueStrings)print insertSqltry:cursor.execute(insertSql)conn.commit()conn.close()except Exception as e:print insertSqlprint e.message

如果有幫到您，打個賞唄

總結

以上是生活随笔為你收集整理的python封装一个效率极高的 批量更新、插入合一的工具的全部內容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網站內容還不錯，歡迎將生活随笔推薦給好友。