python進階 多線程編程 —— threading和queue庫實現多線程編程）

• 摘要
• 多線程實現邏輯封裝
• 模型參數選擇實例

摘要

本文主要介紹了利用python的 threading和queue庫實現多線程編程，并封裝為一個類，方便讀者嵌入自己的業務邏輯。最后以機器學習的一個超參數選擇為例進行演示。

多線程實現邏輯封裝

實例化該類后，在.object_func函數中加入自己的業務邏輯，再調用.run方法即可。

# -*- coding: utf-8 -*- # @Time : 2021/2/4 14:36 # @Author : CyrusMay WJ # @FileName: run.py # @Software: PyCharm # @Blog ：https://blog.csdn.net/Cyrus_May import queue import threadingclass CyrusThread(object):def __init__(self,num_thread = 10,logger=None):""":param num_thread: 線程數:param logger: 日志對象"""self.num_thread = num_threadself.logger = loggerdef object_func(self,args_queue,max_q):while 1:try:arg = args_queue.get_nowait()step = args_queue.qsize()self.logger.info("progress:{}\{}".format(max_q,step))except:self.logger.info("no more arg for args_queue!")break"""此處加入自己的業務邏輯代碼"""def run(self,args):args_queue = queue.Queue()for value in args:args_queue.put(value)threads = []for i in range(self.num_thread):threads.append(threading.Thread(target=self.object_func,args = args_queue))for t in threads:t.start()for t in threads:t.join()

模型參數選擇實例

# -*- coding: utf-8 -*- # @Time : 2021/2/4 14:36 # @Author : CyrusMay WJ # @FileName: run.py # @Software: PyCharm # @Blog ：https://blog.csdn.net/Cyrus_May import queue import threading import numpy as np from sklearn.datasets import load_boston from sklearn.svm import SVR import logging import sysclass CyrusThread(object):def __init__(self,num_thread = 10,logger=None):""":param num_thread: 線程數:param logger: 日志對象"""self.num_thread = num_threadself.logger = loggerdef object_func(self,args_queue,max_q):while 1:try:arg = args_queue.get_nowait()step = args_queue.qsize()self.logger.info("progress:{}\{}".format(max_q,max_q-step))except:self.logger.info("no more arg for args_queue!")break# 業務代碼C, epsilon, gamma = arg[0], arg[1], arg[2]svr_model = SVR(C=C, epsilon=epsilon, gamma=gamma)x, y = load_boston()["data"], load_boston()["target"]svr_model.fit(x, y)self.logger.info("score:{}".format(svr_model.score(x,y)))def run(self,args):args_queue = queue.Queue()max_q = 0for value in args:args_queue.put(value)max_q += 1threads = []for i in range(self.num_thread):threads.append(threading.Thread(target=self.object_func,args = (args_queue,max_q)))for t in threads:t.start()for t in threads:t.join()# 創建日志對象 logger = logging.getLogger() logger.setLevel(logging.INFO) screen_handler = logging.StreamHandler(sys.stdout) screen_handler.setLevel(logging.INFO) formatter = logging.Formatter('%(asctime)s - %(module)s.%(funcName)s:%(lineno)d - %(levelname)s - %(message)s') screen_handler.setFormatter(formatter) logger.addHandler(screen_handler)# 創建需要調整參數的集合 args = [] for C in [i for i in np.arange(0.01,1,0.01)]:for epsilon in [i for i in np.arange(0.001,1,0.01)] + [i for i in range(1,10,1)]:for gamma in [i for i in np.arange(0.001,1,0.01)] + [i for i in range(1,10,1)]:args.append([C,epsilon,gamma])# 創建多線程工具 threading_tool = CyrusThread(num_thread=20,logger=logger) threading_tool.run(args)

運行結果

2021-02-04 20:52:22,824 - run.object_func:31 - INFO - progress:1176219\1 2021-02-04 20:52:22,824 - run.object_func:31 - INFO - progress:1176219\2 2021-02-04 20:52:22,826 - run.object_func:31 - INFO - progress:1176219\3 2021-02-04 20:52:22,833 - run.object_func:31 - INFO - progress:1176219\4 2021-02-04 20:52:22,837 - run.object_func:31 - INFO - progress:1176219\5 2021-02-04 20:52:22,838 - run.object_func:31 - INFO - progress:1176219\6 2021-02-04 20:52:22,841 - run.object_func:31 - INFO - progress:1176219\7 2021-02-04 20:52:22,862 - run.object_func:31 - INFO - progress:1176219\8 2021-02-04 20:52:22,873 - run.object_func:31 - INFO - progress:1176219\9 2021-02-04 20:52:22,884 - run.object_func:31 - INFO - progress:1176219\10 2021-02-04 20:52:22,885 - run.object_func:31 - INFO - progress:1176219\11 2021-02-04 20:52:22,897 - run.object_func:31 - INFO - progress:1176219\12 2021-02-04 20:52:22,900 - run.object_func:31 - INFO - progress:1176219\13 2021-02-04 20:52:22,904 - run.object_func:31 - INFO - progress:1176219\14 2021-02-04 20:52:22,912 - run.object_func:31 - INFO - progress:1176219\15 2021-02-04 20:52:22,920 - run.object_func:31 - INFO - progress:1176219\16 2021-02-04 20:52:22,920 - run.object_func:39 - INFO - score:-0.01674283914287855 2021-02-04 20:52:22,929 - run.object_func:31 - INFO - progress:1176219\17 2021-02-04 20:52:22,932 - run.object_func:39 - INFO - score:-0.007992354170952565 2021-02-04 20:52:22,932 - run.object_func:31 - INFO - progress:1176219\18 2021-02-04 20:52:22,945 - run.object_func:31 - INFO - progress:1176219\19 2021-02-04 20:52:22,954 - run.object_func:31 - INFO - progress:1176219\20 2021-02-04 20:52:22,978 - run.object_func:31 - INFO - progress:1176219\21 2021-02-04 20:52:22,984 - run.object_func:39 - INFO - score:-0.018769934807246536 2021-02-04 20:52:22,985 - run.object_func:31 - INFO - progress:1176219\22

by CyrusMay 2021 02 04

在某個清晨
回望我一生
活得雖認真
卻微小如塵
——————五月天（一半人生）——————

總結

以上是生活随笔為你收集整理的python进阶 多线程编程 —— threading和queue库实现多线程编程的全部內容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網站內容還不錯，歡迎將生活随笔推薦給好友。