文章目錄

• • 需求
  • 實(shí)現(xiàn)
  • • 先導(dǎo)入本次需要用到的包
    • 一些輔助函數(shù)
    • • 如下函數(shù)是得到指定后綴的文件
      • 如下的函數(shù)一個(gè)是讀圖像，一個(gè)是把RGB轉(zhuǎn)成BGR
    • 下面是主要的幾個(gè)處理函數(shù)
    • 在上面幾個(gè)函數(shù)構(gòu)建對(duì)應(yīng)的處理函數(shù)
    • main函數(shù)
    • 按順序執(zhí)行
  • 結(jié)果

需求

本次的需求是邊讀圖像，邊處理圖像（各種變組合），處理完后還要把處理好的圖像保存到指定的文件夾。而且圖像也挺多的，如果按順序一個(gè)一個(gè)處理，那肯定要不少時(shí)間。所以就想到了多線程并發(fā)編程。

實(shí)現(xiàn)

先導(dǎo)入本次需要用到的包

import os import threading from queue import Queue import cv2

一些輔助函數(shù)

如下函數(shù)是得到指定后綴的文件

IMG_EXTENSIONS = ('.jpg', '.jpeg', '.png', '.ppm', '.bmp', '.pgm', '.tif', '.tiff', '.webp')def get_all_files(base, extensions):"""get all files in extensions from base folder, it's a generator"""for root, _, files in sorted(os.walk(base, followlinks=True)):for file in sorted(files):if file.endswith(extensions):yield os.path.join(root, file)def get_all_images(base, image_extensions):"""get all images"""return get_all_files(base, image_extensions)

如下的函數(shù)一個(gè)是讀圖像，一個(gè)是把RGB轉(zhuǎn)成BGR

def default_loader_cv2(path):return cv2.cvtColor(cv2.imread(path), cv2.COLOR_BGR2RGB)def rgb_2_bgr(img):return cv2.cvtColor(img, cv2.COLOR_RGB2BGR)

下面是主要的幾個(gè)處理函數(shù)

def load_image(target_dir, source_file):"""load image here"""target_file = get_save_path(target_dir, source_file)img = default_loader_cv2(source_file)return (target_file, img)def transform(stain_normalizer, img):"""Description:- transform image method, basic resize here, you could do other transform here"""return cv2.resize(img, (256, 256))def save(save_path, img):"""save image method"""cv2.imwrite(save_path, rgb_2_bgr(img))

在上面幾個(gè)函數(shù)構(gòu)建對(duì)應(yīng)的處理函數(shù)

def do_load_image(load_queue: Queue, trainsform_queue: Queue, target_dir:str):while True:file = load_queue.get()if file is None: breaktarget_file = os.path.join(target_dir, source_file)if not os.path.exists(target_file): # skip all the transformed imagesimg = default_loader_cv2(file)trainsform_queue.put((target_file, img))else:passdef do_transforms(trainsform_queue: Queue, save_queue: Queue, stain_normalizer):while True:data = trainsform_queue.get()if data is None: breaktarget_file, img = dataimg_norm = transform(stain_normalizer, img)save_queue.put((target_file, img_norm))def do_save(save_queue:Queue):while True:data = save_queue.get()if data is None: breaktarget_file, img_norm = datasave(target_file, img_norm)

main函數(shù)

在這里，是整個(gè)程度的啟動(dòng)，特別注意線程的啟動(dòng)與結(jié)束順序，不要搞錯(cuò)了，不然程序會(huì)進(jìn)行死循環(huán)。
一般生產(chǎn)者消費(fèi)者，大家看到的都是只有兩個(gè)函數(shù)（一個(gè)生產(chǎn)者，一個(gè)消費(fèi)者），這里實(shí)行的是3個(gè)函數(shù)，load是transform的生產(chǎn)者，transform是save的生產(chǎn)者，這里利用隊(duì)列實(shí)行了3個(gè)隊(duì)列，實(shí)行了數(shù)據(jù)間的傳遞。可以利用這種思想實(shí)行更多層級(jí)的生產(chǎn)者與消費(fèi)者模式。

def main(source_dir, target_dir):# 4104 image, took 224.6297sfiles = get_all_images(source_dir, IMG_EXTENSIONS) # generator could only be iterated 1 time# transform will be the slowest, so load queue would be too much data if you donot maximizeload_queue = Queue(maxsize=5000) trainsform_queue = Queue()save_queue = Queue()for file in files:load_queue.put(file)# start load_threadsload_threads = []for _ in range(2):t = threading.Thread(target=do_load_image,args=(load_queue, trainsform_queue, target_dir))t.start()load_threads.append(t)# start transform_threadstransform_threads = []for _ in range(6):t = threading.Thread(target=do_transforms,args=(trainsform_queue, save_queue, stain_normalizer))t.start()transform_threads.append(t)# start save_threadssave_threads = []for _ in range(4):t = threading.Thread(target=do_save,args=(save_queue,))t.start()save_threads.append(t)# put sentinel load_threads to break the loop# DONOT put thread.join() under this loopfor _ in load_threads:load_queue.put(None)for thread in load_threads:thread.join()# put sentinel transform_threads to break the loop# DONOT put thread.join() under this loopfor thread in transform_threads:trainsform_queue.put(None)for thread in transform_threads:thread.join()# put sentinel transform_threads to break the loop# DONOT put thread.join() under this loopfor thread in save_threads:save_queue.put(None)for thread in save_threads:thread.join()

按順序執(zhí)行

def single_thread(source_dir, target_dir):# 4104 image, took 486.4547sfiles = get_all_images(source_dir, IMG_EXTENSIONS)for file in files:target_file, img = load_image(target_dir, file)img_transform = transform(stain_normalizer, img)save(target_file, img_transform)

結(jié)果

從代碼來(lái)看，單線程的順序執(zhí)行比多線程少不小的代碼，而且結(jié)果也相對(duì)簡(jiǎn)單，基本上不會(huì)出什么問(wèn)題。然后單線程的所要花費(fèi)的時(shí)間卻是多線程的2倍還要多。圖像一共是4104張512x512的3通道png圖像。單線程花費(fèi)時(shí)間是486.4547s，而多線程花費(fèi)時(shí)間是224.6297s。是雖然多線程的代碼多了點(diǎn)，但是從性能上來(lái)說(shuō)，還是比單線程順序執(zhí)行快不少，還是蠻值得的

總結(jié)

以上是生活随笔為你收集整理的python 多线程并发编程（生产者、消费者模式），边读图像，边处理图像，处理完后保存图像实现提高处理效率的全部?jī)?nèi)容，希望文章能夠幫你解決所遇到的問(wèn)題。

如果覺(jué)得生活随笔網(wǎng)站內(nèi)容還不錯(cuò)，歡迎將生活随笔推薦給好友。