64*64*3小圖（12KB），batchSize=128，訓練樣本100萬，

全部load進來內存受不了，load一次需要大半天

訓練時讀入一個batch，ali云服務器上每個batch讀入時間1.9~3.2s不等，迭代一次2s多

由于有多個label不能用caffe自帶的lmdb轉了，輸入是自己寫的python層，試著用pickel

import os, sys import cv2 import numpy as np import numpy.random as npr import cPickle as pickle wk_dir = "/Users/xxx/wkspace/caffe_space/detection/caffe/data/1103reg64/" InputSize = int(sys.argv[1]) BatchSize = int(sys.argv[2]) trainfile = "train.txt" testfile = "test.txt" print "gen imdb with for net input:", InputSize, "batchSize:", BatchSizewith open(wk_dir+trainfile, 'r') as f:trainlines = f.readlines() with open(wk_dir+testfile, 'r') as f:testlines = f.readlines() ####################################### # we seperate train data by batchsize # ####################################### to_dir = wk_dir + "/trainIMDB/" if not os.path.isdir(to_dir):os.makedirs(to_dir)train_list = [] cur_ = 0 sum_ = len(trainlines) for line in trainlines:cur_ += 1words = line.split()image_file_name = words[0]im = cv2.imread(wk_dir + image_file_name)h,w,ch = im.shapeif h!=InputSize or w!=InputSize:im = cv2.resize(im,(InputSize,InputSize))roi = [float(words[2]),float(words[3]),float(words[4]),float(words[5])]train_list.append([im, roi])if (cur_ % BatchSize == 0):print "write batch:" , cur_/BatchSizefid = open(to_dir +'train'+ str(BatchSize) + '_'+str(cur_/BatchSize),'w')pickle.dump(train_list, fid)fid.close()train_list[:] = []print len(train_list), "train data generated\n"########################### # tests # ########################### to_dir = wk_dir + "/testIMDB/" if not os.path.isdir(to_dir):os.makedirs(to_dir) test_list = [] cur_ = 0 sum_ = len(testlines) for line in testlines:cur_ += 1words = line.split()image_file_name = words[0]im = cv2.imread(wk_dir + image_file_name)h,w,ch = im.shapeif h!=InputSize or w!=InputSize:im = cv2.resize(im,(InputSize,InputSize))roi = [float(words[2]),float(words[3]),float(words[4]),float(words[5])]test_list.append([im, roi])if (cur_ % BatchSize == 0):print "write batch:", cur_ / BatchSizefid = open(to_dir +'test'+ str(BatchSize) + '_'+str(cur_/BatchSize), 'w')pickle.dump(test_list, fid)fid.close()test_list[:] = [] print len(test_list), "test data generated\n"

每個batch生成4.8MB的塊（約比128張原圖占3倍磁盤空間）：

訓練時讀入，ali云訓練每個batch時間變為0.2s，可加速10倍

?

mac上是ssd硬盤，本來讀圖就很快，一個batch 0.05s, 改成pickel后反而變慢了，load一個batch需要0.2s。

?

轉載于:https://www.cnblogs.com/zhengmeisong/p/9903539.html

總結

以上是生活随笔為你收集整理的pickel加速caffe读图的全部內容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網站內容還不錯，歡迎將生活随笔推薦給好友。