官網文檔

API文檔

Docker鏡像

自定義Plugin倉庫

0.安裝

1.安裝tensorrt

從官網下載.deb包,要注意的是cuda版本

sudo dpkg -i nv-tensorrt-repo-ubuntu1604-cuda10.0-trt7.0.0.11-ga-20191216_1-1_amd64.deb sudo apt update sudo apt install tensorrt

Engine plan 的兼容性依賴于GPU的compute capability 和 TensorRT 版本, 不依賴于CUDA和CUDNN版本.

2.安裝opencv

sudo apt-get update sudo apt install libopencv-dev

apt-get install tensorrt報錯

https://github.com/NVIDIA/TensorRT/issues/792

tensorrt : Depends: libnvinfer7 (= 7.0.0-1+cuda10.0) but 7.2.2-1+cuda11.1 is to be installedDepends: libnvinfer-plugin7 (= 7.0.0-1+cuda10.0) but 7.2.2-1+cuda11.1 is to be installedDepends: libnvparsers7 (= 7.0.0-1+cuda10.0) but 7.2.2-1+cuda11.1 is to be installedDepends: libnvonnxparsers7 (= 7.0.0-1+cuda10.0) but 7.2.2-1+cuda11.1 is to be installedDepends: libnvinfer-bin (= 7.0.0-1+cuda10.0) but it is not going to be installedDepends: libnvinfer-dev (= 7.0.0-1+cuda10.0) but 7.2.2-1+cuda11.1 is to be installedDepends: libnvinfer-plugin-dev (= 7.0.0-1+cuda10.0) but 7.2.2-1+cuda11.1 is to be installedDepends: libnvparsers-dev (= 7.0.0-1+cuda10.0) but 7.2.2-1+cuda11.1 is to be installedDepends: libnvonnxparsers-dev (= 7.0.0-1+cuda10.0) but 7.2.2-1+cuda11.1 is to be installedDepends: libnvinfer-samples (= 7.0.0-1+cuda10.0) but it is not going to be installedDepends: libnvinfer-doc (= 7.0.0-1+cuda10.0) but it is not going to be installed

mv?/etc/apt/sources.list.d/nvidia-ml.list?/etc/apt/sources.list.d/nvidia-ml.list.bak

在apt-get install tensorrt 即可

1.優化流程:

TensorRT總共有5個階段：創建網絡、構建推理Engine、序列化引擎、反序列化引擎以及執行推理Engine。

其中第1,2,3大概就是c++api寫的網絡結構或者其他第三方格式，經過NetworkDefinition進行定義，采用builder加載模型權重，進行一些參數的優化，然后再用engine序列化成“Plan”（流圖）,其不僅保存了計算時所需的網絡weights也保存了Kernel執行的調度流程。。

而4,5就是推理:采用engine反序列化，創建運行環境，在進行推理即可。

可看出TensorRT在獲得網絡計算流圖后會針對計算流圖進行優化.

深度學習框架在做推理時，會對每一層調用多個/次功能函數。而由于這樣的操作都是在GPU上運行的，從而會帶來多次的CUDA Kernel launch過程。相較于Kernel launch以及每層tensor data讀取來說，kernel的計算是更快更輕量的，從而使得這個程序受限于顯存帶寬并損害了GPU利用率。

TensorRT通過以下三種方式來解決這個問題：

Kernel縱向融合：通過融合相同順序的操作來減少Kernel Launch的消耗以及避免層之間的顯存讀寫操作。如上圖所示，卷積、Bias和Relu層可以融合成一個Kernel，這里稱之為CBR。

Kernel橫向融合：TensorRT會去挖掘輸入數據且filter大小相同但weights不同的層，對于這些層不是使用三個不同的Kernel而是使用一個Kernel來提高效率，如上圖中超寬的1x1 CBR所示,把結構相同但權重不同的層合并成更寬的層,從而減少cuda核心的使用.。

消除concatenation層，通過預分配輸出緩存以及跳躍式的寫入方式來避免這次轉換。

通過這樣的優化，TensorRT可以獲得更小、更快、更高效的計算流圖，其擁有更少層網絡結構以及更少Kernel Launch次數。下表列出了常見幾個網絡在TensorRT優化后的網絡層數量，很明顯的看到TensorRT可以有效的優化網絡結構、減少網絡層數從而帶來性能的提升。

2.torch版lenet轉trt

2.1 torch版代碼:

lenet.py

# coding:utf-8 import torch from torch import nn from torch.nn import functional as Fclass Lenet5(nn.Module):"""for cifar10 dataset."""def __init__(self):super(Lenet5, self).__init__()self.conv1 = nn.Conv2d(1, 6, kernel_size=5, stride=1, padding=0)self.pool1 = nn.AvgPool2d(kernel_size=2, stride=2, padding=0)self.conv2 = nn.Conv2d(6, 16, kernel_size=5, stride=1, padding=0)self.fc1 = nn.Linear(16 * 5 * 5, 120)self.fc2 = nn.Linear(120, 84)self.fc3 = nn.Linear(84, 10)def forward(self, x):# print('input: ', x.shape)x = F.relu(self.conv1(x))# print('conv1', x.shape)x = self.pool1(x)# print('pool1: ', x.shape)x = F.relu(self.conv2(x))# print('conv2', x.shape)x = self.pool1(x)# print('pool2', x.shape)x = x.view(x.size(0), -1)# print('view: ', x.shape)x = F.relu(self.fc1(x))# print('fc1: ', x.shape)x = F.relu(self.fc2(x))x = F.softmax(self.fc3(x), dim=1)return xdef main():import osos.environ["CUDA_VISIBLE_DEVICES"] = "1"print('cuda device count: ', torch.cuda.device_count())torch.manual_seed(1234)net = Lenet5()net = net.to('cuda:0')net.eval()import timest_time = time.time()nums = 10000for i in range(nums):tmp = torch.ones(1, 1, 32, 32).to('cuda:0')out = net(tmp)# print('lenet out shape:', out.shape)print('lenet out:', out)end_time = time.time()print('==cost time{}'.format((end_time - st_time)))torch.save(net, "lenet5.pth")if __name__ == '__main__':main()

將模型權重存儲為.pth,并測試時間為:

2.2.pth存儲為.onnx

為了方便查看網絡結構

# coding:utf-8 import torch from torch import nn from torch.nn import functional as Fclass Lenet5(nn.Module):"""for cifar10 dataset."""def __init__(self):super(Lenet5, self).__init__()self.conv1 = nn.Conv2d(1, 6, kernel_size=5, stride=1, padding=0)self.pool1 = nn.AvgPool2d(kernel_size=2, stride=2, padding=0)self.conv2 = nn.Conv2d(6, 16, kernel_size=5, stride=1, padding=0)self.fc1 = nn.Linear(16 * 5 * 5, 120)self.fc2 = nn.Linear(120, 84)self.fc3 = nn.Linear(84, 10)def forward(self, x):# print('input: ', x.shape)x = F.relu(self.conv1(x))# print('conv1', x.shape)x = self.pool1(x)# print('pool1: ', x.shape)x = F.relu(self.conv2(x))# print('conv2', x.shape)x = self.pool1(x)# print('pool2', x.shape)x = x.view(x.size(0), -1)# print('view: ', x.shape)x = F.relu(self.fc1(x))# print('fc1: ', x.shape)x = F.relu(self.fc2(x))x = F.softmax(self.fc3(x), dim=1)return xdef main():import osos.environ["CUDA_VISIBLE_DEVICES"] = "1"print('cuda device count: ', torch.cuda.device_count())torch.manual_seed(1234)net = Lenet5()net = net.to('cuda:0')net.eval()import timest_time = time.time()nums = 10000for i in range(nums):tmp = torch.ones(1, 1, 32, 32).to('cuda:0')out = net(tmp)# print('lenet out shape:', out.shape)print('lenet out:', out)end_time = time.time()print('==cost time{}'.format((end_time - st_time)))torch.save(net, "lenet5.pth")def model_onnx():input = torch.ones(1, 1, 32, 32, dtype=torch.float32).cuda()model = Lenet5()model = model.cuda()torch.onnx.export(model, input, "./lenet.onnx", verbose=True)if __name__ == '__main__':# main()model_onnx()

抓換onnx,遇到好幾種問題，用這種基本都解決了．

torch.onnx.export(model, # model being runinput, # model input (or a tuple for multiple inputs)"./xxxx.onnx",opset_version=10,verbose=False, # store the trained parameter weights inside the model filetraining=False,do_constant_folding=True,input_names=['input'],output_names=['output'])

2.3?.pth存儲為.wts

將模型權重按照key,value形式存儲為16進制文件, inference.py

import torch from torch import nn from lenet5 import Lenet5 import os import structdef main():print('cuda device count: ', torch.cuda.device_count())net = torch.load('lenet5.pth')net = net.to('cuda:0')net.eval()#print('model: ', net)#print('state dict: ', net.state_dict()['conv1.weight'])tmp = torch.ones(1, 1, 32, 32).to('cuda:0')#print('input: ', tmp)out = net(tmp)print('lenet out:', out)f = open("lenet5.wts", 'w')print('==net.state_dict().keys():', net.state_dict().keys())f.write("{}\n".format(len(net.state_dict().keys())))for k, v in net.state_dict().items():print('key: ', k)print('value: ', v.shape)vr = v.reshape(-1).cpu().numpy()f.write("{} {}".format(k, len(vr)))for vv in vr:# print('=vv:', vv)f.write(" ")# print(struct.pack(">f", float(vv)).hex())#f.write(struct.pack(">f", float(vv)).hex())f.write("\n")print('==f:', f)def test_struct():vv = 16print(struct.pack(">f", float(vv))) #if __name__ == '__main__':main()# test_struct()

2.4 .wts轉換成.engine與利用.engine推理

lenet.cpp?

#include <map> #include <chrono> #include <fstream> #include "NvInfer.h" #include "logging.h" #include "cuda_runtime_api.h"static const int INPUT_H=32; static const int INPUT_W=32; static const int BATCH_SIZE=32; static const int OUTPUT_SIZE=10; static const int INFER_NUMS=10000; const char* INPUT_BLOB_NAME = "data"; const char* OUTPUT_BLOB_NAME = "prob";using namespace nvinfer1; static Logger gLogger;#define CHECK(status) \do\{\auto ret = (status);\if (ret != 0)\{\std::cerr << "Cuda failure: " << ret << std::endl;\abort();\}\} while (0)std::map<std::string, Weights> loadWeights(const std::string file) {std::cout << "Loading weights: " << file << std::endl;std::map<std::string, Weights> weightMap;// Open weights filestd::ifstream input(file);assert(input.is_open() && "Unable to load weight file.");// Read number of weight blobsint32_t count;input >> count;assert(count > 0 && "Invalid weight map file.");while (count--){Weights wt{DataType::kFLOAT, nullptr, 0};uint32_t size;// Read name and type of blobstd::string name;input >> name >> std::dec >> size;wt.type = DataType::kFLOAT;// Load blobuint32_t* val = reinterpret_cast<uint32_t*>(malloc(sizeof(val) * size));for (uint32_t x = 0, y = size; x < y; ++x){input >> std::hex >> val[x];}wt.values = val;wt.count = size;weightMap[name] = wt;}return weightMap; }ICudaEngine* createLenetEngine(unsigned int maxBatchSize, IBuilder* builder, IBuilderConfig* config, DataType dt) {//開始定義網絡 0U無符號整型0INetworkDefinition* network = builder->createNetworkV2(0U);ITensor* input = network->addInput(INPUT_BLOB_NAME, dt, Dims3{1, INPUT_H, INPUT_W});assert(input);std::map<std::string, Weights> weightMap = loadWeights("../lenet5.wts");//載入權重放入weightMap// std::cout<<weightMap["conv1.weight"]<<std::endl; //卷積層IConvolutionLayer* conv1 = network->addConvolution(*input, 6, DimsHW{5, 5}, weightMap["conv1.weight"], weightMap["conv1.bias"]);//設置步長assert(conv1);conv1->setStrideNd(DimsHW{1, 1});//激活層IActivationLayer* relu1 = network->addActivation(*conv1->getOutput(0), ActivationType::kRELU);assert(relu1);//pooling層IPoolingLayer* pool1 = network->addPoolingNd(*relu1->getOutput(0), PoolingType::kAVERAGE, DimsHW{2, 2});assert(pool1);pool1->setStrideNd(DimsHW{2, 2});//卷積層IConvolutionLayer* conv2 = network->addConvolution(*pool1->getOutput(0), 16, DimsHW{5, 5}, weightMap["conv2.weight"], weightMap["conv2.bias"]);//設置步長assert(conv2);conv2->setStrideNd(DimsHW{1, 1}); //激活層IActivationLayer* relu2 = network->addActivation(*conv2->getOutput(0), ActivationType::kRELU);assert(relu2);//pooling層IPoolingLayer* pool2 = network->addPoolingNd(*relu2->getOutput(0), PoolingType::kAVERAGE, DimsHW{2, 2});assert(pool2);pool2->setStrideNd(DimsHW{2, 2});//全連接IFullyConnectedLayer* fc1 = network->addFullyConnected(*pool2->getOutput(0), 120, weightMap["fc1.weight"], weightMap["fc1.bias"]);assert(fc1);//激活層IActivationLayer* relu3 = network->addActivation(*fc1->getOutput(0), ActivationType::kRELU);assert(relu3);//全連接IFullyConnectedLayer* fc2 = network->addFullyConnected(*relu3->getOutput(0), 84, weightMap["fc2.weight"], weightMap["fc2.bias"]);assert(fc2);//激活層IActivationLayer* relu4 = network->addActivation(*fc2->getOutput(0), ActivationType::kRELU);assert(relu4);//全連接IFullyConnectedLayer* fc3 = network->addFullyConnected(*relu4->getOutput(0), OUTPUT_SIZE, weightMap["fc3.weight"], weightMap["fc3.bias"]);assert(fc3);//分類層ISoftMaxLayer *prob = network->addSoftMax(*fc3->getOutput(0));assert(prob);prob->getOutput(0)->setName(OUTPUT_BLOB_NAME);network->markOutput(*prob->getOutput(0));//構造enginebuilder->setMaxBatchSize(maxBatchSize);config->setMaxWorkspaceSize(1<<20);ICudaEngine* engine = builder->buildEngineWithConfig(*network, *config);//放入engine 所以network可以銷毀了network->destroy();// 釋放資源for (auto& mem : weightMap){free((void*) (mem.second.values));}return engine; } void APIToModel(unsigned int maxBatchSize, IHostMemory** modelStream) {//創建builderIBuilder* builder = createInferBuilder(gLogger);//網絡入口　類似pytorch的modelIBuilderConfig* config = builder->createBuilderConfig();//創建模型　搭建網絡層ICudaEngine* engine = createLenetEngine(maxBatchSize, builder, config, DataType::kFLOAT);assert(engine!=nullptr);//序列化engine(*modelStream)= engine->serialize();//銷毀對象 engine->destroy();builder->destroy();}void doInference(IExecutionContext& context, float* input, float *output, int batchSize) {//使用傳進來的context恢復engine。const ICudaEngine& engine = context.getEngine();//輸入輸出總共有兩個，做一下驗證assert(engine.getNbBindings()==2);//void void* buffers[2];//獲取與這個engine相關的輸入輸出tensor的索引sconst int inputIndex = engine.getBindingIndex(INPUT_BLOB_NAME);const int outputIndex = engine.getBindingIndex(OUTPUT_BLOB_NAME);//為輸入輸出tensor開辟顯存。CHECK(cudaMalloc(&buffers[inputIndex], batchSize * INPUT_H * INPUT_W * sizeof(float)));CHECK(cudaMalloc(&buffers[outputIndex], batchSize * OUTPUT_SIZE * sizeof(float)));//創建cuda流，用于管理數據復制，存取，和計算的并發操作cudaStream_t stream;CHECK(cudaStreamCreate(&stream));//從內存到顯存，input是讀入內存中的數據；buffers[inputIndex]是顯存上的存儲區域，用于存放輸入數據CHECK(cudaMemcpyAsync(buffers[inputIndex], input, batchSize * INPUT_H * INPUT_W * sizeof(float), cudaMemcpyHostToDevice, stream));// //啟動cuda核，異步執行推理計算context.enqueue(batchSize, buffers, stream, nullptr);//從顯存到內存，buffers[outputIndex]是顯存中的存儲區，存放模型輸出；output是內存中的數據CHECK(cudaMemcpyAsync(output, buffers[outputIndex], batchSize * OUTPUT_SIZE * sizeof(float), cudaMemcpyDeviceToHost, stream));//如果使用了多個cuda流，需要同步cudaStreamSynchronize(stream);// Release stream and buffers cudaStreamDestroy(stream);CHECK(cudaFree(buffers[inputIndex]));CHECK(cudaFree(buffers[outputIndex]));} int main(int argc, char ** argv) { if (argc!=2){ std::cerr << "arguments not right!" << std::endl;std::cerr << "./lenet -s // serialize model to plan file" << std::endl;std::cerr << "./lenet -d // deserialize plan file and run inference" << std::endl;return -1;}//序列化模型為.engine文件if(std::string(argv[1])=="-s"){ IHostMemory* modelStream{nullptr};//modelStream是一塊內存區域，用來保存序列化文件APIToModel(1, &modelStream);assert(modelStream!=nullptr);//變換為.engine文件std::ofstream p("lenet.engine");if (!p){std::cerr<<"can not open plan file"<<std::endl;return -1;}p.write(reinterpret_cast<const char *>(modelStream->data()), modelStream->size());// p.write(reinterpret_cast<const char*>(modelStream->data()), modelStream->size());//銷毀對象modelStream->destroy();}else if (std::string(argv[1])=="-d"){ char *trtModelStream{nullptr};size_t size{0};std::ifstream file("lenet.engine", std::ios::binary);if (file.good()) {file.seekg(0, file.end);size = file.tellg();file.seekg(0, file.beg);trtModelStream = new char[size];assert(trtModelStream);file.read(trtModelStream, size);file.close();}else{return -1;}//模擬數據float data[INPUT_H*INPUT_W];for (int i=0;i<INPUT_W*INPUT_H;i++){data[i] = 1.0;}//創建運行時環境IRuntime對象IRuntime* runtime = createInferRuntime(gLogger);assert(runtime !=nullptr);ICudaEngine* engine = runtime->deserializeCudaEngine(trtModelStream,size,nullptr);assert(engine !=nullptr);//創建上下文環境，主要用與inference函數中啟動cuda核IExecutionContext* context = engine->createExecutionContext();assert(context !=nullptr);//開始推理, 模擬推理1000次，存儲推理結果float prob[OUTPUT_SIZE];auto start = std::chrono::system_clock::now();//開始時間for (int i=0;i<INFER_NUMS;i++){ // std::cout<<"data[i]:"<<data[i]<<std::endl;doInference(*context, data, prob, 1); }auto end = std::chrono::system_clock::now();//結束時間std::cout << std::chrono::duration_cast<std::chrono::milliseconds>(end - start).count() << "ms" << std::endl;context->destroy();engine->destroy();runtime->destroy();std::cout<<"prob:";for (int i=0;i<OUTPUT_SIZE;i++){ std::cout<<prob[i]<<",";} }else{return -1;}return 0; }

CMakeLists.txt

cmake_minimum_required(VERSION 2.6)project(lenet)add_definitions(-std=c++11)set(TARGET_NAME "lenet")option(CUDA_USE_STATIC_CUDA_RUNTIME OFF) set(CMAKE_CXX_STANDARD 11) set(CMAKE_BUILD_TYPE Debug)include_directories(${PROJECT_SOURCE_DIR}/include) # include and link dirs of cuda and tensorrt, you need adapt them if yours are different # cuda include_directories(/usr/local/cuda/include) link_directories(/usr/local/cuda/lib64) # tensorrt include_directories(/usr/include/x86_64-linux-gnu) link_directories(/usr/lib/x86_64-linux-gnu)#tar包 tensorrt #include_directories(/red_detection/tensorrt_learn/software/TensorRT-7.0.0.11/include) #link_directories(/red_detection/tensorrt_learn/software/TensorRT-7.0.0.11/lib)FILE(GLOB SRC_FILES ${PROJECT_SOURCE_DIR}/lenet.cpp ${PROJECT_SOURCE_DIR}/include/*.h)add_executable(${TARGET_NAME} ${SRC_FILES}) target_link_libraries(${TARGET_NAME} nvinfer) target_link_libraries(${TARGET_NAME} cudart)add_definitions(-O2 -pthread)

./lenet -s 轉換成　.engine文件

./lenet -d 進行推理

推理時間:

可看出時間和torch的相比加快了至少4倍,而結果卻差不多。

一些很不錯的倉庫:

https://github.com/wang-xinyu/tensorrtx

https://github.com/zerollzeng/tiny-tensorrt

創作挑戰賽新人創作獎勵來咯，堅持創作打卡瓜分現金大獎

總結

以上是生活随笔為你收集整理的tensorrt基础知识+torch版lenet转c++ trt的全部內容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網站內容還不錯，歡迎將生活随笔推薦給好友。