在上一篇文章《從零實現一個3D目標檢測算法（2）：點云數據預處理》我們完成了對點云數據的預處理。

從本篇文章，我們開始正式實現PointPillars網絡，我們將按照本系列第一篇文章介紹的網絡具體結構來實現。

---

文章目錄

• • 1. Pytorch基本模塊
  • • 1.1 Empty模塊
    • 1.2 Sequential網絡模塊
  • 2. Pillar Feature Net 實現
  • • 2.1 VFE模塊
    • 2.2 Pillar Scatter模塊
  • 3.

1. Pytorch基本模塊

在工程上，為了方便對網絡的搭建和修改，通常會基于Pytorch實現兩個基本模塊，空網絡層模塊（Empty）和序列網絡模塊（Sequential），文件為pytorch_utils.py。

1.1 Empty模塊

顧名思義，就是構造一個什么也不做的網絡層，當然在這里的具體作用只是讓網絡更加完整（具體使用我們后面會見到），但在這里未參與計算，如果需要參與計算，也可對其進行修改，其構造也比較簡單，就是創建一個名為Empty的類，它會繼承nn.Module，代碼為：

import torch import torch.nn as nn import sys from collections import OrderedDictclass Empty(torch.nn.Module):def __init__(self, *args, **kwargs):super(Empty, self).__init__()def forward(self, *args, **kwargs):if len(args) == 1:return args[0]elif len(args) == 0:return Nonereturn args

在代碼中，*args用來將參數打包成tuple給函數體調用，**kwargs 打包關鍵字參數成dict給函數體調用。而在編寫函數中，參數arg、*args、**kwargs三個參數的位置是一定的。必須是(arg,*args,**kwargs)這個順序，否則程序會報錯，大家可以運行下面的代碼來看看輸出結果：

def function(arg,*args,**kwargs):print(arg,args,kwargs)function(6,7,8,9,a=1, b=2, c=3)

1.2 Sequential網絡模塊

Pytorch本身就帶有此模塊，這里之所以要單獨介紹，是因為在配置文件中，網絡的各種超參數是通過字典所給出的，這里重新編寫，方便我們后面加載網路超參數，這樣我們修改網絡模型時，只需要修改字典中的參數即可，我們創建一個名為Sequential的類，代碼為：

class Sequential(torch.nn.Module):"""A sequential container.Modules will be added to it in the order they are passed in the constructor.Alternatively, an ordered dict of modules can also be passed in."""def __init__(self, *args, **kwargs):super(Sequential, self).__init__()if len(args) == 1 and isinstance(args[0], OrderedDict):for key, module in args[0].items():self.add_module(key, module)else:for idx, module in enumerate(args):self.add_module(str(idx), module)for name, module in kwargs.items():if sys.version_info < (3, 6):raise ValueError("kwargs only supported in py36+")if name in self._modules:raise ValueError("name exists.")self.add_module(name, module)def __getitem__(self, idx):if not (-len(self) <= idx and idx < len(self)):raise IndexError('index {} is out of range'.format(idx))if idx < 0:idx += len(self)it = iter(self._modules.values())for i in range(idx):next(it)return next(it)def __len__(self):return len(self._modules)def add(self, module, name=None):if name is None:name = str(len(self._modules))if name in self._modules:raise KeyError("name exists")self.add_module(name, module)def forward(self, input):for module in self._modules.values():input = module(input)return input

下面提供 Sequential類使用的三個例子，其效果是等價的，大家可以運行看看輸出結果：

model = Sequential(nn.Conv2d(1,20,5), nn.ReLU(), nn.Conv2d(20,64,5), nn.ReLU())model = Sequential(OrderedDict([('conv1', nn.Conv2d(1,20,5)), ('relu1', nn.ReLU()),('conv2', nn.Conv2d(20,64,5)), ('relu2', nn.ReLU())]))model = Sequential(conv1=nn.Conv2d(1,20,5), relu1=nn.ReLU(),conv2=nn.Conv2d(20,64,5),relu2=nn.ReLU())

---

2. Pillar Feature Net 實現

現在，我們開始實現PointPillars網絡的第一部分Feature Net，這一部分主要是生成偽圖像，包括兩個模塊VFE模塊和Pillar Scatter模塊，文件為vfe_utils.py。

2.1 VFE模塊

VFE模塊的作用是將散亂無序的點云劃分為一個個Pillar，然后對其進行特征學習，如下圖所示。
首先我們導入需要的包，包括Pytorch以及上一節我們寫的Empty類。

import torch import torch.nn as nn import torch.nn.functional as F import sys sys.path.append('../') from ..model_utils.pytorch_utils import Empty

首先我們定義一個VoxelFeatureExtractor類，不過這里本身并不會進行任何操作：

class VoxelFeatureExtractor(nn.Module):def __init__(self, **kwargs):super().__init__()def get_output_feature_dim(self):raise NotImplementedErrordef forward(self, **kwargs):raise NotImplementedError

然后我們定義一個paddings_indicator函數。

def get_paddings_indicator(actual_num, max_num, axis=0):"""Create boolean mask by actually number of a padded tensor.Args:actual_num ([type]): [description]max_num ([type]): [description]Returns:[type]: [description]"""actual_num = torch.unsqueeze(actual_num, axis+1) print('actual_num shape is: ', actual_num.shape) # tiled_actual_num: [N, M, 1]max_num_shape = [1] * len(actual_num.shape)max_num_shape[axis+1] = -1max_num = torch.arange(max_num, dtype=torch.int, device=actual_num.device).view(max_num_shape)# tiled_actual_num: [[3,3,3,3,3], [4,4,4,4,4], [2,2,2,2,2]]# tiled_max_num: [[0,1,2,3,4], [0,1,2,3,4], [0,1,2,3,4]]paddings_indicator = actual_num.int() > max_num# paddings_indicator shape: [batch_size, max_num]return paddings_indicator

然后，我們定義一個PFNLayer類，這是一個簡化的PointNet層，輸入特征為10， 輸出特征為64，網絡是論文中提出的線性網絡，只有一層，代碼如下：

class PFNLayer(nn.Module):def __init__(self, in_channels, out_channels, use_norm=True, last_layer=False):"""Pillar Feature Net Layer.The Pillar Feature Net could be composed of a series of these layers, but the PointPillars paper resultsonly used a single PFNLayer.:param in_channels: <int>. Number of input channels. :param out_channels: <int>. Number of output channels. :param use_norm: <bool>. Whether to include BatchNorm. :param last_layer: <bool>. If last_layer, there is no concatenation of features."""super().__init__()self.name = 'PFNLayer'self.last_vfe = last_layer if not self.last_vfe:out_channels = out_channels // 2self.units = out_channels if use_norm: self.linear = nn.Linear(in_channels, self.units, bias=False)self.norm = nn.BatchNorm1d(self.units, eps=1e-3, momentum=0.01)else:self.linear = nn.Linear(in_channels, self.units, bias=True)self.norm = Empty(self.units)def forward(self, inputs):x = self.linear(inputs)total_points, voxel_points, channels = x.shapex = self.norm(x.view(-1, channels)).view(total_points, voxel_points, channels)x = F.relu(x)x_max = torch.max(x, dim=1, keepdim=True)[0] if self.last_vfe:return x_max else:x_repeat = x_max.repeat(1, inputs_shape[1], 1)x_concatenated = torch.cat([x, x_repeat], dim=2)return x_concatenated

下面我們將實現PillarFeatureNetOld2類，這里的作用是生成一個個Pillar，并將點云原來的4維特征$(x,y,z,r)$擴充為10維特征$(x,y,z,r,x_c,y_c,z_c,x_p,y_p,z_p)$，代碼如下：

class PillarFeatureNetOld2(VoxelFeatureExtractor):def __init__(self, num_input_features=4, use_norm=True, num_filters=(64, ), with_distance=False,voxel_size=(0.2, 0.2, 4), pc_range=(0, -40, -3, 70.4, 40, 1)):"""Pillar Feature Net.The network prepares the pillar features and performs forward pass through PFNLayers.:param num_input_features: <int>. Number of input features, either x, y, z or x, y, z, r. :param use_norm: <bool>. Whether to include BatchNorm.:param num_filters: (<int>: N). Number of features in each of the N PFNLayers.:param with_distance: <bool>. Whether to include Euclidean distance to points.:param voxel_size: (<float>: 3). Size of voxels, only utilize x and y size. :param pc_range: (<float>: 6). Point cloud range, only utilize x and y min. """super().__init__()self.name = 'PillarFeatureNetOld2'assert len(num_filters) > 0num_input_features +=6 if with_distance: num_input_features += 1 self.with_distance = with_distanceself.num_filters = num_filters# Create PillarFeatureNetOld layersnum_filters = [num_input_features] + list(num_filters) pfn_layers = []for i in range(len(num_filters) - 1): in_filters = num_filters[i] out_filters = num_filters[i+1] if i < len(num_filters) - 2:last_layer = Falseelse:last_layer = True pfn_layers.append(PFNLayer(in_filters, out_filters, use_norm, last_layer=last_layer))self.pfn_layers = nn.ModuleList(pfn_layers)# Need pillar (voxel) size and x/y offset in order to calculate pillar offsetself.vx = voxel_size[0]self.vy = voxel_size[1]self.vz = voxel_size[2]self.x_offset = self.vx / 2 + pc_range[0]self.y_offset = self.vy / 2 + pc_range[1]self.z_offset = self.vz / 2 + pc_range[2]def get_output_feature_dim(self):return self.num_filters[-1] # 64def forward(self, features, num_voxels, coords):""":param features: (N, max_points_of_each_voxel, 3 + C):param num_voxels: (N):param coors: (z ,y, x):return:"""dtype = features.dtype# Find distance of x, y, and z from cluster center (x, y, z mean)points_mean = features[:, :, :3].sum(dim=1, keepdim=True) / num_voxels.type_as(features).view(-1, 1, 1)print('points_mean shape is: ', points_mean.shape) f_cluster = features[:, :, :3] - points_mean# Find distance of x, y, and z from pillar centerf_center = torch.zeros_like(features[:, :, :3])f_center[:, :, 0] = features[:, :, 0] - (coords[:, 3].to(dtype).unsqueeze(1) * self.vx + self.x_offset)f_center[:, :, 1] = features[:, :, 1] - (coords[:, 2].to(dtype).unsqueeze(1) * self.vy + self.y_offset)f_center[:, :, 2] = features[:, :, 2] - (coords[:, 1].to(dtype).unsqueeze(1) * self.vz + self.z_offset)print('f_center shape is: ', f_center.shape) # Combine together feature decorationsfeatures_ls = [features, f_cluster, f_center]if self.with_distance: # Falsepoints_dist = torch.norm(features[:, :, :3], 2, 2, keepdim=True)features_ls.append(points_dist)features = torch.cat(features_ls, dim=-1)# The feature decorations were calculated without regard to whether pillar was empty. # Need to ensure that empty pillars remain set to zeros.voxel_count = features.shape[1]mask = get_paddings_indicator(num_voxels, voxel_count, axis = 0)mask = torch.unsqueeze(mask, -1).type_as(features)features *= maskprint('161 features shape is: ', features.shape) # Forward pass through PFNLayersfor pfn in self.pfn_layers:features = pfn(features)return features.squeeze()

---

2.2 Pillar Scatter模塊

此模塊生成偽造圖像，圖像維度為$(1,64,496,432)$，文件為pillar_scatter.py：

import torch import torch.nn as nnclass PointPillarsScatter(nn.Module):def __init__(self, input_channels=64, **kwargs):"""Point Pillar's Scatter.Converts learned features from dense tensor to sparse pseudo image.:param output_shape: ([int]: 4). Required output shape of features.:param num_input_features: <int>. Number of input features."""super().__init__()self.nchannels = input_channelsdef forward(self, voxel_features, coords, batch_size, **kwargs):output_shape = kwargs['output_shape']nz, ny, nx = output_shape# batch_canvas will be the final output.batch_canvas = []for batch_itt in range(batch_size):# Create the canvas for this samplecanvas = torch.zeros(self.nchannels, nz*nx*ny, dtype=voxel_features.dtype, \device=voxel_features.device)# Only include non-empty pillarsbatch_mask = coords[:, 0] == batch_itt this_coords = coords[batch_mask, :]indices = this_coords[:, 1].type(torch.long) * nz + this_coords[:, 2].type(torch.long) * nx + \this_coords[:, 3].type(torch.long)indices = indices.type(torch.long)voxels = voxel_features[batch_mask, :]voxels = voxels.t()# Now scatter the blob back to the canvas.canvas[:, indices] = voxels # Append to a list for later stacking.batch_canvas.append(canvas)# Stack to 3-dim tensor (batch-size, nchannels, nrows*ncols)batch_canvas = torch.stack(batch_canvas, 0)# Undo the column stacking to final 4-dim tensorbatch_canvas = batch_canvas.view(batch_size, self.nchannels * nz, ny, nx)return batch_canvas

至此，我們已經實現了特征網絡部分，生成了偽圖像。

---

3.

總結

以上是生活随笔為你收集整理的从零实现一个3D目标检测算法（3）：PointPillars主干网实现（持续更新中）的全部內容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網站內容還不錯，歡迎將生活随笔推薦給好友。