參考ConvNeXt中的Layer Normalization(LN) - 海斌的文章 - 知乎
https://zhuanlan.zhihu.com/p/481901798

Layer Norm本來(lái)是一個(gè)樣本norm自己，如圖所示：

也就是說(shuō)，在[C,H,W]維進(jìn)行歸一化
而ConvNeXt中是這樣：

也就是在C的維度歸一化，即單一像素的所有channel做歸一化。

兩者實(shí)現(xiàn)方式上有所不同。
可以用F.layer_norm實(shí)現(xiàn)，也可以用nn.layer_norm實(shí)現(xiàn)，這里選用F.layer_norm

看ConvNeXt的實(shí)現(xiàn) class LayerNorm2d(nn.LayerNorm):"""LayerNorm on channels for 2d images.Args:num_channels (int): The number of channels of the input tensor.eps (float): a value added to the denominator for numerical stability.Defaults to 1e-5.elementwise_affine (bool): a boolean value that when set to ``True``,this module has learnable per-element affine parameters initializedto ones (for weights) and zeros (for biases). Defaults to True."""def __init__(self, num_channels: int, **kwargs) -> None:super().__init__(num_channels, **kwargs)self.num_channels = self.normalized_shape[0]def forward(self, x):assert x.dim() == 4, 'LayerNorm2d only supports inputs with shape ' \f'(N, C, H, W), but got tensor with shape {x.shape}'return F.layer_norm(x.permute(0, 2, 3, 1), self.normalized_shape, self.weight,self.bias, self.eps).permute(0, 3, 1, 2)

LayerNorm2d繼承的nn.LayerNorm，在初始化時(shí)調(diào)用了父類(lèi)的init函數(shù)，我們來(lái)看看父類(lèi)是干了啥？

def __init__(self, normalized_shape: _shape_t, eps: float = 1e-5, elementwise_affine: bool = True,device=None, dtype=None) -> None:***self.normalized_shape = tuple(normalized_shape)

實(shí)際上這里僅僅是把num_channels，通道數(shù)，存入了self.normalized_shape，并且用一個(gè)tuple保存。
所以說(shuō)，關(guān)鍵是forward里面的F.layer_norm

我們看看傳入的參數(shù)
第一個(gè)參數(shù)是轉(zhuǎn)置后的x，也就是N,C,H,W轉(zhuǎn)置為N,H,W,C，把通道數(shù)放在最后一個(gè)維度，過(guò)F.layer_norm，normalized_shape實(shí)際上就是前面init函數(shù)里傳入的num_channels，就是一個(gè)數(shù)字。

接下來(lái)我們看看F.layer_norm的用法：
其實(shí)和nn.layernorm基本是一樣的，只是不用事先實(shí)例化，這樣的話(huà)參數(shù)要一起傳進(jìn)去。
對(duì)于nn.layernorm來(lái)說(shuō)，

args: normalized_shape (int or list or torch.Size): input shape from an expected input of size If a single integer is used, it is treated as a singleton list, and this module willnormalize over the last dimension which is expected to be of that specific size.eps: a value added to the denominator for numerical stability. Default: 1e-5

上面這段話(huà)的意思就是說(shuō)，這個(gè)normalized_shape可以是一個(gè)數(shù)，也可以是一個(gè)list，如果是一個(gè)數(shù)，則默認(rèn)在最后一維歸一化，且這個(gè)數(shù)需要等于最后一維的維度；如果是一個(gè)list，則這個(gè)list需要匹配從后往前的不同維度的維數(shù)
eg.

a.shape = (100. 96, 8, 8)#也就是100個(gè)96維的8x8的feature

如果是普通的layer norm，normalized_shape=[96, 8, 8]
如果這個(gè)參數(shù)是8，則在最后一維進(jìn)行歸一化
如果希望在所有point的channel歸一化，如ConvNeXt
則先轉(zhuǎn)置，把channel轉(zhuǎn)到最后一維，然后normalized_shape=num_channel即可

那么ConvNeXt的layer norm如何改成普通的layer norm呢？
如代碼所示：

class LayerNorm2d(nn.LayerNorm):def __init__(self, num_channels: int, **kwargs) -> None:super().__init__(num_channels, **kwargs)self.num_channels = self.normalized_shape[0]def forward(self, x):return F.layer_norm(x.permute(0, 2, 3, 1), self.normalized_shape, self.weight,self.bias, self.eps).permute(0, 3, 1, 2)

實(shí)際上只需要改forward里即可，即不轉(zhuǎn)置，且normalized_shape參數(shù)傳入x的最后三3維度

class LayerNorm2d(nn.LayerNorm):def __init__(self, num_channels: int, **kwargs) -> None:super().__init__(num_channels, **kwargs)self.num_channels = self.normalized_shape[0]def forward(self, x):return F.layer_norm(x, x.shape[1:], self.weight,self.bias, self.eps))

總結(jié)

以上是生活随笔為你收集整理的Layer Norm的全部?jī)?nèi)容，希望文章能夠幫你解決所遇到的問(wèn)題。

如果覺(jué)得生活随笔網(wǎng)站內(nèi)容還不錯(cuò)，歡迎將生活随笔推薦給好友。