一.總體結構

由于rnn等循環神經網絡有時序依賴，導致無法并行計算，而Transformer主體框架是一個encoder-decoder結構，去掉了RNN序列結構，完全基于attention和全連接。同時為了彌補詞與詞之間時序信息，將詞位置embedding成向量輸入模型．

二．每一步拆分

1.padding mask

對于輸入序列一般我們都要進行padding補齊，也就是說設定一個統一長度N，在較短的序列后面填充0到長度為N。對于那些補零的數據來說，我們的attention機制不應該把注意力放在這些位置上，所以我們需要進行一些處理。具體的做法是，把這些位置的值加上一個非常大的負數(負無窮)，這樣經過softmax后，這些位置的權重就會接近0。Transformer的padding mask實際上是一個張量，每個值都是一個Boolean，值為false的地方就是要進行處理的地方。

def padding_mask(seq_k, seq_q):len_q = seq_q.size(1)print('=len_q:', len_q)# `PAD` is 0pad_mask_ = seq_k.eq(0)#每句話的pad maskprint('==pad_mask_:', pad_mask_)pad_mask = pad_mask_.unsqueeze(1).expand(-1, len_q, -1) # shape [B, L_q, L_k]#作用于attention的maskprint('==pad_mask', pad_mask)return pad_maskdef debug_padding_mask():Bs = 2inputs_len = np.random.randint(1, 5, Bs).reshape(Bs, 1)print('==inputs_len:', inputs_len)vocab_size = 6000 # 詞匯數max_seq_len = int(max(inputs_len))# vocab_size = int(max(inputs_len))x = np.zeros((Bs, max_seq_len), dtype=np.int)for s in range(Bs):for j in range(inputs_len[s][0]):x[s][j] = j + 1x = torch.LongTensor(torch.from_numpy(x))print('x.shape', x.shape)mask = padding_mask(seq_k=x, seq_q=x)print('==mask:', mask.shape)if __name__ == '__main__':debug_padding_mask()

2.Position encoding

其也叫做Position embedding，由于Transformer模型沒有使用RNN，故Position encoding(PE)的目的就是實現文本序列的順序（或者說位置）信息而出現的。

代碼實現如下：輸入batch內的詞位置，輸出是batch內的每個詞的位置embedding向量.

class PositionalEncoding(nn.Module):def __init__(self, d_model, max_seq_len):"""初始化Args:d_model: 一個標量。模型的維度，論文默認是512max_seq_len: 一個標量。文本序列的最大長度"""super(PositionalEncoding, self).__init__()# 根據論文給的公式，構造出PE矩陣position_encoding = np.array([[pos / np.power(10000, 2.0 * (j // 2) / d_model) for j in range(d_model)]for pos in range(max_seq_len)]).astype(np.float32)# 偶數列使用sin，奇數列使用cosposition_encoding[:, 0::2] = np.sin(position_encoding[:, 0::2])position_encoding[:, 1::2] = np.cos(position_encoding[:, 1::2])# 在PE矩陣的第一行，加上一行全是0的向量，代表這`PAD`的positional encoding# 在word embedding中也經常會加上`UNK`，代表位置單詞的word embedding，兩者十分類似# 那么為什么需要這個額外的PAD的編碼呢？很簡單，因為文本序列的長度不一，我們需要對齊，# 短的序列我們使用0在結尾補全，我們也需要這些補全位置的編碼，也就是`PAD`對應的位置編碼position_encoding = torch.from_numpy(position_encoding) # [max_seq_len, model_dim]# print('==position_encoding.shape:', position_encoding.shape)pad_row = torch.zeros([1, d_model])position_encoding = torch.cat((pad_row, position_encoding)) # [max_seq_len+1, model_dim]# print('==position_encoding.shape:', position_encoding.shape)# 嵌入操作，+1是因為增加了`PAD`這個補全位置的編碼，# Word embedding中如果詞典增加`UNK`，我們也需要+1?？窗?#xff0c;兩者十分相似self.position_encoding = nn.Embedding(max_seq_len + 1, d_model)self.position_encoding.weight = nn.Parameter(position_encoding,requires_grad=False)def forward(self, input_len):"""神經網絡的前向傳播。Args:input_len: 一個張量，形狀為[BATCH_SIZE, 1]。每一個張量的值代表這一批文本序列中對應的長度。Returns:返回這一批序列的位置編碼，進行了對齊。"""# 找出這一批序列的最大長度max_len = torch.max(input_len)tensor = torch.cuda.LongTensor if input_len.is_cuda else torch.LongTensor# 對每一個序列的位置進行對齊，在原序列位置的后面補上0# 這里range從1開始也是因為要避開PAD(0)的位置input_pos = tensor([list(range(1, len + 1)) + [0] * (max_len - len) for len in input_len])# print('==input_pos:', input_pos)#pad補齊# print('==input_pos.shape:', input_pos.shape)#[bs, max_len]return self.position_encoding(input_pos)def debug_posion():"""d_model:模型的維度"""bs = 16x_sclar = np.random.randint(1, 30, bs).reshape(bs, 1)model = PositionalEncoding(d_model=512, max_seq_len=int(max(x_sclar)))x = torch.from_numpy(x_sclar)#[bs, 1]print('===x:', x)print('====x.shape', x.shape)out = model(x)print('==out.shape:', out.shape)#[bs, max_seq_len, model_dim] if __name__ == '__main__':debug_posion()

3.Scaled dot-product attention實現

Q,K,V:可看成一個batch內詞的三個embedding向量和矩陣相乘得到的,而這個矩陣就是需要學習的，通過Ｑ,K獲取attention score作用于V上獲取加權的V.這樣一句話的不同詞就獲取了不同關注度．注意，Ｑ,Ｋ,Ｖ這 3 個向量一般比原來的詞向量的長度更小。假設這 3 個向量的長度是64 ，而原始的詞向量或者最終輸出的向量的長度是 512（Ｑ,Ｋ,Ｖ這 3 個向量的長度，和最終輸出的向量長度，是有倍數關系的）

上圖中，有兩個詞向量：Thinking 的詞向量 x1 和 Machines 的詞向量 x2。以 x1 為例，X1 乘以 WQ 得到 q1，q1 就是 X1 對應的 Query 向量。同理，X1 乘以 WK 得到 k1，k1 是 X1 對應的 Key 向量；X1 乘以 WV 得到 v1，v1 是 X1 對應的 Value 向量。

對應代碼實現:?

class ScaledDotProductAttention(nn.Module):"""Scaled dot-product attention mechanism."""def __init__(self, attention_dropout=0.5):super(ScaledDotProductAttention, self).__init__()self.dropout = nn.Dropout(attention_dropout)self.softmax = nn.Softmax(dim=2)def forward(self, q, k, v, scale=None, attn_mask=None):"""前向傳播.Args:q: Queries張量，形狀為[B, L_q, D_q]k: Keys張量，形狀為[B, L_k, D_k]v: Values張量，形狀為[B, L_v, D_v]，一般來說就是kscale: 縮放因子，一個浮點標量attn_mask: Masking張量，形狀為[B, L_q, L_k]Returns:上下文張量和attetention張量"""attention = torch.bmm(q, k.transpose(1, 2)) # [B, sequence, sequence]print('===attention.shape', attention)if scale:attention = attention * scaleif attn_mask is not None:# 給需要mask的地方設置一個負無窮attention = attention.masked_fill_(attn_mask, -np.inf)print('===attention.shape', attention)attention = self.softmax(attention) # [B, sequence, sequence]# print('===attention.shape', attention.shape)attention = self.dropout(attention) # [B, sequence, sequence]# print('===attention.shape', attention.shape)context = torch.bmm(attention, v) # [B, sequence, dim]return context, attentiondef debug_scale_attention():model = ScaledDotProductAttention()# B, L_q, D_q = 32, 100, 128B, L_q, D_q = 2, 4, 10pading_mask = torch.tensor([[[False, False, False, False],[False, False, False, False],[False, False, False, False],[False, False, False, False]],[[False, False, True, True],[False, False, True, True],[False, False, True, True],[False, False, True, True]]])q, k, v = torch.rand(B, L_q, D_q), torch.rand(B, L_q, D_q), torch.rand(B, L_q, D_q)print('==q.shape:', q.shape)print('====k.shape', k.shape)print('==v.shape:', v.shape)out = model(q, k, v, attn_mask=pading_mask) if __name__ == '__main__':debug_scale_attention()

注意q和k,v維度可以不一樣

import torch.nn as nn d_model = 256 nhead = 8 multihead_attn1 = nn.MultiheadAttention(d_model, nhead, dropout=0.1) src1 = torch.rand((256, 1, 256)) src2 = torch.rand((1024, 1, 256)) src2_key_padding_mask = torch.zeros((1, 1024)) src12 = multihead_attn1(query=src1,key=src2,value=src2, attn_mask=None,key_padding_mask=src2_key_padding_mask)[0]print('=src12.shape:', src12.shape)key_padding_mask = torch.zeros((1, 1024)) num_heads = 8 q = torch.rand((256, 1, 256)) tgt_len, bsz, embed_dim = q.size() head_dim = embed_dim // num_heads q = q.contiguous().view(tgt_len, bsz * num_heads, head_dim).transpose(0, 1) print('==q.shape:', q.shape) k = torch.rand((1024, 1, 256)) v = torch.rand((1024, 1, 256)) k = k.contiguous().view(-1, bsz * num_heads, head_dim).transpose(0, 1) src_len = k.size(1) v = v.contiguous().view(-1, bsz * num_heads, head_dim).transpose(0, 1) print('==k.shape:', k.shape) print('==v.shape:', v.shape) attn_output_weights = torch.bmm(q, k.transpose(1, 2)) print('==attn_output_weights.shape:', attn_output_weights.shape) if key_padding_mask is not None:attn_output_weights = attn_output_weights.view(bsz, num_heads, tgt_len, src_len)attn_output_weights = attn_output_weights.masked_fill(key_padding_mask.unsqueeze(1).unsqueeze(2),float('-inf'),)attn_output_weights = attn_output_weights.view(bsz * num_heads, tgt_len, src_len) attn_output_weights = F.softmax(attn_output_weights, dim=-1) print('==attn_output_weights.shape:', attn_output_weights.shape) attn_output = torch.bmm(attn_output_weights, v) print('==attn_output.shape:', attn_output.shape) attn_output = attn_output.transpose(0, 1).contiguous().view(tgt_len, bsz, embed_dim) print('==attn_output.shape:', attn_output.shape)

?

4.Multi-Head Attention

? ? ? ? ? ? ? ? ? ? ??

其中Ｈ就是Multi-Head，可看出首先對Ｑ,K,V進行一次線性變換，然后進行切分，對每一個切分的部分進行attention(Scaled dot-product attention),然后最后將結果進行合并．有一種類似通道加權的感覺．

對應代碼實現:?

class MultiHeadAttention(nn.Module):def __init__(self, model_dim=512, num_heads=8, dropout=0.0):"""model_dim:詞向量維度num_heads:頭個數"""super(MultiHeadAttention, self).__init__()self.dim_per_head = model_dim // num_heads#split個數也就是每個head要處理維度self.num_heads = num_headsself.linear_k = nn.Linear(model_dim, self.dim_per_head * num_heads)self.linear_v = nn.Linear(model_dim, self.dim_per_head * num_heads)self.linear_q = nn.Linear(model_dim, self.dim_per_head * num_heads)self.dot_product_attention = ScaledDotProductAttention(dropout)self.linear_final = nn.Linear(model_dim, model_dim)self.dropout = nn.Dropout(dropout)self.layer_norm = nn.LayerNorm(model_dim)def forward(self, key, value, query, attn_mask=None):residual = query# [B, sequence, model_dim]dim_per_head = self.dim_per_headnum_heads = self.num_headsbatch_size = key.size(0)# linear projectionkey = self.linear_k(key)# [B, sequence, model_dim]value = self.linear_v(value)# [B, sequence, model_dim]query = self.linear_q(query)# [B, sequence, model_dim]# print('===key.shape:', key.shape)# print('===value.shape:', value.shape)# print('==query.shape:', query.shape)# split by headskey = key.view(batch_size * num_heads, -1, dim_per_head)# [B* num_heads, sequence, model_dim//*num_heads]value = value.view(batch_size * num_heads, -1, dim_per_head)# [B* num_heads, sequence, model_dim//*num_heads]query = query.view(batch_size * num_heads, -1, dim_per_head)# [B* num_heads, sequence, model_dim//*num_heads]# print('===key.shape:', key.shape)# print('===value.shape:', value.shape)# print('==query.shape:', query.shape)if attn_mask:attn_mask = attn_mask.repeat(num_heads, 1, 1)# scaled dot product attentionscale = (key.size(-1) // num_heads) ** -0.5context, attention = self.dot_product_attention(query, key, value, scale, attn_mask)# print('===context.shape', context.shape)# [B* num_heads, sequence, model_dim//*num_heads]# print('===attention.shape', attention.shape)# [B* num_heads, sequence, sequence]# concat headscontext = context.view(batch_size, -1, dim_per_head * num_heads)# [B, sequence, model_dim]# print('===context.shape', context.shape)# final linear projectionoutput = self.linear_final(context)# [B, sequence, model_dim]# print('===context.shape', context.shape)# dropoutoutput = self.dropout(output)# add residual and norm layeroutput = self.layer_norm(residual + output)# [B, sequence, model_dim]# print('==output.shape:', output.shape)return output, attention def debug_mutil_head_attention():model = MultiHeadAttention()B, L_q, D_q = 32, 100, 512q, k, v = torch.rand(B, L_q, D_q), torch.rand(B, L_q, D_q), torch.rand(B, L_q, D_q)# print('==q.shape:', q.shape)# [B, sequence, model_dim]# print('====k.shape', k.shape)# [B, sequence, model_dim]# print('==v.shape:', v.shape)# [B, sequence, model_dim]out, _ = model(q, k, v)# [B, sequence, model_dim]print('==out.shape:', out.shape) if __name__ == '__main__':debug_mutil_head_attention()

5.Positional-wise feed forward network（前饋神經網絡層）

如上圖中畫框就是其所在,

代碼：

#Position-wise Feed Forward Networks class PositionalWiseFeedForward(nn.Module):def __init__(self, model_dim=512, ffn_dim=2048, dropout=0.0):"""model_dim:詞向量的維度ffn_dim:卷積輸出的維度"""super(PositionalWiseFeedForward, self).__init__()self.w1 = nn.Conv1d(model_dim, ffn_dim, 1)self.w2 = nn.Conv1d(ffn_dim, model_dim, 1)self.dropout = nn.Dropout(dropout)self.layer_norm = nn.LayerNorm(model_dim)def forward(self, x):#[B, sequence, model_dim]output = x.transpose(1, 2)#[B, model_dim, sequence]# print('===output.shape:', output.shape)output = self.w2(F.relu(self.w1(output)))#[B, model_dim, sequence]output = self.dropout(output.transpose(1, 2))#[B, sequence, model_dim]# add residual and norm layeroutput = self.layer_norm(x + output)return outputdef debug_PositionalWiseFeedForward():B, L_q, D_q = 32, 100, 512x = torch.rand(B, L_q, D_q)model = PositionalWiseFeedForward()out = model(x)print('==out.shape:', out.shape) if __name__ == '__main__':debug_PositionalWiseFeedForward()

6.encoder實現

其共有6層4,5的結構,可看出q k v 均來自同一文本.

def sequence_mask(seq):batch_size, seq_len = seq.size()mask = torch.triu(torch.ones((seq_len, seq_len), dtype=torch.uint8),diagonal=1)mask = mask.unsqueeze(0).expand(batch_size, -1, -1) # [B, L, L]return maskdef padding_mask(seq_k, seq_q):len_q = seq_q.size(1)# `PAD` is 0pad_mask = seq_k.eq(0)pad_mask = pad_mask.unsqueeze(1).expand(-1, len_q, -1) # shape [B, L_q, L_k]return pad_maskclass EncoderLayer(nn.Module):"""一個encode的layer實現"""def __init__(self, model_dim=512, num_heads=8, ffn_dim=2018, dropout=0.0):super(EncoderLayer, self).__init__()self.attention = MultiHeadAttention(model_dim, num_heads, dropout)self.feed_forward = PositionalWiseFeedForward(model_dim, ffn_dim, dropout)def forward(self, inputs, attn_mask=None):# self attention# [B, sequence, model_dim] [B* num_heads, sequence, sequence]context, attention = self.attention(inputs, inputs, inputs, attn_mask)# feed forward networkoutput = self.feed_forward(context) # [B, sequence, model_dim]return output, attentionclass Encoder(nn.Module):"""編碼器實現 總共6層"""def __init__(self,vocab_size,max_seq_len,num_layers=6,model_dim=512,num_heads=8,ffn_dim=2048,dropout=0.0):super(Encoder, self).__init__()self.encoder_layers = nn.ModuleList([EncoderLayer(model_dim, num_heads, ffn_dim, dropout) for _ in range(num_layers)])self.seq_embedding = nn.Embedding(vocab_size + 1, model_dim, padding_idx=0)self.pos_embedding = PositionalEncoding(model_dim, max_seq_len)# [bs, max_seq_len] [bs, 1]def forward(self, inputs, inputs_len):output = self.seq_embedding(inputs) # [bs, max_seq_len, model_dim]print('========output.shape', output.shape)# 加入位置信息embeddingoutput += self.pos_embedding(inputs_len) # [bs, max_seq_len, model_dim]print('========output.shape', output.shape)self_attention_mask = padding_mask(inputs, inputs)attentions = []for encoder in self.encoder_layers:output, attention = encoder(output, attn_mask=None)# output, attention = encoder(output, self_attention_mask)attentions.append(attention)return output, attentionsdef debug_encoder():Bs = 16inputs_len = np.random.randint(1, 30, Bs).reshape(Bs, 1)# print('==inputs_len:', inputs_len) # 模擬獲取每個詞的長度vocab_size = 6000 # 詞匯數max_seq_len = int(max(inputs_len))# vocab_size = int(max(inputs_len))x = np.zeros((Bs, max_seq_len), dtype=np.int)for s in range(Bs):for j in range(inputs_len[s][0]):x[s][j] = j+1x = torch.LongTensor(torch.from_numpy(x))inputs_len = torch.from_numpy(inputs_len)#[Bs, 1]model = Encoder(vocab_size=vocab_size, max_seq_len=max_seq_len)# x = torch.LongTensor([list(range(1, max_seq_len + 1)) for _ in range(Bs)])#模擬每個單詞print('==x.shape:', x.shape)print(x)model(x, inputs_len=inputs_len)if __name__ == '__main__':debug_encoder()

7.Sequence Mask

樣本：“我/愛/機器/學習”和 "i/ love /machine/ learning"

訓練：
7.1. 把“我/愛/機器/學習”embedding后輸入到encoder里去，最后一層的encoder最終輸出的outputs [10, 512]（假設我們采用的embedding長度為512，而且batch size = 1),此outputs 乘以新的參數矩陣，可以作為decoder里每一層用到的K和V；

7.2. 將<bos>作為decoder的初始輸入，將decoder的最大概率輸出詞 A1和‘i’做cross entropy計算error。

7.3. 將<bos>，"i" 作為decoder的輸入，將decoder的最大概率輸出詞 A2 和‘love’做cross entropy計算error。

7.4. 將<bos>，"i"，"love" 作為decoder的輸入，將decoder的最大概率輸出詞A3和'machine' 做cross entropy計算error。

7.5. 將<bos>，"i"，"love "，"machine" 作為decoder的輸入，將decoder最大概率輸出詞A4和‘learning’做cross entropy計算error。

7.6. 將<bos>，"i"，"love "，"machine"，"learning" 作為decoder的輸入，將decoder最大概率輸出詞A5和終止符</s>做cross entropy計算error。

可看出上述訓練過程是挨個單詞串行進行的，故引入sequence mask,用于并行訓練.

作用生成

8.decoder實現

也是循環6層,可以看出decoder的soft-attention，q來自于decoder，k和v來自于encoder。它體現的是encoder對decoder的加權貢獻。

class DecoderLayer(nn.Module):"""解碼器的layer實現"""def __init__(self, model_dim, num_heads=8, ffn_dim=2048, dropout=0.0):super(DecoderLayer, self).__init__()self.attention = MultiHeadAttention(model_dim, num_heads, dropout)self.feed_forward = PositionalWiseFeedForward(model_dim, ffn_dim, dropout)# [B, sequence, model_dim]　[B, sequence, model_dim]def forward(self,dec_inputs,enc_outputs,self_attn_mask=None,context_attn_mask=None):# self attention, all inputs are decoder inputs# [B, sequence, model_dim] [B* num_heads, sequence, sequence]dec_output, self_attention = self.attention(key=dec_inputs, value=dec_inputs, query=dec_inputs, attn_mask=self_attn_mask)# context attention# query is decoder's outputs, key and value are encoder's inputs# [B, sequence, model_dim] [B* num_heads, sequence, sequence]dec_output, context_attention = self.attention(key=enc_outputs, value=enc_outputs, query=dec_output, attn_mask=context_attn_mask)# decoder's output, or contextdec_output = self.feed_forward(dec_output) # [B, sequence, model_dim]return dec_output, self_attention, context_attentionclass Decoder(nn.Module):"""解碼器"""def __init__(self,vocab_size,max_seq_len,num_layers=6,model_dim=512,num_heads=8,ffn_dim=2048,dropout=0.0):super(Decoder, self).__init__()self.num_layers = num_layersself.decoder_layers = nn.ModuleList([DecoderLayer(model_dim, num_heads, ffn_dim, dropout) for _ inrange(num_layers)])self.seq_embedding = nn.Embedding(vocab_size + 1, model_dim, padding_idx=0)self.pos_embedding = PositionalEncoding(model_dim, max_seq_len)def forward(self, inputs, inputs_len, enc_output, context_attn_mask=None):output = self.seq_embedding(inputs)output += self.pos_embedding(inputs_len)print('==output.shape:', output.shape)self_attention_padding_mask = padding_mask(inputs, inputs)seq_mask = sequence_mask(inputs)self_attn_mask = torch.gt((self_attention_padding_mask + seq_mask), 0)self_attentions = []context_attentions = []for decoder in self.decoder_layers:# [B, sequence, model_dim] [B* num_heads, sequence, sequence] [B* num_heads, sequence, sequence]output, self_attn, context_attn = decoder(output, enc_output, self_attn_mask=None, context_attn_mask=None)self_attentions.append(self_attn)context_attentions.append(context_attn)return output, self_attentions, context_attentionsdef debug_decoder():Bs = 2model_dim = 512vocab_size = 6000 #詞匯數inputs_len = np.random.randint(1, 5, Bs).reshape(Bs, 1)#batch里每句話的單詞個數inputs_len = torch.from_numpy(inputs_len) # [Bs, 1]max_seq_len = int(max(inputs_len))x = np.zeros((Bs, max_seq_len), dtype=np.int)for s in range(Bs):for j in range(inputs_len[s][0]):x[s][j] = j + 1x = torch.LongTensor(torch.from_numpy(x))#模擬每個單詞# x = torch.LongTensor([list(range(1, max_seq_len + 1)) for _ in range(Bs)])print('==x:', x)print('==x.shape:', x.shape)model = Decoder(vocab_size=vocab_size, max_seq_len=max_seq_len, model_dim=model_dim)enc_output = torch.rand(Bs, max_seq_len, model_dim) #[B, sequence, model_dim]print('==enc_output.shape:', enc_output.shape)out, self_attentions, context_attentions = model(inputs=x, inputs_len=inputs_len, enc_output=enc_output)print('==out.shape:', out.shape)#[B, sequence, model_dim]print('==len(self_attentions):', len(self_attentions), self_attentions[0].shape)print('==len(context_attentions):', len(context_attentions), context_attentions[0].shape)if __name__ == '__main__':debug_decoder()

9.transformer

將encoder和decoder組合起來即可．

class Transformer(nn.Module):def __init__(self,src_vocab_size,src_max_len,tgt_vocab_size,tgt_max_len,num_layers=6,model_dim=512,num_heads=8,ffn_dim=2048,dropout=0.2):super(Transformer, self).__init__()self.encoder = Encoder(src_vocab_size, src_max_len, num_layers, model_dim,num_heads, ffn_dim, dropout)self.decoder = Decoder(tgt_vocab_size, tgt_max_len, num_layers, model_dim,num_heads, ffn_dim, dropout)self.linear = nn.Linear(model_dim, tgt_vocab_size, bias=False)self.softmax = nn.Softmax(dim=2)def forward(self, src_seq, src_len, tgt_seq, tgt_len):context_attn_mask = padding_mask(tgt_seq, src_seq)print('==context_attn_mask.shape', context_attn_mask.shape)output, enc_self_attn = self.encoder(src_seq, src_len)output, dec_self_attn, ctx_attn = self.decoder(tgt_seq, tgt_len, output, context_attn_mask)output = self.linear(output)output = self.softmax(output)return output, enc_self_attn, dec_self_attn, ctx_attn def debug_transoform():Bs = 4#需要翻譯的encode_inputs_len = np.random.randint(1, 10, Bs).reshape(Bs, 1)src_vocab_size = 6000 # 詞匯數encode_max_seq_len = int(max(encode_inputs_len))encode_x = np.zeros((Bs, encode_max_seq_len), dtype=np.int)for s in range(Bs):for j in range(encode_inputs_len[s][0]):encode_x[s][j] = j + 1encode_x = torch.LongTensor(torch.from_numpy(encode_x))#翻譯的結果decode_inputs_len = np.random.randint(1, 10, Bs).reshape(Bs, 1)target_vocab_size = 5000 # 詞匯數decode_max_seq_len = int(max(decode_inputs_len))decode_x = np.zeros((Bs, decode_max_seq_len), dtype=np.int)for s in range(Bs):for j in range(decode_inputs_len[s][0]):decode_x[s][j] = j + 1decode_x = torch.LongTensor(torch.from_numpy(decode_x))encode_inputs_len = torch.from_numpy(encode_inputs_len) # [Bs, 1]decode_inputs_len = torch.from_numpy(decode_inputs_len) # [Bs, 1]model = Transformer(src_vocab_size=src_vocab_size, src_max_len=encode_max_seq_len, tgt_vocab_size=target_vocab_size, tgt_max_len=decode_max_seq_len)# x = torch.LongTensor([list(range(1, max_seq_len + 1)) for _ in range(Bs)])#模擬每個單詞print('==encode_x.shape:', encode_x.shape)print('==decode_x.shape:', decode_x.shape)model(encode_x, encode_inputs_len, decode_x, decode_inputs_len) if __name__ == '__main__':debug_transoform()

10.總結

（１）：相比lstm而言，其能夠實現并行，而lstm由于依賴上一時刻只能串行輸出；
（２）：利用self-attention將每個詞之間距離縮短為1，大大緩解了長距離依賴問題，所以網絡相比lstm能夠堆疊得更深；
（３）：Transformer可以同時融合前后位置的信息，而雙向LSTM只是簡單的將兩個方向的結果相加，嚴格來說仍然是單向的；
（４）：完全基于attention的Transformer，可以表達字與字之間的相關關系，可解釋性更強；
（５）：Transformer位置信息只能依靠position encoding，故當語句較短時效果不一定比lstm好；
（６）：attention計算量為O(n^2), n為文本長度，計算量較大；
（７）：相比CNN能夠捕獲全局的信息，而不是局部信息，所以CNN缺乏對數據的整體把握。

三.CV中的self-attention

介紹完了nlp的self-attention，現在介紹CV中的,如下圖所示。

1.feature map通過1*1卷積獲得,q,k,v三個向量,q與k轉置相乘得到attention矩陣，進行softmax歸一化到0到1,在作用于V,得到每個像素的加權.

2.softmax

3,加權求和

import torch import torch.nn as nn import torch.nn.functional as Fclass Self_Attn(nn.Module):""" Self attention Layer"""def __init__(self, in_dim):super(Self_Attn, self).__init__()self.chanel_in = in_dimself.query_conv = nn.Conv2d(in_channels=in_dim, out_channels=in_dim // 8, kernel_size=1)self.key_conv = nn.Conv2d(in_channels=in_dim, out_channels=in_dim // 8, kernel_size=1)self.value_conv = nn.Conv2d(in_channels=in_dim, out_channels=in_dim, kernel_size=1)self.gamma = nn.Parameter(torch.zeros(1))self.softmax = nn.Softmax(dim=-1)def forward(self, x):"""inputs :x : input feature maps( B * C * W * H)returns :out : self attention value + input featureattention: B * N * N (N is Width*Height)"""m_batchsize, C, width, height = x.size()proj_query = self.query_conv(x).view(m_batchsize, -1, width * height).permute(0, 2, 1) # B*N*Cproj_key = self.key_conv(x).view(m_batchsize, -1, width * height) # B*C*Nenergy = torch.bmm(proj_query, proj_key) # batch的matmul B*N*Nattention = self.softmax(energy) # B * (N) * (N)proj_value = self.value_conv(x).view(m_batchsize, -1, width * height) # B * C * Nout = torch.bmm(proj_value, attention.permute(0, 2, 1)) # B*C*Nout = out.view(m_batchsize, C, width, height) # B*C*H*Wout = self.gamma * out + xreturn out, attentiondef debug_attention():attention_module = Self_Attn(in_dim=128)#B,C,H,Wx = torch.rand((2, 128, 100, 100))attention_module(x)if __name__ == '__main__':debug_attention()

參考：

舉個例子講下transformer的輸入輸出細節及其他 - 知乎

The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time.

machine-learning-notes/transformer_pytorch.ipynb at master · luozhouyang/machine-learning-notes · GitHub

總結

以上是生活随笔為你收集整理的从attention到Transformer+CV中的self-attention的全部內容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網站內容還不錯，歡迎將生活随笔推薦給好友。