目錄

1.LSTM-Encoder-Decoder模型結(jié)構(gòu)及簡要解析

2.general LSTM Unit——使用pytorch nn.LSTM組件構(gòu)建LSTM_AE

1.定義Encoder:

2.定義Decoder:

3.定義LSTM_AE:

3.自定義LSTM Unit——自定義LSTM_AE

1.定義Encoder/Decoedr:

2.自定義LSTM_AE:

---

1.LSTM-Encoder-Decoder模型結(jié)構(gòu)及簡要解析

圖1? LSTM_AE模型示意圖（借圖）

LSTM_Encoder-Decoder結(jié)構(gòu)包含一個Encoder和一個Decoder，輸入以時間序列方式給出（圖中左側(cè)綠色節(jié)點），LSTM E你coder Unit對輸入計算得到輸出（圖中左側(cè)藍色方塊，需要注意示意圖中是按照時序方向展開的結(jié)構(gòu)，實際上其所有LSTM Unit都是同一個），得到編碼結(jié)果（圖中黃色節(jié)點）；將編碼結(jié)果當作Decoder的初始輸入，計算得到Decoder第一個輸出，將前一個Decoder單元的輸出結(jié)果當作下一個單元的輸入，依次進行計算得到結(jié)果...

---

進行下面的內(nèi)容前需要一些注釋：

general LSTM Unit是指【最原始版本的LSTM方式】在單元內(nèi)進行計算的【LSTM單元】

自定義LSTM Unit是指【自定義LSTM Unit中需要的計算】得到的【LSTM單元】

---

2.general LSTM Unit——使用pytorch nn.LSTM組件構(gòu)建LSTM_AE

首先給出general LSTM Unit中完成的計算：

圖2? 基本LSTM單元中進行的計算?

其思想簡單易懂：定義【輸入門，遺忘門，輸出們】，根據(jù)【輸入-輸入門，上一個時間步的記憶狀態(tài)c_t-1 - 遺忘門】得到當前時間步的記憶狀態(tài)c_t；根據(jù)數(shù)據(jù)對【輸出門-當前時間步的記憶狀態(tài)c_t】得到當前時間步的輸出h_t

如果基本LSTM單元就能滿足需要，可以使用nn.Module中提供的組件LSTM搭建模型：

1.定義Encoder:

class lstm_encoder(nn.Module):def __init__(self,input_size,hidden_size,num_layers=1):'''param:input_size:the feature dim of input Xhidden_size:the feature dim of hidden state hnum_layers:the num of stacked lstm layers'''super(lstm_encoder,self).__init__()self.input_size=input_sizeself.hidden_size=hidden_sizeself.num_layers=num_layersself.lstm=nn.LSTM(input_size=input_size,hidden_size=hidden_size,num_layers=num_layers)def forward(self,x):'''param:x:input of lstm unit,(seq_len,batch,feature_dim)output:seq_out:give all the hidden state in the sequencelast:the last h & c'''self.seq_out,self.last=self.lstm(x.view(x.shape[0],x.shape[1],self.input_size))return self.seq_out,self.last

?輸入以【batch_size,seq_length,input_feature_dim】或【seq_length,batch_size,input_feature_dim】方式給出，前者指定LSTM參數(shù)batch_first為True。

圖3? 理解LSTM在depth和t方向的展開(借圖)

LSTM參數(shù)中num_layers指定的是depth方向上的層數(shù)，而其在時序上的展開根據(jù)輸入的seq_length大小自動展開。

2.定義Decoder:

class lstm_decoder(nn.Module):def __init__(self,input_size,hidden_size,num_layers=1):'''param:input_size:the feature dim of input xhidden_size:the dim of hidden state hnum_layers:the stacked num of lstm units'''super(lstm_decoder,self).__init__()self.input_size=input_sizeself.hidden_size=hidden_sizeself.num_layers=num_layersself.lstm=nn.LSTM(input_size=input_size,hidden_size=hidden_size,num_layers=num_layers)self.linear=nn.Linear(hidden_size,input_size)def forward(self,x,encoder_hidden_state): #這里的輸入實際上只需要encoder的最后一個輸出，而將x輸入的原因是使用''' teacher force 模式訓(xùn)練模型時需要目標作為輸入param:x:should be 2D (batch_size,input_size)encoder_hidden_state:the output of encoderoutput:de_seq_out: all the hidden state in the sequencede_last:the last h &c'''self.de_seq_out,self.de_last=self.lstm(x.unsqueeze(0),encoder_hidden_state)self.de_seq_out=self.linear(self.de_seq_out.squeeze(0))return self.de_seq_out,self.de_last

Decoder中的輸入需要按照【batch_size,input_feature_dim】形狀給出，即seq_length=1，然后通過循環(huán)依次得到多個時間步的輸出。

需要對輸出進行線性重構(gòu)將其變成輸入特征大小

3.定義LSTM_AE:

class lstm_seq2seq(nn.Module):def __init__(self,input_size,hidden_size):'''param:input_size:the feature dim of input xhidden_size:the dim of hidden state size'''super(lstm_seq2seq,self).__init__()self.input_size=input_sizeself.hidden_size=hidden_sizeself.encoder=lstm_encoder(input_size=input_size,hidden_size=hidden_size)self.decoder=lstm_decoder(input_size=input_size,hidden_size=hidden_size)def forward(self,input_batch,target_len,batch_size):# 初始化encoder的隱狀態(tài)和輸出outputs = torch.zeros(target_len, batch_size, input_batch.shape[2]) # (s,b,f)(h, c) = self.encoder.init_hidden(batch_size)outputs = outputs.cuda()h = h.cuda()c = c.cuda()encoder_output, (h, c) = self.encoder(input_batch)# 設(shè)置decoder的隱狀態(tài)和輸入decoder_input = input_batch[-1, :, :] # (batch_size,input_size)deh = hdec = c# 時序性的預(yù)測,上個單元的輸出作為輸入，上個單元的隱狀態(tài)傳遞到下個單元for t in range(target_len):decoder_output, (deh, dec) = self.decoder(decoder_input, (deh, dec))outputs[t] = decoder_outputdecoder_input = decoder_outputreturn outputs

---

3.自定義LSTM Unit——自定義LSTM_AE

有時general LSTM Unit中的計算無法滿足模型的需要，此時需要自定義LSTM Unit中的計算：

1.定義Encoder/Decoedr:

class RelationalMemory(nn.Module):def __init__(#超參):super(RelationalMemory, self).__init__()#超參賦值#引入attention，根據(jù) c_t-1 計算 c_t-1_newdef multihead_attention(self, memory):"""Perform multi-head attention from 'Attention is All You Need'.Implementation of the attention mechanism fromhttps://arxiv.org/abs/1706.03762.Args:memory: Memory tensor to perform attention on.Returns:new_memory: New memory tensor."""qkv = self.qkv_projector(memory)qkv = self.qkv_layernorm(qkv)mem_slots = memory.shape[1] # denoted as N ([256,2,1536])# split the qkv to multiple heads H# [B, N, F] => [B, N, H, F/H]qkv_reshape = qkv.view(qkv.shape[0], mem_slots, self.num_heads, self.qkv_size) #[256,2,1,1536]# [B, N, H, F/H] => [B, H, N, F/H]qkv_transpose = qkv_reshape.permute(0, 2, 1, 3)#[256,1,2,1536]# [B, H, N, key_size], [B, H, N, key_size], [B, H, N, value_size]q, k, v = torch.split(qkv_transpose, [self.key_size, self.key_size, self.value_size], -1)q = q * (self.key_size ** -0.5)# make it [B, H, N, N]dot_product = torch.matmul(q, k.permute(0, 1, 3, 2))weights = F.softmax(dot_product, dim=-1)# output is [B, H, N, V]output = torch.matmul(weights, v)# [B, H, N, V] => [B, N, H, V] => [B, N, H*V]output_transpose = output.permute(0, 2, 1, 3).contiguous()new_memory = output_transpose.view((output_transpose.shape[0], output_transpose.shape[1], -1))new_memory = self.drop(new_memory)return new_memory#根據(jù) c_t-1_new 和 h_t-1 計算 c_t 和 h_t def forward(self, inputs, memory, treat_input_as_matrix=False):"""Forward step of the relational memory core.Args:inputs: Tensor input.memory: Memory output from the previous time step.treat_input_as_matrix: Optional, whether to treat `input` as a sequenceof matrices. Default to False, in which case the input is flattenedinto a vector.Returns:output: This time step's output.next_memory: The next version of memory to use."""memory_plus_input = torch.cat([memory, inputs_reshape], dim=1)next_memory = self.multihead_attention(memory_plus_input)# cut out the concatenated input vectors from the original memory slotsn = inputs_reshape.shape[1]next_memory = next_memory[:, :-n, :]if self.gate_style == 'unit' or self.gate_style == 'memory':# these gates are sigmoid-applied ones for equation 7input_gate, forget_gate = self.create_gates(inputs_reshape, memory)# equation 7 calculationnext_memory = input_gate * torch.tanh(next_memory)next_memory += forget_gate * memoryreturn next_memory

總而言之：

1.引入attention機制，根據(jù) c_t-1 計算 c_t-1_new

2.自定義LSTM Unit中的計算，根據(jù)?c_t-1_new和? h_t-1? 得到? c_t 和 h_t?

2.自定義LSTM_AE:

class VAE(nn.Module):"""docstring for VAE"""def __init__(self, feature_size,project_size, max_frames,hidden_size, nbits):super(VAE, self).__init__()self.Encoder = RelationalMemory(mem_slots=1, head_size=hidden_size, input_size=feature_size, num_heads=1, num_blocks=1, forget_bias=1., input_bias=0,gate_style='unit', attention_mlp_layers=2, key_size=None, return_all_outputs=False)self.Decoder = RelationalMemory(mem_slots=1, head_size=hidden_size, input_size=feature_size, num_heads=1, num_blocks=1, forget_bias=1., input_bias=0,gate_style='unit', attention_mlp_layers=2, key_size=None, return_all_outputs=False)...def forward(self,videos,anchor1,anchor2,anchor3):...outputs = []for i in range(self.max_frame):e_memory = self.Encoder(v[:, i, :], e_memory)bb = self.binary_restore(e_memory).transpose(0,1)[0]binary = self.activation(bb)m = self.binary_embed(binary)d_memory = self.Decoder.initial_state(batch_size, trainable=True).to(device)# d_memory = anchor1.unsqueeze(1)for i in range(self.max_frame):d_memory = self.Decoder(m,d_memory)restored_frame = self.frame_restore(d_memory)# restored_frame = self.drop(restored_frame)m = restored_frameoutputs.append(restored_frame)outputs = torch.cat([o for o in outputs], 1).contiguous() return outputs, bb

總結(jié)

以上是生活随笔為你收集整理的使用pytorch构建LSTM_AE模型的两种方式的全部內(nèi)容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網(wǎng)站內(nèi)容還不錯，歡迎將生活随笔推薦給好友。