TPLinker 联合抽取 实体链接方式+源码分析
關(guān)系抽取–TPLinker： https://blog.csdn.net/weixin_42223207/article/details/116425447
Tagging
TPLinker模型需要對(duì)關(guān)系三元組(subject, relation, object)進(jìn)行手動(dòng)Tagging，過程分為三部分：
（1）entity head to entity tail (EH-TO-ET)
（2）subject head to object head (SH-to-OH)
（3）subject tail to object tail (ST-to-OT)
標(biāo)記示例見下圖，EH-TO-ET用紫色表示，SH-to-OH用紅色表示，ST-to-OT用藍(lán)色表示。
論文筆記：路由器 TPLinker 也來做關(guān)系抽取：https://zhuanlan.zhihu.com/p/304104571
關(guān)系抽取之TPLinker解讀加源碼分析：https://zhuanlan.zhihu.com/p/342300800
關(guān)于解碼過程
通過實(shí)體抽取得到字典D后，遍歷關(guān)系，通過關(guān)系得到有關(guān)系的兩個(gè)實(shí)體的尾部E， 再通過關(guān)系得到有關(guān)系的兩個(gè)實(shí)體的頭部，結(jié)合字典D得到這兩個(gè)實(shí)體尾部set(s),set(o)(此為真實(shí)的標(biāo)簽），再通過set(s),set(o)在不在E里面，來判斷是否成功抽取了一條三元組。
其中相關(guān)的公式：
文章的大概內(nèi)容進(jìn)行了解讀，現(xiàn)在到了代碼解讀環(huán)節(jié)：
輸入數(shù)據(jù)部分：
輸入的seq的長(zhǎng)度為seq_len ，獲取句子的最大長(zhǎng)度
train and valid max token num
max_tok_num = 0 all_data = train_data + valid_data for sample in all_data:tokens = tokenize(sample["text"])max_tok_num = max(max_tok_num, len(tokens)) max_tok_num # 獲取句子的最大長(zhǎng)度接下來對(duì)文本超過長(zhǎng)度的進(jìn)行劃窗處理：
if max_tok_num > hyper_parameters["max_seq_len"]: # 截?cái)嚅L(zhǎng)度train_data = preprocessor.split_into_short_samples(train_data, hyper_parameters["max_seq_len"], sliding_len = hyper_parameters["sliding_len"], encoder = config["encoder"] #超過長(zhǎng)度則滑動(dòng)窗口得到新的樣本) valid_data = preprocessor.split_into_short_samples(valid_data, hyper_parameters["max_seq_len"], sliding_len = hyper_parameters["sliding_len"], encoder = config["encoder"])接下來看劃窗的具體操作
def split_into_short_samples(self, sample_list, max_seq_len, sliding_len = 50, encoder = "BERT", data_type = "train"):new_sample_list = []for sample in tqdm(sample_list, desc = "Splitting into subtexts"):text_id = sample["id"]text = sample["text"]tokens = self._tokenize(text)tok2char_span = self._get_tok2char_span_map(text) #返回句子中單詞的偏移量# sliding at token levelsplit_sample_list = []for start_ind in range(0, len(tokens), sliding_len): #sliding_len 滑動(dòng)窗口的大小，if encoder == "BERT": # if use bert, do not split a word into two sampleswhile "##" in tokens[start_ind]:start_ind -= 1end_ind = start_ind + max_seq_len # 結(jié)束的長(zhǎng)度char_span_list = tok2char_span[start_ind:end_ind] #截?cái)?/span>char_level_span = [char_span_list[0][0], char_span_list[-1][1]] #第一個(gè)詞到最后一個(gè)詞的長(zhǎng)度sub_text = text[char_level_span[0]:char_level_span[1]]#原始文本截?cái)?/span>new_sample = {"id": text_id,"text": sub_text,"tok_offset": start_ind, #token的偏移量"char_offset": char_level_span[0], #每個(gè)字符的偏移量}if data_type == "test": # test setif len(sub_text) > 0:split_sample_list.append(new_sample)else: # train or valid dataset, only save spo and entities in the subtext# sposub_rel_list = []for rel in sample["relation_list"]:subj_tok_span = rel["subj_tok_span"]obj_tok_span = rel["obj_tok_span"]# if subject and object are both in this subtext, add this spo to new sampleif subj_tok_span[0] >= start_ind and subj_tok_span[1] <= end_ind \and obj_tok_span[0] >= start_ind and obj_tok_span[1] <= end_ind: new_rel = copy.deepcopy(rel)new_rel["subj_tok_span"] = [subj_tok_span[0] - start_ind, subj_tok_span[1] - start_ind] # start_ind: 單詞級(jí)別的偏移量new_rel["obj_tok_span"] = [obj_tok_span[0] - start_ind, obj_tok_span[1] - start_ind]new_rel["subj_char_span"][0] -= char_level_span[0] # 字符級(jí)別的偏移量new_rel["subj_char_span"][1] -= char_level_span[0]new_rel["obj_char_span"][0] -= char_level_span[0]new_rel["obj_char_span"][1] -= char_level_span[0]sub_rel_list.append(new_rel)# entitysub_ent_list = []for ent in sample["entity_list"]:tok_span = ent["tok_span"]# if entity in this subtext, add the entity to new sampleif tok_span[0] >= start_ind and tok_span[1] <= end_ind: new_ent = copy.deepcopy(ent)new_ent["tok_span"] = [tok_span[0] - start_ind, tok_span[1] - start_ind]new_ent["char_span"][0] -= char_level_span[0]new_ent["char_span"][1] -= char_level_span[0]sub_ent_list.append(new_ent)# eventif "event_list" in sample:sub_event_list = []for event in sample["event_list"]:trigger_tok_span = event["trigger_tok_span"]if trigger_tok_span[1] > end_ind or trigger_tok_span[0] < start_ind:continuenew_event = copy.deepcopy(event)new_arg_list = []for arg in new_event["argument_list"]:if arg["tok_span"][0] >= start_ind and arg["tok_span"][1] <= end_ind:new_arg_list.append(arg)new_event["argument_list"] = new_arg_listsub_event_list.append(new_event)new_sample["event_list"] = sub_event_list # maybe emptynew_sample["entity_list"] = sub_ent_list # maybe emptynew_sample["relation_list"] = sub_rel_list # maybe emptysplit_sample_list.append(new_sample)# all segments covered, no need to continueif end_ind > len(tokens):breaknew_sample_list.extend(split_sample_list)return new_sample_list輸入數(shù)據(jù)，DataMaker4Bert中定義：
class DataMaker4Bert():def __init__(self, tokenizer, handshaking_tagger):self.tokenizer = tokenizerself.handshaking_tagger = handshaking_taggerdef get_indexed_data(self, data, max_seq_len, data_type = "train"): #index轉(zhuǎn)換為dataindexed_samples = []for ind, sample in tqdm(enumerate(data), desc = "Generate indexed train or valid data"):text = sample["text"]# codes for bert inputcodes = self.tokenizer.encode_plus(text, return_offsets_mapping = True, add_special_tokens = False,max_length = max_seq_len, truncation = True,pad_to_max_length = True)# taggingspots_tuple = Noneif data_type != "test":spots_tuple = self.handshaking_tagger.get_spots(sample) #獲取實(shí)體,頭，尾標(biāo)簽# get codesinput_ids = torch.tensor(codes["input_ids"]).long()attention_mask = torch.tensor(codes["attention_mask"]).long()token_type_ids = torch.tensor(codes["token_type_ids"]).long()tok2char_span = codes["offset_mapping"]sample_tp = (sample,input_ids,attention_mask,token_type_ids,tok2char_span,spots_tuple,)indexed_samples.append(sample_tp) return indexed_samples輸入的是tokenizer和handshakingtagger，tokenizer為bert等一系列模型的標(biāo)準(zhǔn)輸入，而get_spots函數(shù)獲取了實(shí)體，頭，尾的標(biāo)簽，具體看下代碼
def get_spots(self, sample):'''entity spot and tail_rel spot: (span_pos1, span_pos2, tag_id)head_rel spot: (rel_id, span_pos1, span_pos2, tag_id)'''ent_matrix_spots, head_rel_matrix_spots, tail_rel_matrix_spots = [], [], [] for rel in sample["relation_list"]:subj_tok_span = rel["subj_tok_span"]obj_tok_span = rel["obj_tok_span"]ent_matrix_spots.append((subj_tok_span[0], subj_tok_span[1] - 1, self.tag2id_ent["ENT-H2T"])) #sub token的[起始位置，尾部位置，實(shí)體標(biāo)簽（1)]ent_matrix_spots.append((obj_tok_span[0], obj_tok_span[1] - 1, self.tag2id_ent["ENT-H2T"]))# obj token的[起始位置，尾部位置，實(shí)體標(biāo)簽（1)]if subj_tok_span[0] <= obj_tok_span[0]:head_rel_matrix_spots.append((self.rel2id[rel["predicate"]], subj_tok_span[0], obj_tok_span[0], self.tag2id_head_rel["REL-SH2OH"]))#【關(guān)系類別，實(shí)體_1 頭部，實(shí)體_2頭部，關(guān)系標(biāo)簽（1）】else:head_rel_matrix_spots.append((self.rel2id[rel["predicate"]], obj_tok_span[0], subj_tok_span[0], self.tag2id_head_rel["REL-OH2SH"]))#【關(guān)系類別，實(shí)體_1 頭部，實(shí)體_2頭部，關(guān)系標(biāo)簽（2）】if subj_tok_span[1] <= obj_tok_span[1]:tail_rel_matrix_spots.append((self.rel2id[rel["predicate"]], subj_tok_span[1] - 1, obj_tok_span[1] - 1, self.tag2id_tail_rel["REL-ST2OT"]))#【關(guān)系類別，實(shí)體_1 尾部，實(shí)體_2尾部，關(guān)系標(biāo)簽（1）】else:tail_rel_matrix_spots.append((self.rel2id[rel["predicate"]], obj_tok_span[1] - 1, subj_tok_span[1] - 1, self.tag2id_tail_rel["REL-OT2ST"]))#【關(guān)系類別，實(shí)體_1 尾部，實(shí)體_2尾部，關(guān)系標(biāo)簽（2）】return ent_matrix_spots, head_rel_matrix_spots, tail_rel_matrix_spots 獲取輸入的數(shù)據(jù)indexed_train_data = data_maker.get_indexed_data(train_data, max_seq_len) #獲取輸入 # index_train_data = data_maker.get_indexed_data(train_test_data,max_seq_len) indexed_valid_data = data_maker.get_indexed_data(valid_data, max_seq_len) tokenizer = BertTokenizerFast.from_pretrained(config["bert_path"], add_special_tokens = False, do_lower_case = False) data_maker = DataMaker4Bert(tokenizer, handshaking_tagger) #(sample,input_ids，attention_mask,token_type_ids,tok2char_span,spots_tuple,)接下來則是定義HandshakingTaggingScheme
max_seq_len = min(max_tok_num, hyper_parameters["max_seq_len"]) #max_len 長(zhǎng)度 rel2id = json.load(open(rel2id_path, "r", encoding = "utf-8")) handshaking_tagger = HandshakingTaggingScheme(rel2id = rel2id, max_seq_len = max_seq_len) #初始化查看具體的定義
class HandshakingTaggingScheme(object):"""docstring for HandshakingTaggingScheme"""def __init__(self, rel2id, max_seq_len):super(HandshakingTaggingScheme, self).__init__()self.rel2id = rel2idself.id2rel = {ind:rel for rel, ind in rel2id.items()}self.tag2id_ent = { #實(shí)體頭尾"O": 0,"ENT-H2T": 1, # entity head to entity tail}self.id2tag_ent = {id_:tag for tag, id_ in self.tag2id_ent.items()}self.tag2id_head_rel = { #sub,obj頭對(duì)頭標(biāo)識(shí)1，obj頭對(duì)sub頭標(biāo)識(shí)2"O": 0,"REL-SH2OH": 1, # subject head to object head"REL-OH2SH": 2, # object head to subject head}self.id2tag_head_rel = {id_:tag for tag, id_ in self.tag2id_head_rel.items()}self.tag2id_tail_rel = {"O": 0, "REL-ST2OT": 1, # subject tail to object tail"REL-OT2ST": 2, # object tail to subject tail}self.id2tag_tail_rel = {id_:tag for tag, id_ in self.tag2id_tail_rel.items()}# mapping shaking sequence and matrixself.matrix_size = max_seq_len# e.g. [(0, 0), (0, 1), (0, 2), (1, 1), (1, 2), (2, 2)] #轉(zhuǎn)換成矩陣上三角矩陣平鋪self.shaking_ind2matrix_ind = [(ind, end_ind) for ind in range(self.matrix_size) for end_ind in list(range(self.matrix_size))[ind:]]self.matrix_ind2shaking_ind = [[0 for i in range(self.matrix_size)] for j in range(self.matrix_size)]for shaking_ind, matrix_ind in enumerate(self.shaking_ind2matrix_ind): #上三角矩陣，上三角每個(gè)元素儲(chǔ)存著上三角鋪平序列的相對(duì)應(yīng)的位置序號(hào)self.matrix_ind2shaking_ind[matrix_ind[0]][matrix_ind[1]] = shaking_ind這里比較關(guān)鍵的是shaking_ind2matrix_ind，與matrix_ind2shaking_ind，其中shaking_ind2matrix_ind如下所示，是一個(gè)上三角鋪平序列
[(0, 0), (0, 1), (0, 2), (0, 3), (0, 4), (0, 5), (0, 6), (0, 7), (0, 8), (0, 9), (0, 10), (0, 11), (0, 12), (0, 13), …]
而matrix_ind2shaking_ind為優(yōu)化前的二維矩陣，其中上三角每個(gè)元素儲(chǔ)存著上三角鋪平序列的相對(duì)應(yīng)的位置序號(hào)
[[0, 0, 0, 0, 0, 0, 0, 0, 0, …], [0, 0, 0, 0, 0, 0, 0, 0, 0, …], [0, 0, 0, 0, 0, 0, 0, 0, 0, …], [0, 0, 0, 0, 0, 0, 0, 0, 0, …], [0, 0, 0, 0, 0, 0, 0, 0, 0, …], [0, 0, 0, 0, 0, 0, 0, 0, 0, …], [0, 0, 0, 0, 0, 0, 0, 0, 0, …], [0, 0, 0, 0, 0, 0, 0, 0, 0, …], [0, 0, 0, 0, 0, 0, 0, 0, 0, …], [0, 0, 0, 0, 0, 0, 0, 0, 0, …], [0, 0, 0, 0, 0, 0, 0, 0, 0, …], [0, 0, 0, 0, 0, 0, 0, 0, 0, …], [0, 0, 0, 0, 0, 0, 0, 0, 0, …], [0, 0, 0, 0, 0, 0, 0, 0, 0, …], …]
最后組成上三角矩陣（打印的結(jié)果沒有填滿）
[[0, 1, 2, 3, 4, 5, 6, 7, 8, …], [0, 0, 0, 0, 0, 0, 0, 0, 0, …], [0, 0, 0, 0, 0, 0, 0, 0, 0, …], [0, 0, 0, 0, 0, 0, 0, 0, 0, …], [0, 0, 0, 0, 0, 0, 0, 0, 0, …], [0, 0, 0, 0, 0, 0, 0, 0, 0, …], [0, 0, 0, 0, 0, 0, 0, 0, 0, …], [0, 0, 0, 0, 0, 0, 0, 0, 0, …], [0, 0, 0, 0, 0, 0, 0, 0, 0, …], [0, 0, 0, 0, 0, 0, 0, 0, 0, …], [0, 0, 0, 0, 0, 0, 0, 0, 0, …], [0, 0, 0, 0, 0, 0, 0, 0, 0, …], [0, 0, 0, 0, 0, 0, 0, 0, 0, …], [0, 0, 0, 0, 0, 0, 0, 0, 0, …], …]
載入數(shù)據(jù)
查看DataLoader的返回值：
def generate_batch(self, batch_data, data_type = "train"):sample_list = []input_ids_list = []attention_mask_list = []token_type_ids_list = [] tok2char_span_list = []ent_spots_list = []head_rel_spots_list = []tail_rel_spots_list = []for tp in batch_data:sample_list.append(tp[0])input_ids_list.append(tp[1])attention_mask_list.append(tp[2]) token_type_ids_list.append(tp[3]) tok2char_span_list.append(tp[4])if data_type != "test":ent_matrix_spots, head_rel_matrix_spots, tail_rel_matrix_spots = tp[5]ent_spots_list.append(ent_matrix_spots)head_rel_spots_list.append(head_rel_matrix_spots)tail_rel_spots_list.append(tail_rel_matrix_spots)# @specific: indexed by bert tokenizerbatch_input_ids = torch.stack(input_ids_list, dim = 0)batch_attention_mask = torch.stack(attention_mask_list, dim = 0)batch_token_type_ids = torch.stack(token_type_ids_list, dim = 0)batch_ent_shaking_tag, batch_head_rel_shaking_tag, batch_tail_rel_shaking_tag = None, None, Noneif data_type != "test":batch_ent_shaking_tag = self.handshaking_tagger.sharing_spots2shaking_tag4batch(ent_spots_list)batch_head_rel_shaking_tag = self.handshaking_tagger.spots2shaking_tag4batch(head_rel_spots_list)batch_tail_rel_shaking_tag = self.handshaking_tagger.spots2shaking_tag4batch(tail_rel_spots_list)return sample_list, \batch_input_ids, batch_attention_mask, batch_token_type_ids, tok2char_span_list, \batch_ent_shaking_tag, batch_head_rel_shaking_tag, batch_tail_rel_shaking_tag其中比較重要的是batch_ent_shaking_tag, batch_head_rel_shaking_tag, batch_tail_rel_shaking_tag這三個(gè)函數(shù),在實(shí)體batch_ent_shaking_tag中用到了handshaking_tagger的sharing_spots2shaking_tag4batch的函數(shù)
def sharing_spots2shaking_tag4batch(self, batch_spots):'''convert spots to batch shaking seq tag因長(zhǎng)序列的stack是費(fèi)時(shí)操作，所以寫這個(gè)函數(shù)用作生成批量shaking tag如果每個(gè)樣本生成一條shaking tag再stack，一個(gè)32的batch耗時(shí)1s，太昂貴spots: [(start_ind, end_ind, tag_id), ], for entiyreturn: batch_shake_seq_tag: (batch_size, shaking_seq_len)'''shaking_seq_len = self.matrix_size * (self.matrix_size + 1) // 2batch_shaking_seq_tag = torch.zeros(len(batch_spots), shaking_seq_len).long()for batch_id, spots in enumerate(batch_spots):for sp in spots:shaking_ind = self.matrix_ind2shaking_ind[sp[0]][sp[1]] #在矩陣中找到實(shí)體的的start_int，跟end_int的位置值tag_id = sp[2]batch_shaking_seq_tag[batch_id][shaking_ind] = tag_id #鋪平上三角矩陣，標(biāo)注序列中的實(shí)體標(biāo)識(shí)1的位置return batch_shaking_seq_tag在關(guān)系batch_head_rel_shaking_tag與batch_tail_rel_shaking_tag中用到了spots2shaking_tag4batch
def spots2shaking_tag4batch(self, batch_spots):
‘’’
convert spots to batch shaking seq tag
spots: [(rel_id, start_ind, end_ind, tag_id), ], for head relation and tail_relation
return:
batch_shake_seq_tag: (batch_size, rel_size, shaking_seq_len)
‘’’
shaking_seq_len = self.matrix_size * (self.matrix_size + 1) // 2
batch_shaking_seq_tag = torch.zeros(len(batch_spots), len(self.rel2id), shaking_seq_len).long()
for batch_id, spots in enumerate(batch_spots):
for sp in spots:
shaking_ind = self.matrix_ind2shaking_ind[sp[1]][sp[2]]
tag_id = sp[3]
rel_id = sp[0]
batch_shaking_seq_tag[batch_id][rel_id][shaking_ind] = tag_id
return batch_shaking_seq_tag
跟實(shí)體類似，只是多了關(guān)系總數(shù)，然后整個(gè)函數(shù)的返回為
return sample_list,
batch_input_ids, tok2char_span_list,
batch_ent_shaking_tag, batch_head_rel_shaking_tag, batch_tail_rel_shaking_tag
初始化模型
rel_extractor = TPLinkerBert(encoder,
len(rel2id),
hyper_parameters[“shaking_type”],
hyper_parameters[“inner_enc_type”],
hyper_parameters[“dist_emb_size”],
hyper_parameters[“ent_add_dist”],
hyper_parameters[“rel_add_dist”],
)
模型的具體定義
self.head_rel_fc_list與self.tail_rel_fc_list相同，是關(guān)系的一個(gè)全連接（標(biāo)簽有三個(gè)0,1,2），每一種關(guān)系有一個(gè)獨(dú)立的MLP層，self.head_rel_fc_list是列表的形式。
上面為關(guān)系和實(shí)體關(guān)系預(yù)測(cè)，實(shí)體和各個(gè)關(guān)系都經(jīng)過了mlp層，我們假設(shè)有5種關(guān)系，則會(huì)有11層MLP即為：1個(gè)實(shí)體預(yù)測(cè)層+（1個(gè)頭部層+1個(gè)尾部層）*5
def forward(self, input_ids, attention_mask, token_type_ids):# input_ids, attention_mask, token_type_ids: (batch_size, seq_len)context_outputs = self.encoder(input_ids, attention_mask, token_type_ids) # 0 last_hidden 1 pooled# last_hidden_state: (batch_size, seq_len, hidden_size)last_hidden_state = context_outputs[0]# shaking_hiddens: (batch_size, 1 + ... + seq_len, hidden_size)shaking_hiddens = self.handshaking_kernel(last_hidden_state) #鋪平上三角矩陣shaking_hiddens4ent = shaking_hiddensshaking_hiddens4rel = shaking_hiddens# add distance embeddings if it is setif self.dist_emb_size != -1:# set self.dist_embbedingshidden_size = shaking_hiddens.size()[-1]if self.dist_embbedings is None:dist_emb = torch.zeros([self.dist_emb_size, hidden_size]).to(shaking_hiddens.device)for d in range(self.dist_emb_size):for i in range(hidden_size):if i % 2 == 0:dist_emb[d][i] = math.sin(d / 10000**(i / hidden_size))else:dist_emb[d][i] = math.cos(d / 10000**((i - 1) / hidden_size))seq_len = input_ids.size()[1]dist_embbeding_segs = []for after_num in range(seq_len, 0, -1): #鋪平dist_embbeding_segs.append(dist_emb[:after_num, :])self.dist_embbedings = torch.cat(dist_embbeding_segs, dim = 0)if self.ent_add_dist:shaking_hiddens4ent = shaking_hiddens + self.dist_embbedings[None,:,:].repeat(shaking_hiddens.size()[0], 1, 1)if self.rel_add_dist:shaking_hiddens4rel = shaking_hiddens + self.dist_embbedings[None,:,:].repeat(shaking_hiddens.size()[0], 1, 1)# if self.dist_emb_size != -1 and self.ent_add_dist: # shaking_hiddens4ent = shaking_hiddens + self.dist_embbedings[None,:,:].repeat(shaking_hiddens.size()[0], 1, 1) # else: # shaking_hiddens4ent = shaking_hiddens # if self.dist_emb_size != -1 and self.rel_add_dist: # shaking_hiddens4rel = shaking_hiddens + self.dist_embbedings[None,:,:].repeat(shaking_hiddens.size()[0], 1, 1) # else: # shaking_hiddens4rel = shaking_hiddensent_shaking_outputs = self.ent_fc(shaking_hiddens4ent) #實(shí)體預(yù)測(cè)，（0，1）head_rel_shaking_outputs_list = []nn.ModuleList()for fc in self.head_rel_fc_list:head_rel_shaking_outputs_list.append(fc(shaking_hiddens4rel)) #對(duì)每一種關(guān)系頭進(jìn)行分類tail_rel_shaking_outputs_list = []for fc in self.tail_rel_fc_list: #對(duì)每一種關(guān)系尾進(jìn)行分類tail_rel_shaking_outputs_list.append(fc(shaking_hiddens4rel))head_rel_shaking_outputs = torch.stack(head_rel_shaking_outputs_list, dim = 1) #n種關(guān)系拼接在一起tail_rel_shaking_outputs = torch.stack(tail_rel_shaking_outputs_list, dim = 1) #n種關(guān)系拼接在一起return ent_shaking_outputs, head_rel_shaking_outputs, tail_rel_shaking_outputsent_shaking_outputs為實(shí)體預(yù)測(cè)，head_rel_shaking_outputs_list對(duì)關(guān)系頭進(jìn)行分類，tail_rel_shaking_outputs_list對(duì)關(guān)系尾進(jìn)行分類，其中關(guān)鍵函數(shù)shaking_hiddens4ent中的HandshakingKernel函數(shù)定義如下
class HandshakingKernel(nn.Module):def __init__(self, hidden_size, shaking_type, inner_enc_type):super().__init__()self.shaking_type = shaking_typeif shaking_type == "cat":self.combine_fc = nn.Linear(hidden_size * 2, hidden_size) #fc層elif shaking_type == "cat_plus":self.combine_fc = nn.Linear(hidden_size * 3, hidden_size)elif shaking_type == "cln":self.tp_cln = LayerNorm(hidden_size, hidden_size, conditional = True)elif shaking_type == "cln_plus":self.tp_cln = LayerNorm(hidden_size, hidden_size, conditional = True)self.inner_context_cln = LayerNorm(hidden_size, hidden_size, conditional = True)self.inner_enc_type = inner_enc_type #一層單向lstmif inner_enc_type == "mix_pooling":self.lamtha = Parameter(torch.rand(hidden_size))elif inner_enc_type == "lstm":self.inner_context_lstm = nn.LSTM(hidden_size, hidden_size, num_layers = 1, bidirectional = False, batch_first = True)def enc_inner_hiddens(self, seq_hiddens, inner_enc_type = "lstm"):# seq_hiddens: (batch_size, seq_len, hidden_size)def pool(seqence, pooling_type):if pooling_type == "mean_pooling":pooling = torch.mean(seqence, dim = -2)elif pooling_type == "max_pooling":pooling, _ = torch.max(seqence, dim = -2)elif pooling_type == "mix_pooling":pooling = self.lamtha * torch.mean(seqence, dim = -2) + (1 - self.lamtha) * torch.max(seqence, dim = -2)[0]return poolingif "pooling" in inner_enc_type:inner_context = torch.stack([pool(seq_hiddens[:, :i+1, :], inner_enc_type) for i in range(seq_hiddens.size()[1])], dim = 1)elif inner_enc_type == "lstm":inner_context, _ = self.inner_context_lstm(seq_hiddens)return inner_contextdef forward(self, seq_hiddens):'''seq_hiddens: (batch_size, seq_len, hidden_size)return:shaking_hiddenss: (batch_size, (1 + seq_len) * seq_len / 2, hidden_size) (32, 5+4+3+2+1, 5)'''#一句話中每個(gè)字與剩下的字構(gòu)成上三角矩陣如：長(zhǎng)度為5則的到的為[[batch,5,hidden_size],[batch,4,hidden_size]...]seq_len = seq_hiddens.size()[-2] #句子的長(zhǎng)度shaking_hiddens_list = []for ind in range(seq_len):hidden_each_step = seq_hiddens[:, ind, :] #取每個(gè)batch的每個(gè)字的維度visible_hiddens = seq_hiddens[:, ind:, :] # 從當(dāng)前取到最后repeat_hiddens = hidden_each_step[:, None, :].repeat(1, seq_len - ind, 1) #復(fù)制dim=1的維度跟visible維度保持一致if self.shaking_type == "cat":#選擇的是cat模式，可以在配置文件中設(shè)置shaking_hiddens = torch.cat([repeat_hiddens, visible_hiddens], dim = -1) #將當(dāng)前每個(gè)字的維度與其后的每個(gè)字的維度拼接在一起shaking_hiddens = torch.tanh(self.combine_fc(shaking_hiddens))#過一個(gè)線性層elif self.shaking_type == "cat_plus":inner_context = self.enc_inner_hiddens(visible_hiddens, self.inner_enc_type)shaking_hiddens = torch.cat([repeat_hiddens, visible_hiddens, inner_context], dim = -1)shaking_hiddens = torch.tanh(self.combine_fc(shaking_hiddens))elif self.shaking_type == "cln":shaking_hiddens = self.tp_cln(visible_hiddens, repeat_hiddens)elif self.shaking_type == "cln_plus":inner_context = self.enc_inner_hiddens(visible_hiddens, self.inner_enc_type)shaking_hiddens = self.tp_cln(visible_hiddens, repeat_hiddens)shaking_hiddens = self.inner_context_cln(shaking_hiddens, inner_context)shaking_hiddens_list.append(shaking_hiddens) #添加到列表中long_shaking_hiddens = torch.cat(shaking_hiddens_list, dim = 1)#鋪平上三角矩陣return long_shaking_hiddens輸入的seq_hiddens維度是[batch,seq_len, hiddensize]，是一句話經(jīng)過bert編碼過后的值，而HandshakingKernel函數(shù)的作用是將矩陣變?yōu)樯先蔷仃?#xff0c;即本身矩陣為[seq_len * seq_len]，在經(jīng)過函數(shù)過后為每一行都減1，最后通過long_shakinghiddens把函數(shù)把結(jié)果鋪平，得到[seq_len+(seq_len -1) + (seq_len -2)…+1],對(duì)應(yīng)了圖片部分。
整個(gè)函數(shù)先是循環(huán)每句話中的詞，當(dāng)ind是0時(shí)，hidden_each_step代表了循環(huán)的每個(gè)詞的編碼[batch,1,hidden_size]，visiblehiddens是循環(huán)到的這個(gè)單詞以及之后的單詞的編碼，維度就是[batch,seq_len,hidden_size],repeat_hiddens對(duì)hidden_each_step的第二個(gè)維度進(jìn)行了復(fù)制，維度為[batch,seq_len,hidden_size]，將當(dāng)前單詞和其后的各個(gè)單詞的編碼進(jìn)行拼接維度是[batch,seq_len,hidden_size*2]組成上三角矩陣的一行,在經(jīng)過MLP層后shakinghiddens的維度是[batch,seq_len,hidden_size]，之后每一行依次類推。
關(guān)于loss部分
total_loss, total_ent_sample_acc, total_head_rel_sample_acc, total_tail_rel_sample_acc = 0., 0., 0., 0.for batch_ind, batch_train_data in enumerate(dataloader):t_batch = time.time()z = (2 * len(rel2id) + 1) # 2倍的關(guān)系steps_per_ep = len(dataloader) #有多少數(shù)據(jù)total_steps = hyper_parameters["loss_weight_recover_steps"] + 1 # + 1 avoid division by zero error #加速loss在一定的步數(shù)回歸current_step = steps_per_ep * ep + batch_ind # ？w_ent = max(1 / z + 1 - current_step / total_steps, 1 / z)w_rel = min((len(rel2id) / z) * current_step / total_steps, (len(rel2id) / z))loss_weights = {"ent": w_ent, "rel": w_rel} #給予不同任務(wù)的權(quán)重loss, ent_sample_acc, head_rel_sample_acc, tail_rel_sample_acc = train_step(batch_train_data, optimizer, loss_weights)scheduler.step()total_loss += losstotal_ent_sample_acc += ent_sample_acctotal_head_rel_sample_acc += head_rel_sample_acctotal_tail_rel_sample_acc += tail_rel_sample_accavg_loss = total_loss / (batch_ind + 1)avg_ent_sample_acc = total_ent_sample_acc / (batch_ind + 1)avg_head_rel_sample_acc = total_head_rel_sample_acc / (batch_ind + 1)avg_tail_rel_sample_acc = total_tail_rel_sample_acc / (batch_ind + 1)隨著step加大，w_ent的權(quán)重遞減，w_rel權(quán)重遞增。先關(guān)注實(shí)體，保證實(shí)體抽準(zhǔn)確，后面關(guān)注關(guān)系的抽取，由于目前工作原因，更多細(xì)節(jié)待閑時(shí)在進(jìn)行解讀。
百度信息抽取Lic2020關(guān)系抽取：https://zhuanlan.zhihu.com/p/138858558
總結(jié)
以上是生活随笔為你收集整理的TPLinker 联合抽取 实体链接方式+源码分析的全部?jī)?nèi)容，希望文章能夠幫你解決所遇到的問題。
- 上一篇： 技术论坛 | CCF YOCSEF上海即
- 下一篇： 一文搞懂HMM（隐马尔可夫模型）-Vit