系列文章目錄

命名實體識別NER探索（1） https://duanzhihua.blog.csdn.net/article/details/108338970 命名實體識別NER探索（2） https://duanzhihua.blog.csdn.net/article/details/108391645 命名實體識別NER探索(3)-Bi-LSTM+CRF模型 https://duanzhihua.blog.csdn.net/article/details/108392532 命名實體識別NER探索(4) 通過scikit-learn、pytorch實現HMM 及CRF模型 https://duanzhihua.blog.csdn.net/article/details/108650903 Viterbi算法實戰案例（天氣變化、詞性預測） https://duanzhihua.blog.csdn.net/article/details/104992597

文章目錄

• 系列文章目錄
• 本文內容
• 數據集
• Bert+BiLSTM+CRF模型代碼
• BERT模型運行日志
• Bert+BiLSTM+CRF模型運行結果
• 總結

---

本文內容

通過Bert+Bi-LSTM+CRF模型探索中文關鍵信息實體識別。

• 使用BERT預訓練模型，獲取每一個標識的向量表示特征
• 輸入BiLSTM模型學習文本之間的關系
• 通過CRF層獲取每個標識的分類結果
  BERT+BiLSTM+CRF模型圖
  

數據集

數據集用的是客服熱線的內部話單數據，將客服人員接聽的語音數據自動翻譯為文本數據，然后從文本數據中提取具體的地址信息。數據記錄格式如下：

,工號8888，為您服務,，唉，您好，我想咨詢一下，就是這種呃建筑工地深跟半夜還在施工，噪音這種，呃有什么規定和要求嗎？能,呃有什么方式能讓他反映給 他們,處理嘛，這種,對對,，對的，對的對的,，地址是在普陀區東興路這里,東，愛心叫新舊的新,，唉，東興路呃，88弄,，唉,，呃，新湖明珠,，新舊的新,河， 湖水的湖,，唉，明白的明唉，明珠的朱,，對對對,對，他現在是有一半，我們是住在對面嘛，他現在那邊還正在,，呃就是還在施工，就是現在這會還在施工,唉, ，早上有早上大概8888點鐘就有了,，呃，中堅,，唉，晚上晚上現在到88點多還沒停對的,，嗯嗯，那聲音比較吵那個,，嗯嗯,，能夠對對對對對因為或者是至少 有個,您好，請問什么幫您,？呃您好女士這邊您主要是反映，嗯就是說是建筑工地施工噪音擾民對吧？嗯，那么我想問一下他這個施工的時間段具體呃就地址是哪 里,，普陀區東興路是東西南北的東，呃，新是哪個新啊,？新舊的新噢，東興路哪里呢,？是88弄的啊，呃，叫什么名字呢他,新湖名松筠就地深,，湖水的湖呃,， 明珠,到上面做民族,，呃這個是小區是吧，它新建的這個工地的,，現在那么他這個是，嗯嗯是施工時間大概是從早上有來，早上有時候嗎？還是,早上這邊嗯,， 五六點鐘，嗯，然后呢一直到設備嗯,，到現在還沒有聽到吧,，噢，好的好的,噢，那知道了，我幫您反映一下，那就是來電，你的訴求是希望管理部門,能夠吃制

標注集在標準BIOES（B表示實體開頭，E表示實體結尾，I表示在實體內部，O表示非實體）基礎上，采用內部的地址實體標注。例如’Q-B’, ‘Q-I’, ‘Q-E’,分別表示區的開頭，區的中間詞，區的結尾

浦 Q-B 東 Q-I 新 Q-I 區 Q-E ..... , M ， M 就 M 是 M 說 M 呃 M 徐 Q-B 匯 Q-I 區 Q-E 康 Z-B 健 Z-I 街 Z-I 道 Z-E 桂 L-B 平 L-I 路 L-E , M

Bert+BiLSTM+CRF模型代碼

基于github網上大佬的Bert+BiLSTM+CRF基線模型改進，關鍵代碼如下：

import tensorflow as tf from tf_utils.bert_modeling import BertModel, BertConfig, get_assignment_map_from_checkpoint from tensorflow.contrib.crf import crf_log_likelihood from tensorflow.contrib.layers.python.layers import initializers from tf_utils import rnncell as rnnclass Model:def __init__(self, config):self.config = config# 模型的數據占位符self.input_x_word = tf.placeholder(tf.int32, [None, None], name="input_x_word")self.input_x_len = tf.placeholder(tf.int32, name='input_x_len')self.input_mask = tf.placeholder(tf.int32, [None, None], name='input_mask')self.input_relation = tf.placeholder(tf.int32, [None, None], name='input_relation') # 實體NER的真實標簽self.keep_prob = tf.placeholder(tf.float32, name='dropout_keep_prob')self.is_training = tf.placeholder(tf.bool, None, name='is_training')# BERT Embeddingself.init_embedding(bert_init=True)output_layer = self.word_embedding# 超參數設置self.relation_num = self.config.relation_numself.initializer = initializers.xavier_initializer()self.lstm_dim = self.config.lstm_dimself.embed_dense_dim = self.config.embed_dense_dimself.dropout = self.config.dropoutself.model_type = self.config.model_typeprint('Run Model Type:', self.model_type)# idcnn的超參數self.layers = [{'dilation': 1},{'dilation': 1},{'dilation': 2},]self.filter_width = 3self.num_filter = self.lstm_dimself.embedding_dim = self.embed_dense_dimself.repeat_times = 4self.cnn_output_width = 0# CRF超參數used = tf.sign(tf.abs(self.input_x_word))length = tf.reduce_sum(used, reduction_indices=1)self.lengths = tf.cast(length, tf.int32)self.batch_size = tf.shape(self.input_x_word)[0]self.num_steps = tf.shape(self.input_x_word)[-1]if self.model_type == 'bilstm':lstm_inputs = tf.nn.dropout(output_layer, self.dropout)lstm_outputs = self.biLSTM_layer(lstm_inputs, self.lstm_dim, self.lengths)self.logits = self.project_layer(lstm_outputs)elif self.model_type == 'idcnn':model_inputs = tf.nn.dropout(output_layer, self.dropout)model_outputs = self.IDCNN_layer(model_inputs)self.logits = self.project_layer_idcnn(model_outputs)else:raise KeyError# 計算損失self.loss = self.loss_layer(self.logits, self.lengths)def biLSTM_layer(self, lstm_inputs, lstm_dim, lengths, name=None):""":param lstm_inputs: [batch_size, num_steps, emb_size]:return: [batch_size, num_steps, 2*lstm_dim]"""with tf.name_scope("char_BiLSTM" if not name else name):lstm_cell = {}for direction in ["forward", "backward"]:with tf.name_scope(direction):lstm_cell[direction] = rnn.CoupledInputForgetGateLSTMCell(lstm_dim,use_peepholes=True,initializer=self.initializer,state_is_tuple=True)outputs, final_states = tf.nn.bidirectional_dynamic_rnn(lstm_cell["forward"],lstm_cell["backward"],lstm_inputs,dtype=tf.float32,sequence_length=lengths)return tf.concat(outputs, axis=2)def project_layer(self, lstm_outputs, name=None):"""hidden layer between lstm layer and logits:param lstm_outputs: [batch_size, num_steps, emb_size]:return: [batch_size, num_steps, num_tags]"""with tf.name_scope("project" if not name else name):with tf.name_scope("hidden"):W = tf.get_variable("HW", shape=[self.lstm_dim * 2, self.lstm_dim],dtype=tf.float32, initializer=self.initializer)b = tf.get_variable("Hb", shape=[self.lstm_dim], dtype=tf.float32,initializer=tf.zeros_initializer())output = tf.reshape(lstm_outputs, shape=[-1, self.lstm_dim * 2])hidden = tf.tanh(tf.nn.xw_plus_b(output, W, b))# project to score of tagswith tf.name_scope("logits"):W = tf.get_variable("LW", shape=[self.lstm_dim, self.relation_num],dtype=tf.float32, initializer=self.initializer)b = tf.get_variable("Lb", shape=[self.relation_num], dtype=tf.float32,initializer=tf.zeros_initializer())pred = tf.nn.xw_plus_b(hidden, W, b)return tf.reshape(pred, [-1, self.num_steps, self.relation_num], name='pred_logits')def IDCNN_layer(self, model_inputs, name=None):""":param idcnn_inputs: [batch_size, num_steps, emb_size]:return: [batch_size, num_steps, cnn_output_width]"""model_inputs = tf.expand_dims(model_inputs, 1)with tf.variable_scope("idcnn" if not name else name):shape = [1, self.filter_width, self.embedding_dim,self.num_filter]print(shape)filter_weights = tf.get_variable("idcnn_filter",shape=[1, self.filter_width, self.embedding_dim, self.num_filter],initializer=self.initializer)layerInput = tf.nn.conv2d(model_inputs,filter_weights,strides=[1, 1, 1, 1],padding="SAME",name="init_layer")finalOutFromLayers = []totalWidthForLastDim = 0for j in range(self.repeat_times):for i in range(len(self.layers)):dilation = self.layers[i]['dilation']isLast = True if i == (len(self.layers) - 1) else Falsewith tf.variable_scope("atrous-conv-layer-%d" % i,reuse=tf.AUTO_REUSE):w = tf.get_variable("filterW",shape=[1, self.filter_width, self.num_filter,self.num_filter],initializer=tf.contrib.layers.xavier_initializer())b = tf.get_variable("filterB", shape=[self.num_filter])conv = tf.nn.atrous_conv2d(layerInput,w,rate=dilation,padding="SAME")conv = tf.nn.bias_add(conv, b)conv = tf.nn.relu(conv)if isLast:finalOutFromLayers.append(conv)totalWidthForLastDim += self.num_filterlayerInput = convfinalOut = tf.concat(axis=3, values=finalOutFromLayers)keepProb = tf.cond(self.is_training, lambda: 0.8, lambda: 1.0)# keepProb = 1.0 if reuse else 0.5finalOut = tf.nn.dropout(finalOut, keepProb)finalOut = tf.squeeze(finalOut, [1])finalOut = tf.reshape(finalOut, [-1, totalWidthForLastDim])self.cnn_output_width = totalWidthForLastDimreturn finalOutdef project_layer_idcnn(self, idcnn_outputs, name=None):""":param lstm_outputs: [batch_size, num_steps, emb_size]:return: [batch_size, num_steps, num_tags]"""with tf.name_scope("project" if not name else name):# project to score of tagswith tf.name_scope("logits"):W = tf.get_variable("PLW", shape=[self.cnn_output_width, self.relation_num],dtype=tf.float32, initializer=self.initializer)b = tf.get_variable("PLb", initializer=tf.constant(0.001, shape=[self.relation_num]))pred = tf.nn.xw_plus_b(idcnn_outputs, W, b)return tf.reshape(pred, [-1, self.num_steps, self.relation_num], name='pred_logits')def loss_layer(self, project_logits, lengths, name=None):"""計算CRF的loss:param project_logits: [1, num_steps, num_tags]:return: scalar loss"""with tf.name_scope("crf_loss" if not name else name):small = -1000.0# pad logits for crf lossstart_logits = tf.concat([small * tf.ones(shape=[self.batch_size, 1, self.relation_num]), tf.zeros(shape=[self.batch_size, 1, 1])],axis=-1)pad_logits = tf.cast(small * tf.ones([self.batch_size, self.num_steps, 1]), tf.float32)logits = tf.concat([project_logits, pad_logits], axis=-1)logits = tf.concat([start_logits, logits], axis=1)targets = tf.concat([tf.cast(self.relation_num * tf.ones([self.batch_size, 1]), tf.int32), self.input_relation], axis=-1)self.trans = tf.get_variable(name="transitions",shape=[self.relation_num + 1, self.relation_num + 1], # 1# shape=[self.relation_num, self.relation_num], # 1initializer=self.initializer)log_likelihood, self.trans = crf_log_likelihood(inputs=logits,tag_indices=targets,# tag_indices=self.input_relation,transition_params=self.trans,# sequence_lengths=lengthssequence_lengths=lengths + 1) # + 1return tf.reduce_mean(-log_likelihood, name='loss')def init_embedding(self, bert_init=True):"""對BERT的Embedding降維:param bert_init::return:"""with tf.name_scope('embedding'):word_embedding = self.bert_embed(bert_init)print('self.embed_dense_dim:', self.config.embed_dense_dim)word_embedding = tf.layers.dense(word_embedding, self.config.embed_dense_dim, activation=tf.nn.relu)hidden_size = word_embedding.shape[-1].valueself.word_embedding = word_embeddingprint(word_embedding.shape)self.output_layer_hidden_size = hidden_sizedef bert_embed(self, bert_init=True):"""讀取BERT的TF模型:param bert_init::return:"""bert_config_file = self.config.bert_config_filebert_config = BertConfig.from_json_file(bert_config_file)# batch_size, max_seq_length = get_shape_list(self.input_x_word)# bert_mask = tf.pad(self.input_mask, [[0, 0], [2, 0]], constant_values=1) # tensor左邊填充2列model = BertModel(config=bert_config,is_training=self.is_training, # 微調input_ids=self.input_x_word,input_mask=self.input_mask,token_type_ids=None,use_one_hot_embeddings=False)layer_logits = []for i, layer in enumerate(model.all_encoder_layers):layer_logits.append(tf.layers.dense(layer, 1,kernel_initializer=tf.truncated_normal_initializer(stddev=0.02),name="layer_logit%d" % i))layer_logits = tf.concat(layer_logits, axis=2) # 第三維度拼接layer_dist = tf.nn.softmax(layer_logits)seq_out = tf.concat([tf.expand_dims(x, axis=2) for x in model.all_encoder_layers], axis=2)pooled_output = tf.matmul(tf.expand_dims(layer_dist, axis=2), seq_out)pooled_output = tf.squeeze(pooled_output, axis=2)pooled_layer = pooled_output# char_bert_outputs = pooled_laRERyer[:, 1: max_seq_length - 1, :] # [batch_size, seq_length, embedding_size]char_bert_outputs = pooled_layerif self.config.use_origin_bert:final_hidden_states = model.get_sequence_output() # 原生bertself.config.embed_dense_dim = 768else:final_hidden_states = char_bert_outputs # 多層融合bertself.config.embed_dense_dim = 512tvars = tf.trainable_variables()init_checkpoint = self.config.bert_file # './chinese_L-12_H-768_A-12/bert_model.ckpt'assignment_map, initialized_variable_names = get_assignment_map_from_checkpoint(tvars, init_checkpoint)if bert_init:tf.train.init_from_checkpoint(init_checkpoint, assignment_map)tf.logging.info("**** Trainable Variables ****")for var in tvars:init_string = ""if var.name in initialized_variable_names:init_string = ", *INIT_FROM_CKPT*"print(" name = {}, shape = {}{}".format(var.name, var.shape, init_string))print('init bert from checkpoint: {}'.format(init_checkpoint))return final_hidden_states

BERT模型運行日志

nohup: ignoring input WARNING:tensorflow:From /data/Test/12345-BERT-NER/optimization.py:155: The name tf.train.AdamOptimizer is deprecated. Pleas e use tf.compat.v1.train.AdamOptimizer instead.WARNING:tensorflow:From /data/Test/12345-BERT-NER/bert/tokenization.py:125: The name tf.gfile.GFile is deprecated. Please u se tf.io.gfile.GFile instead.WARNING:tensorflow:From train_fine_tune.py:41: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto i nstead.WARNING:tensorflow:From train_fine_tune.py:43: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.2020-10-21 20:56:51.457173: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this T ensorFlow binary was not compiled to use: AVX2 AVX512F FMA 2020-10-21 20:56:51.476659: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2300000000 Hz 2020-10-21 20:56:51.482845: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x4faeb90 initialized for platfor m Host (this does not guarantee that XLA will be used). Devices: 2020-10-21 20:56:51.482889: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Ve rsion 2020-10-21 20:56:51.486817: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic lib rary libcuda.so.1 2020-10-21 20:56:51.629666: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x4faad50 initialized for platfor m CUDA (this does not guarantee that XLA will be used). Devices: 2020-10-21 20:56:51.629747: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Tesla P40, Compu te Capability 6.1 2020-10-21 20:56:51.632698: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 0 with properties: name: Tesla P40 major: 6 minor: 1 memoryClockRate(GHz): 1.531 pciBusID: 0000:b6:00.0 2020-10-21 20:56:51.633321: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic lib rary libcudart.so.10.0 2020-10-21 20:56:51.637701: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic lib rary libcublas.so.10.0 2020-10-21 20:56:51.641439: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic lib rary libcufft.so.10.0 2020-10-21 20:56:51.641968: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic lib rary libcurand.so.10.0 2020-10-21 20:56:51.645511: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic lib rary libcusolver.so.10.0 2020-10-21 20:56:51.648178: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic lib rary libcusparse.so.10.0 2020-10-21 20:56:51.654540: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic lib rary libcudnn.so.7 2020-10-21 20:56:51.657031: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1767] Adding visible gpu devices: 0 2020-10-21 20:56:51.657081: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic lib rary libcudart.so.10.0 2020-10-21 20:56:51.658901: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1180] Device interconnect StreamExecutor wit h strength 1 edge matrix: 2020-10-21 20:56:51.658922: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1186] 0 2020-10-21 20:56:51.658936: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1199] 0: N 2020-10-21 20:56:51.661417: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1325] Created TensorFlow device (/job:localh ost/replica:0/task:0/device:GPU:0 with 21625 MB memory) -> physical GPU (device: 0, name: Tesla P40, pci bus id: 0000:b6:00 .0, compute capability: 6.1) WARNING:tensorflow:From /data/Test/12345-BERT-NER/model.py:13: The name tf.placeholder is deprecated. Please use tf.compat. v1.placeholder instead.WARNING:tensorflow:From /data/Test/12345-BERT-NER/tf_utils/bert_modeling.py:175: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead.WARNING:tensorflow:From /data/Test/12345-BERT-NER/tf_utils/bert_modeling.py:416: The name tf.get_variable is deprecated. Pl ease use tf.compat.v1.get_variable instead.WARNING:tensorflow:From /data/Test/12345-BERT-NER/tf_utils/bert_modeling.py:497: The name tf.assert_less_equal is deprecate d. Please use tf.compat.v1.assert_less_equal instead.WARNING:tensorflow: The TensorFlow contrib module will not be included in TensorFlow 2.0. For more information, please see:* https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md* https://github.com/tensorflow/addons* https://github.com/tensorflow/io (for I/O related ops) If you depend on functionality not listed there, please file an issue.WARNING:tensorflow:From /data/Test/12345-BERT-NER/tf_utils/bert_modeling.py:364: calling dropout (from tensorflow.python.op s.nn_ops) with keep_prob is deprecated and will be removed in a future version. Instructions for updating: Please use `rate` instead of `keep_prob`. Rate should be set to `rate = 1 - keep_prob`. WARNING:tensorflow:From /data/Test/12345-BERT-NER/tf_utils/bert_modeling.py:874: dense (from tensorflow.python.layers.core)is deprecated and will be removed in a future version. Instructions for updating: Use keras.layers.Dense instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/layers/core.py:187: Layer.apply (fromtensorflow.python.keras.engine.base_layer) is deprecated and will be removed in a future version. Instructions for updating: Please use `layer.__call__` method instead. WARNING:tensorflow:From /data/Test/12345-BERT-NER/tf_utils/bert_modeling.py:282: The name tf.erf is deprecated. Please use tf.math.erf instead.WARNING:tensorflow:From /data/Test/12345-BERT-NER/model.py:288: The name tf.trainable_variables is deprecated. Please use t f.compat.v1.trainable_variables instead.WARNING:tensorflow:From /data/Test/12345-BERT-NER/model.py:292: The name tf.train.init_from_checkpoint is deprecated. Pleas e use tf.compat.v1.train.init_from_checkpoint instead.WARNING:tensorflow:From /data/Test/12345-BERT-NER/model.py:294: The name tf.logging.info is deprecated. Please use tf.compa t.v1.logging.info instead.#### /data/Test/12345-BERT-NER GPU ID: 0 Model Type: bilstm Fine Tune Learning Rate: 5e-05 Data dir: ./data/12345_entity_recog/clear_csv_data/ Pretrained Model Vocab: ./data/pretrained_model/BERT/vocab.txt bilstm embedding 256 ... 798 Get the train iter data and dev iter data........! ... 198name = bert/embeddings/word_embeddings:0, shape = (21128, 768), *INIT_FROM_CKPT*name = bert/embeddings/token_type_embeddings:0, shape = (2, 768), *INIT_FROM_CKPT*name = bert/embeddings/position_embeddings:0, shape = (512, 768), *INIT_FROM_CKPT*name = bert/embeddings/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*name = bert/embeddings/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*name = bert/encoder/layer_0/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*name = bert/encoder/layer_0/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*name = bert/encoder/layer_0/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*name = bert/encoder/layer_0/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*name = bert/encoder/layer_0/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*name = bert/encoder/layer_0/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*name = bert/encoder/layer_0/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*name = bert/encoder/layer_0/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*name = bert/encoder/layer_0/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*name = bert/encoder/layer_0/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*name = bert/encoder/layer_0/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*name = bert/encoder/layer_0/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*name = bert/encoder/layer_0/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*name = bert/encoder/layer_0/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*name = bert/encoder/layer_0/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*name = bert/encoder/layer_0/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*name = bert/encoder/layer_1/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*name = bert/encoder/layer_1/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*name = bert/encoder/layer_1/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*name = bert/encoder/layer_1/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*name = bert/encoder/layer_1/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*name = bert/encoder/layer_1/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*name = bert/encoder/layer_1/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*name = bert/encoder/layer_1/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*name = bert/encoder/layer_1/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*name = bert/encoder/layer_1/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*name = bert/encoder/layer_1/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*name = bert/encoder/layer_1/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*name = bert/encoder/layer_1/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*name = bert/encoder/layer_1/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*name = bert/encoder/layer_1/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*name = bert/encoder/layer_1/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*name = bert/encoder/layer_2/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*name = bert/encoder/layer_2/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*name = bert/encoder/layer_2/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*name = bert/encoder/layer_2/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*name = bert/encoder/layer_2/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*name = bert/encoder/layer_2/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*name = bert/encoder/layer_2/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*name = bert/encoder/layer_2/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*name = bert/encoder/layer_2/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*name = bert/encoder/layer_2/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*name = bert/encoder/layer_2/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*name = bert/encoder/layer_2/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*name = bert/encoder/layer_2/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*name = bert/encoder/layer_2/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*name = bert/encoder/layer_2/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*name = bert/encoder/layer_2/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*name = bert/encoder/layer_3/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*name = bert/encoder/layer_3/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*name = bert/encoder/layer_3/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*name = bert/encoder/layer_3/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*name = bert/encoder/layer_3/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*name = bert/encoder/layer_3/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*name = bert/encoder/layer_3/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*name = bert/encoder/layer_3/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*name = bert/encoder/layer_3/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*name = bert/encoder/layer_3/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*name = bert/encoder/layer_3/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*name = bert/encoder/layer_3/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*name = bert/encoder/layer_3/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*name = bert/encoder/layer_3/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*name = bert/encoder/layer_3/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*name = bert/encoder/layer_3/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*name = bert/encoder/layer_4/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*name = bert/encoder/layer_4/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*name = bert/encoder/layer_4/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*name = bert/encoder/layer_4/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*name = bert/encoder/layer_4/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*name = bert/encoder/layer_4/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*name = bert/encoder/layer_4/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*name = bert/encoder/layer_4/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*name = bert/encoder/layer_4/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*name = bert/encoder/layer_4/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*name = bert/encoder/layer_4/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*name = bert/encoder/layer_4/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*name = bert/encoder/layer_4/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*name = bert/encoder/layer_4/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*name = bert/encoder/layer_4/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*name = bert/encoder/layer_4/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*name = bert/encoder/layer_5/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*name = bert/encoder/layer_5/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*name = bert/encoder/layer_5/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*name = bert/encoder/layer_5/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*name = bert/encoder/layer_5/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*name = bert/encoder/layer_5/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*name = bert/encoder/layer_5/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*name = bert/encoder/layer_5/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*name = bert/encoder/layer_5/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*name = bert/encoder/layer_5/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*name = bert/encoder/layer_5/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*name = bert/encoder/layer_5/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*name = bert/encoder/layer_5/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*name = bert/encoder/layer_5/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*name = bert/encoder/layer_5/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*name = bert/encoder/layer_5/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*name = bert/encoder/layer_6/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*name = bert/encoder/layer_6/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*name = bert/encoder/layer_6/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*name = bert/encoder/layer_6/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*name = bert/encoder/layer_6/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*name = bert/encoder/layer_6/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*name = bert/encoder/layer_6/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*name = bert/encoder/layer_6/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*name = bert/encoder/layer_6/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*name = bert/encoder/layer_6/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*name = bert/encoder/layer_6/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*name = bert/encoder/layer_6/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*name = bert/encoder/layer_6/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*name = bert/encoder/layer_6/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*name = bert/encoder/layer_6/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*name = bert/encoder/layer_6/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*name = bert/encoder/layer_7/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*name = bert/encoder/layer_7/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*name = bert/encoder/layer_7/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*name = bert/encoder/layer_7/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*name = bert/encoder/layer_7/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*name = bert/encoder/layer_7/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*name = bert/encoder/layer_7/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*name = bert/encoder/layer_7/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*name = bert/encoder/layer_7/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*name = bert/encoder/layer_7/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*name = bert/encoder/layer_7/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*name = bert/encoder/layer_7/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*name = bert/encoder/layer_7/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*name = bert/encoder/layer_7/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*name = bert/encoder/layer_7/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*name = bert/encoder/layer_7/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*name = bert/encoder/layer_8/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*name = bert/encoder/layer_8/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*name = bert/encoder/layer_8/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*name = bert/encoder/layer_8/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*name = bert/encoder/layer_8/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*name = bert/encoder/layer_8/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*name = bert/encoder/layer_8/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*name = bert/encoder/layer_8/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*name = bert/encoder/layer_8/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*name = bert/encoder/layer_8/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*name = bert/encoder/layer_8/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*name = bert/encoder/layer_8/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*name = bert/encoder/layer_8/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*name = bert/encoder/layer_8/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*name = bert/encoder/layer_8/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*name = bert/encoder/layer_8/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*name = bert/encoder/layer_9/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*name = bert/encoder/layer_9/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*name = bert/encoder/layer_9/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*name = bert/encoder/layer_9/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*name = bert/encoder/layer_9/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*name = bert/encoder/layer_9/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*name = bert/encoder/layer_9/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*name = bert/encoder/layer_9/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*name = bert/encoder/layer_9/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*name = bert/encoder/layer_9/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*name = bert/encoder/layer_9/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*name = bert/encoder/layer_9/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*name = bert/encoder/layer_9/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*name = bert/encoder/layer_9/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*name = bert/encoder/layer_9/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*name = bert/encoder/layer_9/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*name = bert/encoder/layer_10/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*name = bert/encoder/layer_10/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*name = bert/encoder/layer_10/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*name = bert/encoder/layer_10/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*WARNING:tensorflow:From /data/Tes t/12345-BERT-NER/model.py:93: bidirectional_dynamic_rnn (from tensorflow.python.ops.rnn) is deprecated and will be removed in a future version. Instructions for updating: Please use `keras.layers.Bidirectional(keras.layers.RNN(cell))`, which is equivalent to this API WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/rnn.py:464: dynamic_rnn (from ten sorflow.python.ops.rnn) is deprecated and will be removed in a future version. ......

Bert+BiLSTM+CRF模型運行結果

文本信息： [CLS],工號8079，為您服務,，呃，現在想跟你們反映一下，就是淮海中路41568號,，呃，他們的房子啊，有人在改造,，呃，，，我看也溝通了,，剛才呢跟76589打電話說要跟我們79299聯系，7489，那個電話打不通，那么跟你們反映一下,，請有關部門來來看一下，呃，這樣做是不是和服務有關的規定,沒有,A,，呃黃埔區的,，呃，淮海中路啊,，還在海上呢，還還是,，呃項懷忠的淮啊，三點水,，現在有人在條889847號的強辯在改造房屋在條房屋墻,這個,，唉，這個誰來管一下,，看一下,A,房管部門啊,，258236,，呃骸,，唉，55745打不通,，對,，唉，再跟你們反映一下,，呃,，唉,，好的,，呃，最好給我一個回復吧,，85164036,，沒有家里電話,唉,，唉，我叫吳總件,,您好，請問什么可以幫您,，請問一下，這個事情之前有向我們15543電話反映過嗎,？你剛才跟我說是淮海中路039弄8號對嗎,？請問什么區,，淮海中路的寫法是淮海戰役的淮海，忠心的忠，對嗎,？你要投訴它什么呢,？噢，就投訴他教會群眾想了,，你剛才說你剛才說向哪里啊？894947反映啊,，87217是什么,房？管是59531,，呃，154297，然[SEP] 打標文本的地址信息： [{'word': '淮海中路', 'start': 30, 'end': 34, 'type': 'L-E'}, {'word': '黃埔區', 'start': 154, 'end': 157, 'type': 'Q-E'},{'word': '039弄8號', 'start': 391, 'end': 397, 'type': 'H-E'}] 預測的地址信息： [{'word': '淮海中路', 'start': 30, 'end': 34, 'type': 'L-E'}, {'word': '41568號', 'start': 34, 'end': 40, 'type': 'H-E'}, {'word': '黃埔區', 'start': 154, 'end': 157, 'type': 'Q-E'}, {'word': '039弄8號', 'start': 391, 'end': 397, 'type': 'H-E'}]

總結

本文簡單介紹了Bert+BiLSTM+CRF模型的概念，及Bert+BiLSTM+CRF模型的案例應用。

總結

以上是生活随笔為你收集整理的命名实体识别NER探索(5) Bert+BiLSTM+CRF模型实战应用的全部內容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網站內容還不錯，歡迎將生活随笔推薦給好友。