寫在前面

已經發布：

【NLP傻瓜式教程】手把手帶你CNN文本分類(附代碼)

【NLP傻瓜式教程】手把手帶你RNN文本分類(附代碼)

繼續NLP傻瓜式教程系列，今天的教程是基于FAIR的Bag of Tricks for Efficient Text Classification^[1]。也就是我們常說的fastText。

最讓人欣喜的這篇論文配套提供了fasttext工具包。這個工具包代碼質量非常高，論文結果一鍵還原，目前已經是包裝地非常專業了，這是fastText官網和其github代碼庫，以及提供了python接口，可以直接通過pip安裝。這樣準確率高又快的模型絕對是實戰利器。

為了更好地理解fasttext原理，我們現在直接復現來一遍，但是代碼中僅僅實現了最簡單的基于單詞的詞向量求平均，并未使用b-gram的詞向量，所以自己實現的文本分類效果會低于facebook開源的庫。

論文概覽

?

We can train fastText on more than one billion words in less than ten minutes using a standard multicore CPU, and classify half a million sentences among 312K classes in less than a minute.

首先引用論文中的一段話來看看作者們是怎么評價fasttext模型的表現的。

這篇論文的模型非常之簡單，之前了解過word2vec的同學可以發現這跟CBOW的模型框架非常相似。

對應上面這個模型，比如輸入是一句話，到就是這句話的單詞或者是n-gram。每一個都對應一個向量，然后對這些向量取平均就得到了文本向量，然后用這個平均向量取預測標簽。當類別不多的時候，就是最簡單的softmax；當標簽數量巨大的時候，就要用到「hierarchical softmax」了。

模型真的很簡單，也沒什么可以說的了。下面提一下論文中的兩個tricks：

• 「hierarchical softmax」
  類別數較多時，通過構建一個霍夫曼編碼樹來加速softmax layer的計算，和之前word2vec中的trick相同

• 「N-gram features」
  只用unigram的話會丟掉word order信息，所以通過加入N-gram features進行補充 用hashing來減少N-gram的存儲

看了論文的實驗部分，如此簡單的模型竟然能取得這么好的效果 ！

但是也有人指出論文中選取的數據集都是對句子詞序不是很敏感的數據集，所以得到文中的試驗結果并不奇怪。

代碼實現

看完閹割版代碼大家記得去看看源碼噢~ 跟之前系列的一樣，定義一個fastTextModel類，然后寫網絡框架，輸入輸出placeholder，損失，訓練步驟等。

class fastTextModel(BaseModel):"""A simple implementation of fasttext for text classification"""def __init__(self, sequence_length, num_classes, vocab_size,embedding_size, learning_rate, decay_steps, decay_rate,l2_reg_lambda, is_training=True,initializer=tf.random_normal_initializer(stddev=0.1)):self.vocab_size = vocab_sizeself.embedding_size = embedding_sizeself.num_classes = num_classesself.sequence_length = sequence_lengthself.learning_rate = learning_rateself.decay_steps = decay_stepsself.decay_rate = decay_rateself.is_training = is_trainingself.l2_reg_lambda = l2_reg_lambdaself.initializer = initializerself.input_x = tf.placeholder(tf.int32, [None, self.sequence_length], name='input_x')self.input_y = tf.placeholder(tf.int32, [None, self.num_classes], name='input_y')self.global_step = tf.Variable(0, trainable=False, name='global_step')self.instantiate_weight()self.logits = self.inference()self.loss_val = self.loss()self.train_op = self.train()self.predictions = tf.argmax(self.logits, axis=1, name='predictions')correct_prediction = tf.equal(self.predictions, tf.argmax(self.input_y, 1))self.accuracy = tf.reduce_mean(tf.cast(correct_prediction, 'float'), name='accuracy')def instantiate_weight(self):with tf.name_scope('weights'):self.Embedding = tf.get_variable('Embedding', shape=[self.vocab_size, self.embedding_size],initializer=self.initializer)self.W_projection = tf.get_variable('W_projection', shape=[self.embedding_size, self.num_classes],initializer=self.initializer)self.b_projection = tf.get_variable('b_projection', shape=[self.num_classes])def inference(self):"""1. word embedding2. average embedding3. linear classifier:return:"""# embedding layerwith tf.name_scope('embedding'):words_embedding = tf.nn.embedding_lookup(self.Embedding, self.input_x)self.average_embedding = tf.reduce_mean(words_embedding, axis=1)logits = tf.matmul(self.average_embedding, self.W_projection) +self.b_projectionreturn logitsdef loss(self):# losswith tf.name_scope('loss'):losses = tf.nn.softmax_cross_entropy_with_logits(labels=self.input_y, logits=self.logits)data_loss = tf.reduce_mean(losses)l2_loss = tf.add_n([tf.nn.l2_loss(cand_var) for cand_var in tf.trainable_variables()if 'bias' not in cand_var.name]) * self.l2_reg_lambdadata_loss += l2_loss * self.l2_reg_lambdareturn data_lossdef train(self):with tf.name_scope('train'):learning_rate = tf.train.exponential_decay(self.learning_rate, self.global_step,self.decay_steps, self.decay_rate,staircase=True)train_op = tf.contrib.layers.optimize_loss(self.loss_val, global_step=self.global_step,learning_rate=learning_rate, optimizer='Adam')return train_op def prepocess():"""For load and process data:return:"""print("Loading data...")x_text, y = data_process.load_data_and_labels(FLAGS.positive_data_file, FLAGS.negative_data_file)# bulid vocabularymax_document_length = max(len(x.split(' ')) for x in x_text)vocab_processor = learn.preprocessing.VocabularyProcessor(max_document_length)x = np.array(list(vocab_processor.fit_transform(x_text)))# shufflenp.random.seed(10)shuffle_indices = np.random.permutation(np.arange(len(y)))x_shuffled = x[shuffle_indices]y_shuffled = y[shuffle_indices]# split train/test datasetdev_sample_index = -1 * int(FLAGS.dev_sample_percentage * float(len(y)))x_train, x_dev = x_shuffled[:dev_sample_index], x_shuffled[dev_sample_index:]y_train, y_dev = y_shuffled[:dev_sample_index], y_shuffled[dev_sample_index:]del x, y, x_shuffled, y_shuffledprint('Vocabulary Size: {:d}'.format(len(vocab_processor.vocabulary_)))print('Train/Dev split: {:d}/{:d}'.format(len(y_train), len(y_dev)))return x_train, y_train, vocab_processor, x_dev, y_devdef train(x_train, y_train, vocab_processor, x_dev, y_dev):with tf.Graph().as_default():session_conf = tf.ConfigProto(# allows TensorFlow to fall back on a device with a certain operation implementedallow_soft_placement= FLAGS.allow_soft_placement,# allows TensorFlow log on which devices (CPU or GPU) it places operationslog_device_placement=FLAGS.log_device_placement)sess = tf.Session(config=session_conf)with sess.as_default():# initialize cnnfasttext = fastTextModel(sequence_length=x_train.shape[1],num_classes=y_train.shape[1],vocab_size=len(vocab_processor.vocabulary_),embedding_size=FLAGS.embedding_size,l2_reg_lambda=FLAGS.l2_reg_lambda,is_training=True,learning_rate=FLAGS.learning_rate,decay_steps=FLAGS.decay_steps,decay_rate=FLAGS.decay_rate)# output dir for models and summariestimestamp = str(time.time())out_dir = os.path.abspath(os.path.join(os.path.curdir, 'run', timestamp))if not os.path.exists(out_dir):os.makedirs(out_dir)print('Writing to {} \n'.format(out_dir))# checkpoint dir. checkpointing – saving the parameters of your model to restore them later on.checkpoint_dir = os.path.abspath(os.path.join(out_dir, FLAGS.ckpt_dir))checkpoint_prefix = os.path.join(checkpoint_dir, 'model')if not os.path.exists(checkpoint_dir):os.makedirs(checkpoint_dir)saver = tf.train.Saver(tf.global_variables(), max_to_keep=FLAGS.num_checkpoints)# Write vocabularyvocab_processor.save(os.path.join(out_dir, 'vocab'))# Initialize allsess.run(tf.global_variables_initializer())def train_step(x_batch, y_batch):"""A single training step:param x_batch::param y_batch::return:"""feed_dict = {fasttext.input_x: x_batch,fasttext.input_y: y_batch,}_, step, loss, accuracy = sess.run([fasttext.train_op, fasttext.global_step, fasttext.loss_val, fasttext.accuracy],feed_dict=feed_dict)time_str = datetime.datetime.now().isoformat()print("{}: step {}, loss {:g}, acc {:g}".format(time_str, step, loss, accuracy))def dev_step(x_batch, y_batch):"""Evaluate model on a dev setDisable dropout:param x_batch::param y_batch::param writer::return:"""feed_dict = {fasttext.input_x: x_batch,fasttext.input_y: y_batch,}step, loss, accuracy = sess.run([fasttext.global_step, fasttext.loss_val, fasttext.accuracy],feed_dict=feed_dict)time_str = datetime.datetime.now().isoformat()print("dev results:{}: step {}, loss {:g}, acc {:g}".format(time_str, step, loss, accuracy))# generate batchesbatches = data_process.batch_iter(list(zip(x_train, y_train)), FLAGS.batch_size, FLAGS.num_epochs)# training loopfor batch in batches:x_batch, y_batch = zip(*batch)train_step(x_batch, y_batch)current_step = tf.train.global_step(sess, fasttext.global_step)if current_step % FLAGS.validate_every == 0:print('\n Evaluation:')dev_step(x_dev, y_dev)print('')path = saver.save(sess, checkpoint_prefix, global_step=current_step)print('Save model checkpoint to {} \n'.format(path))def main(argv=None):x_train, y_train, vocab_processor, x_dev, y_dev = prepocess()train(x_train, y_train, vocab_processor, x_dev, y_dev)if __name__ == '__main__':tf.app.run()

對啦，我這里使用的數據集還是之前訓練CNN時的那一份

「完整代碼可以在公眾號后臺回復"ft"獲取。」

本文參考資料

[1]

Bag of Tricks for Efficient Text Classification: https://arxiv.org/abs/1607.01759

往期精彩回顧適合初學者入門人工智能的路線及資料下載機器學習在線手冊深度學習在線手冊AI基礎下載（pdf更新到25集）備注：加入本站微信群或者qq群，請回復“加群”獲取一折本站知識星球優惠券，請回復“知識星球”喜歡文章，點個在看

總結

以上是生活随笔為你收集整理的【NLP傻瓜式教程】手把手带你fastText文本分类(附代码)的全部內容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網站內容還不錯，歡迎將生活随笔推薦給好友。