目的

分析jieba如何從句子中拆分詞的

預備知識

xxx

jieba api

jieba提供三種分詞方式

if cut_all: # 123cut_block = self.__cut_allelif HMM: # 123cut_block = self.__cut_DAGelse: # 123cut_block = self.__cut_DAG_NO_HMM

流程

建立圖(dag)

根據圖來計算如何切分

圖的原理

圖是如何建立的

只有在詞頻表里的詞才會記錄下來，儲存上對應的index作為圖

def get_DAG(self, sentence):self.check_initialized() # 初始化一下DAG = {} # 圖其實就是一個dict[int, list[int]]N = len(sentence)for k in xrange(N):tmplist = [] # 用來存儲是詞的索引i = kfrag = sentence[k]while i < N and frag in self.FREQ:if self.FREQ[frag]: # FREQ是一個dict[str, int]，這個是提前load進來的詞頻表，可以根據自己的需求更改tmplist.append(i)i += 1frag = sentence[k:i + 1]if not tmplist:tmplist.append(k)DAG[k] = tmplistreturn DAG

第一種分詞邏輯，直接遍歷出詞

def __cut_all(self, sentence):dag = self.get_DAG(sentence)old_j = -1eng_scan = 0eng_buf = u''for k, L in iteritems(dag):if eng_scan == 1 and not re_eng.match(sentence[k]):eng_scan = 0yield eng_bufif len(L) == 1 and k > old_j:word = sentence[k:L[0] + 1]if re_eng.match(word):if eng_scan == 0:eng_scan = 1eng_buf = wordelse:eng_buf += wordif eng_scan == 0:yield wordold_j = L[0]else:for j in L:if j > k:yield sentence[k:j + 1]old_j = jif eng_scan == 1:yield eng_buf

第二種HMM

也是先做一個圖

計算一個路徑，那么問題是路徑是干什么的，如何計算路徑

def __cut_DAG(self, sentence):DAG = self.get_DAG(sentence)route = {}self.calc(sentence, DAG, route) # 這里就是計算路徑x = 0buf = ''N = len(sentence)while x < N:y = route[x][1] + 1l_word = sentence[x:y]if y - x == 1:buf += l_wordelse:if buf:if len(buf) == 1:yield bufbuf = ''else:if not self.FREQ.get(buf):recognized = finalseg.cut(buf)for t in recognized:yield telse:for elem in buf:yield elembuf = ''yield l_wordx = yif buf:if len(buf) == 1:yield bufelif not self.FREQ.get(buf):recognized = finalseg.cut(buf)for t in recognized:yield telse:for elem in buf:yield elem

route

def calc(self, sentence, DAG, route): # route是一個dict[int, tuple[int, int]]，是索引所對應的N = len(sentence)route[N] = (0, 0)logtotal = log(self.total)for idx in xrange(N - 1, -1, -1):route[idx] = max((log(self.FREQ.get(sentence[idx:x + 1]) or 1) -logtotal + route[x + 1][0], x) for x in DAG[idx])

ref

jieba分詞流程之DAG、Route

總結

以上是生活随笔為你收集整理的NLP jieba分词源码解析的全部內容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網站內容還不錯，歡迎將生活随笔推薦給好友。