本文實例講述了python實現中文分詞FMM算法。分享給大家供大家參考。具體分析如下：

FMM算法的最簡單思想是使用貪心算法向前找n個，如果這n個組成的詞在詞典中出現，就ok，如果沒有出現，那么找n-1個...然后繼續下去。假如n個詞在詞典中出現，那么從n+1位置繼續找下去，直到句子結束。

import re

def PreProcess(sentence,edcode="utf-8"):

sentence = sentence.decode(edcode)

sentence=re.sub(u"[。，,！……!《》<>\"':：？\?、\|“”‘'；]"," ",sentence)

return sentence

def FMM(sentence,diction,result = [],maxwordLength = 4,edcode="utf-8"):

i = 0

sentence = PreProcess(sentence,edcode)

length = len(sentence)

while i < length:

# find the ascii word

tempi=i

tok=sentence[i:i+1]

while re.search("[0-9A-Za-z\-\+#@_\.]{1}",tok)<>None:

i= i+1

tok=sentence[i:i+1]

if i-tempi>0:

result.append(sentence[tempi:i].lower().encode(edcode))

# find chinese word

left = len(sentence[i:])

if left == 1:

"""go to 4 step over the FMM"""

"""should we add the last one? Yes, if not blank"""

if sentence[i:] <> " ":

result.append(sentence[i:].encode(edcode))

return result

m = min(left,maxwordLength)

for j in xrange(m,0,-1):

leftword = sentence[i:j+i].encode(edcode)

# print leftword.decode(edcode)

if LookUp(leftword,diction):

# find the left word in dictionary

# it's the right one

i = j+i

result.append(leftword)

break

elif j == 1:

"""only one word, add into result, if not blank"""

if leftword.decode(edcode) <> " ":

result.append(leftword)

i = i+1

else:

continue

return result

def LookUp(word,dictionary):

if dictionary.has_key(word):

return True

return False

def ConvertGBKtoUTF(sentence):

return sentence.decode('gbk').encode('utf-8')

dictions = {}

dictions["ab"] = 1

dictions["cd"] = 2

dictions["abc"] = 1

dictions["ss"] = 1

dictions[ConvertGBKtoUTF("好的")] = 1

dictions[ConvertGBKtoUTF("真的")] = 1

sentence = "asdfa好的是這樣嗎vasdiw呀真的daf dasfiw asid是嗎？"

s = FMM(ConvertGBKtoUTF(sentence),dictions)

for i in s:

print i.decode("utf-8")

test = open("test.txt","r")

for line in test:

s = FMM(CovertGBKtoUTF(line),dictions)

for i in s:

print i.decode("utf-8")

運行結果如下：

asdfa

好的

是

這

樣

嗎

vasdiw

呀

真的

daf

dasfiw

asid

是

嗎

？

希望本文所述對大家的Python程序設計有所幫助。

本文標題: python實現中文分詞FMM算法實例

本文地址: http://www.cppcns.com/jiaoben/python/127911.html

總結

以上是生活随笔為你收集整理的中文分词算法python代码_python实现中文分词FMM算法实例的全部內容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網站內容還不錯，歡迎將生活随笔推薦給好友。