python集群

Natural Language Processing has made huge advancements in the last years. Currently, various implementations of neural networks are cutting edge and it seems that everybody talks about them. But, sometimes a simpler solution might be preferable. After all, one should try to walk before running. In this short article, I am going to demonstrate a simple method for clustering documents with Python. All code is available at GitHub (please note that it might be better to view the code in nbviewer).

在過去的幾年中，自然語言處理取得了巨大的進步。 當前，神經網絡的各種實現都是最前沿的，似乎每個人都在談論它們。 但是，有時更簡單的解決方案可能更可取。 畢竟，應該在跑步之前嘗試走路。 在這篇簡短的文章中，我將演示一種使用Python集群文檔的簡單方法。 所有代碼都可以在GitHub上獲得 (請注意，最好在nbviewer中查看代碼)。

We are going to cluster Wikipedia articles using k-means algorithm. The steps for doing that are the following:

我們將使用k-means算法對Wikipedia文章進行聚類。 為此，請執行以下步驟：

fetch some Wikipedia articles,

獲取一些維基百科文章，

2. represent each article as a vector,

2.將每篇文章表示為矢量，

3. perform k-means clustering,

3.執行k均值聚類，

4. evaluate the result.

4.評估結果。

1.獲取維基百科文章 (1. Fetch Wikipedia articles)

Using the wikipedia package it is very easy to download content from Wikipedia. For this example, we will use the content of the articles for:

使用Wikipedia軟件包，可以很容易地從Wikipedia下載內容。 在此示例中，我們將文章的內容用于：

• Data Science
  數據科學
• Artificial intelligence
  人工智能
• Machine Learning
  機器學習
• European Central Bank
  歐洲中央銀行
• Bank
  銀行
• Financial technology
  金融技術
• International Monetary Fund
  國際貨幣基金組織
• Basketball
  籃球
• Swimming
  游泳的
• Tennis
  網球

The content of each Wikipedia article is stored wiki_list while the title of each article is stored in variable title.

每個Wikipedia文章的內容都存儲在wiki_list中，而每個文章的標題都存儲在可變標題中。

import pandas as pd
import wikipediaarticles=['Data Science','Artificial intelligence','Machine Learning','European Central Bank','Bank','Financial technology','International Monetary Fund','Basketball','Swimming','Tennis']wiki_lst=[]
title=[]
for article in articles:
print("loading content: ",article)
wiki_lst.append(wikipedia.page(article).content)
title.append(article)print("examine content")
wiki_lstAt the top of the github page there is a button that will allow you to execute the notebook in Google’s Colab. Feel free to experiment!

2.將每篇文章表示為矢量 (2. Represent each article as a vector)

Since we are going to use k-means, we need to represent each article as a numeric vector. A popular method is to use term-frequency/inverse-document-frequency (tf-idf). Put it simply, with this method for each word w and document d we calculate:

由于我們將使用k均值，因此我們需要將每篇文章表示為一個數值向量。 一種流行的方法是使用術語頻率/文檔反向頻率(tf-idf) 。 簡而言之，使用此方法對每個單詞w和文檔d進行計算：

tf(w,d): the ratio of the number of appearances of w in d divided with the total number of words in d.

TF(W，d)：W的d與d的字的總數除以出現次數的比率。

idf(w): the logarithm of the fraction of the total number of documents divided by the number of documents that contain w.

idf(w)：文檔總數的分數的對數除以包含w的文檔數。

tfidf(w,d)=tf(w,d) x idf(w)

tfidf(w，d)= tf(w，d)x idf(w)

It is recommended that common, stop words are excluded. All the calculations are easily done with sklearn’s TfidfVectorizer.

建議排除常見的停用詞。 使用sklearn的TfidfVectorizer可以輕松完成所有計算。

from sklearn.feature_extraction.text import TfidfVectorizer
vectorizer = TfidfVectorizer(stop_words={'english'})
X = vectorizer.fit_transform(wiki_lst)

(To be honest, the calculation of sklearn is slightly different. You can read about in the documentation.)

(說實話，sklearn的計算略有不同。您可以在文檔中閱讀。)

3.執行k均值聚類 (3. Perform k-means clustering)

Each row of variable X is a vector representation of a Wikipedia article. Hence, we can use X as input for the k-means algorithm.

變量X的每一行都是Wikipedia文章的矢量表示。 因此，我們可以將X用作k-means算法的輸入。

First, we must decide on the number of clusters. Here, we will use the elbow method.

首先，我們必須決定群集的數量。 在這里，我們將使用彎頭法 。

import matplotlib.pyplot as plt
from sklearn.cluster import KMeansSum_of_squared_distances = []
K = range(2,10)for k in K:
km = KMeans(n_clusters=k, max_iter=200, n_init=10)
km = km.fit(X)
Sum_of_squared_distances.append(km.inertia_)plt.plot(K, Sum_of_squared_distances, 'bx-')
plt.xlabel('k')
plt.ylabel('Sum_of_squared_distances')
plt.title('Elbow Method For Optimal k')
plt.show()

The plot is almost a straight line, probably because we have to few articles. But at a closer examination a dent appears for k=4 or k=6. We will try to cluster into 6 groups.

該圖幾乎是一條直線，可能是因為我們篇幅很少。 但是在仔細檢查時，凹痕出現為k = 4或k = 6。 我們將嘗試分為6組。

true_k = 6
model = KMeans(n_clusters=true_k, init='k-means++', max_iter=200, n_init=10)
model.fit(X)
labels=model.labels_
wiki_cl=pd.DataFrame(list(zip(title,labels)),columns=['title','cluster'])
print(wiki_cl.sort_values(by=['cluster']))Result of clustering聚類結果

4.評估結果 (4. Evaluate the result)

Since we have used only 10 articles, it is fairly easy to evaluate the clustering just by examining what articles are contained in each cluster. That would be difficult for a large corpus. A nice way is to create a word cloud from the articles of each cluster.

由于我們僅使用了10篇文章，因此僅通過檢查每個聚類中包含哪些文章就可以很容易地評估聚類。 對于大型語料庫來說，這將是困難的。 一種不錯的方法是根據每個群集的文章創建詞云。

from wordcloud import WordCloud
result={'cluster':labels,'wiki':wiki_lst}
result=pd.DataFrame(result)
for k in range(0,true_k):
s=result[result.cluster==k]
text=s['wiki'].str.cat(sep=' ')
text=text.lower()
text=' '.join([word for word in text.split()])
wordcloud = WordCloud(max_font_size=50, max_words=100, background_color="white").generate(text)
print('Cluster: {}'.format(k))
print('Titles')
titles=wiki_cl[wiki_cl.cluster==k]['title']
print(titles.to_string(index=False))
plt.figure()
plt.imshow(wordcloud, interpolation="bilinear")
plt.axis("off")
plt.show()Wordclouds of the clusters集群的Wordclouds

• Cluster 0 consists of articles on European Central Bank, Bank and International Monetary Fund
  第0組包含有關歐洲中央銀行，銀行和國際貨幣基金組織的文章
• Cluster 1 consists of articles on Artificial intelligence and Machine Learning
  第1組包含有關人工智能和機器學習的文章
• Cluster 2 has an article for Swimming
  類別2有一篇關于游泳的文章
• Cluster 3 has an article for Data Science
  集群3上有一篇關于數據科學的文章
• Cluster 4 has articles on Basketball and Tennis
  第4組中有關籃球和網球的文章
• Cluster 5 has an article on Financial Technology
  第5組中有一篇關于金融技術的文章

It might seem odd that swimming is not in the same cluster as basketball and tennis. Or that AI and Machine Learning are not in the same group with Data Science. That is because we choose to create 6 clusters. But by looking at the word clouds we can see that articles on basket and tennis have words like game, player, team, ball in common while the article on swimming does not.

游泳與籃球和網球不在同一組，這似乎很奇怪。 或者，人工智能和機器學習與數據科學不在同一個組中。 那是因為我們選擇創建6個集群。 但是通過看云這個詞，我們可以看到籃筐和網球上的文章有共同的詞，例如比賽，球員，球隊，球，而游泳方面的文章則沒有。

At the top of the GitHub page, there is a button that will allow you to execute the notebook in Google’s Colab. You can easily try a different number of clusters. The results might surprise you!

在GitHub頁面的頂部，有一個按鈕，可讓您在Google的Colab中執行筆記本。 您可以輕松嘗試其他數量的群集。 結果可能會讓您感到驚訝！

翻譯自: https://towardsdatascience.com/clustering-documents-with-python-97314ad6a78d

python集群

總結

以上是生活随笔為你收集整理的python集群_使用Python集群文档的全部內容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網站內容還不錯，歡迎將生活随笔推薦給好友。