聚類(lèi)Clustering
This page describes clustering algorithms in MLlib. The guide for clustering in the RDD-based API also has relevant information about these algorithms. 本文描述MLlib中的聚類(lèi)算法。基于RDD-API中的聚類(lèi)指南提供了有關(guān)這些算法的相關(guān)信息。
Table of Contents
? K-means
o Input Columns
o Output Columns
? Latent Dirichlet allocation (LDA)
? Bisecting k-means
? Gaussian Mixture Model (GMM)
o Input Columns
o Output Columns
? Power Iteration Clustering (PIC)
K-means
k-means is one of the most commonly used clustering algorithms that clusters the data points into a predefined number of clusters. The MLlib implementation includes a parallelized variant of the k-means++ method called kmeans||.
KMeans is implemented as an Estimator and generates a KMeansModel as the base model.
k均值是最常用的聚類(lèi)算法之一，將數(shù)據(jù)點(diǎn)聚集成預(yù)定數(shù)量的聚類(lèi)。MLlib實(shí)現(xiàn)包括k-means ++方法的并行變體，稱(chēng)為kmeans ||。。
KMeans實(shí)現(xiàn)，Estimator生成KMeansModel作為基本模型。

Examples
? Scala
? Java
? Python
? R
Refer to the Scala API docs for more details.
import org.apache.spark.ml.clustering.KMeans
import org.apache.spark.ml.evaluation.ClusteringEvaluator

// Loads data.
val dataset = spark.read.format(“l(fā)ibsvm”).load(“data/mllib/sample_kmeans_data.txt”)

// Trains a k-means model.
val kmeans = new KMeans().setK(2).setSeed(1L)
val model = kmeans.fit(dataset)

// Make predictions
val predictions = model.transform(dataset)

// Evaluate clustering by computing Silhouette score
val evaluator = new ClusteringEvaluator()

val silhouette = evaluator.evaluate(predictions)
println(s"Silhouette with squared euclidean distance = $silhouette")

// Shows the result.
println("Cluster Centers: ")
model.clusterCenters.foreach(println)
Find full example code at “examples/src/main/scala/org/apache/spark/examples/ml/KMeansExample.scala” in the Spark repo.
Latent Dirichlet allocation (LDA)
LDA is implemented as an Estimator that supports both EMLDAOptimizer and OnlineLDAOptimizer, and generates a LDAModel as the base model. Expert users may cast a LDAModel generated by EMLDAOptimizer to a DistributedLDAModel if needed.
LDA實(shí)現(xiàn)Estimator，支持EMLDAOptimizer和OnlineLDAOptimizer，生成LDAModel作為基礎(chǔ)模型。專(zhuān)家用戶(hù)可以將LDAModel生成的 EMLDAOptimizer轉(zhuǎn)換為DistributedLDAModel。
Examples
? Scala
? Java
? Python
? R
Refer to the Scala API docs for more details.
import org.apache.spark.ml.clustering.LDA

// Loads data.
val dataset = spark.read.format(“l(fā)ibsvm”)
.load(“data/mllib/sample_lda_libsvm_data.txt”)

// Trains a LDA model.
val lda = new LDA().setK(10).setMaxIter(10)
val model = lda.fit(dataset)

val ll = model.logLikelihood(dataset)
val lp = model.logPerplexity(dataset)
println(s"The lower bound on the log likelihood of the entire corpus: $ll")
println(s"The upper bound on perplexity: $lp")

// Describe topics.
val topics = model.describeTopics(3)
println(“The topics described by their top-weighted terms:”)
topics.show(false)

// Shows the result.
val transformed = model.transform(dataset)
transformed.show(false)
Find full example code at “examples/src/main/scala/org/apache/spark/examples/ml/LDAExample.scala” in the Spark repo.
Bisecting k-means
Bisecting k-means is a kind of hierarchical clustering using a divisive (or “top-down”) approach: all observations start in one cluster, and splits are performed recursively as one moves down the hierarchy.
Bisecting K-means can often be much faster than regular K-means, but it will generally produce a different clustering.
BisectingKMeans is implemented as an Estimator and generates a BisectingKMeansModel as the base model.
將k均值平分是一種使用除法（或“自上而下”）方法的分層聚類(lèi)：所有觀測(cè)值都在一個(gè)聚類(lèi)中開(kāi)始，當(dāng)一個(gè)聚結(jié)向下移動(dòng)時(shí)，遞歸執(zhí)行拆分。
平分K均值通常會(huì)比常規(guī)K均值快得多，但通常會(huì)產(chǎn)生不同的聚類(lèi)。
BisectingKMeans實(shí)現(xiàn)，Estimator并生成BisectingKMeansModel作為基本模型。
Examples
? Scala
? Java
? Python
? R
Refer to the Scala API docs for more details.
import org.apache.spark.ml.clustering.BisectingKMeans
import org.apache.spark.ml.evaluation.ClusteringEvaluator

// Loads data.
val dataset = spark.read.format(“l(fā)ibsvm”).load(“data/mllib/sample_kmeans_data.txt”)

// Trains a bisecting k-means model.
val bkm = new BisectingKMeans().setK(2).setSeed(1)
val model = bkm.fit(dataset)

// Make predictions
val predictions = model.transform(dataset)

// Evaluate clustering by computing Silhouette score
val evaluator = new ClusteringEvaluator()

val silhouette = evaluator.evaluate(predictions)
println(s"Silhouette with squared euclidean distance = $silhouette")

// Shows the result.
println("Cluster Centers: ")
val centers = model.clusterCenters
centers.foreach(println)
Find full example code at “examples/src/main/scala/org/apache/spark/examples/ml/BisectingKMeansExample.scala” in the Spark repo.
Gaussian Mixture Model (GMM)
A Gaussian Mixture Model represents a composite distribution whereby points are drawn from one of k Gaussian sub-distributions, each with its own probability. The spark.ml implementation uses the expectation-maximization algorithm to induce the maximum-likelihood model given a set of samples.
GaussianMixture is implemented as an Estimator and generates a GaussianMixtureModel as the base model.
高斯混合模型 代表一個(gè)復(fù)合分布，繪制?高斯子分布，每個(gè)具有其相應(yīng)的概率。該spark.ml實(shí)現(xiàn)使用 期望最大化 算法，給定一組樣本，得出最大似然模型。
GaussianMixture實(shí)現(xiàn)，Estimator并生成GaussianMixtureModel作為基本模型。

Examples
? Scala
? Java
? Python
? R
Refer to the Scala API docs for more details.
import org.apache.spark.ml.clustering.GaussianMixture

// Loads data
val dataset = spark.read.format(“l(fā)ibsvm”).load(“data/mllib/sample_kmeans_data.txt”)

// Trains Gaussian Mixture Model
val gmm = new GaussianMixture()
.setK(2)
val model = gmm.fit(dataset)

// output parameters of mixture model model
for (i <- 0 until model.getK) {
println(s"Gaussian KaTeX parse error: Undefined control sequence: \nweight at position 3: i:\?n?w?e?i?g?h?t?={model.weights(i)}\n" +
s"mu=KaTeX parse error: Undefined control sequence: \nsigma at position 26: …ssians(i).mean}\?n?s?i?g?m?a?=\n{model.gaussians(i).cov}\n")
}
Find full example code at “examples/src/main/scala/org/apache/spark/examples/ml/GaussianMixtureExample.scala” in the Spark repo.
Power Iteration Clustering (PIC)
Power Iteration Clustering (PIC) is a scalable graph clustering algorithm developed by Lin and Cohen. From the abstract: PIC finds a very low-dimensional embedding of a dataset using truncated power iteration on a normalized pair-wise similarity matrix of the data.
spark.ml’s PowerIterationClustering implementation takes the following parameters:
功率迭代聚類(lèi)（PIC）是Lin和Cohen開(kāi)發(fā)的可伸縮圖聚類(lèi)算法。PIC在數(shù)據(jù)的標(biāo)準(zhǔn)化成對(duì)相似度矩陣上使用截?cái)嗟膬绱蔚?#xff0c;發(fā)現(xiàn)了數(shù)據(jù)集的非常低維的嵌入。
spark.ml的PowerIterationClustering實(shí)現(xiàn)采用以下參數(shù)：
? k: the number of clusters to create
? initMode: param for the initialization algorithm
? maxIter: param for maximum number of iterations
? srcCol: param for the name of the input column for source vertex IDs
? dstCol: name of the input column for destination vertex IDs
? weightCol: Param for weight column name
? k：要?jiǎng)?chuàng)建的聚類(lèi)數(shù)
? initMode：初始化算法的參數(shù)
? maxIter：最大迭代次數(shù)的參數(shù)
? srcCol：參數(shù)，用于源頂點(diǎn)ID的輸入列的名稱(chēng)
? dstCol：目標(biāo)頂點(diǎn)ID的輸入列的名稱(chēng)
? weightCol：權(quán)重列名稱(chēng)的參數(shù)
Examples
? Scala
? Java
? Python
? R
Refer to the Scala API docs for more details.
import org.apache.spark.ml.clustering.PowerIterationClustering

val dataset = spark.createDataFrame(Seq(
(0L, 1L, 1.0),
(0L, 2L, 1.0),
(1L, 2L, 1.0),
(3L, 4L, 1.0),
(4L, 0L, 0.1)
)).toDF(“src”, “dst”, “weight”)

val model = new PowerIterationClustering().
setK(2).
setMaxIter(20).
setInitMode(“degree”).
setWeightCol(“weight”)

val prediction = model.assignClusters(dataset).select(“id”, “cluster”)

// Shows the cluster assignment
prediction.show(false)
Find full example code at “examples/src/main/scala/org/apache/spark/examples/ml/PowerIterationClusteringExample.scala” in the Spark repo.

總結(jié)

以上是生活随笔為你收集整理的聚类Clustering的全部?jī)?nèi)容，希望文章能夠幫你解決所遇到的問(wèn)題。

如果覺(jué)得生活随笔網(wǎng)站內(nèi)容還不錯(cuò)，歡迎將生活随笔推薦給好友。