首先更正一點(diǎn)，前篇博客里面說(shuō)到一個(gè)Canopy的測(cè)試的例子里面有這樣的一句代碼：

buildClusters(Configuration conf, Path input, Path output, DistanceMeasure measure, double t1, double t2, double t3, double t4, int clusterFilter, boolean runSequential)

我以前認(rèn)為clusterFilter是分類(lèi)的數(shù)目，其實(shí)應(yīng)該是每個(gè)分類(lèi)中至少含有的(樣本數(shù)+1)才對(duì)，而非分類(lèi)數(shù)，因?yàn)镃anopy就是要得到分類(lèi)數(shù)目的。

下面分析CanopyMapper：

首先把CanopyMapper改編為可以不依靠hadoop可以跑的代碼，即純java代碼，如下：

package mahout.test.canopy.debug;import java.util.ArrayList;import java.util.Collection;import java.util.HashMap;import java.util.List;import java.util.Map;import org.apache.hadoop.conf.Configuration;import org.apache.hadoop.io.Text;import org.apache.mahout.clustering.canopy.Canopy;import org.apache.mahout.clustering.canopy.CanopyClusterer;import org.apache.mahout.clustering.canopy.CanopyConfigKeys;import org.apache.mahout.common.distance.DistanceMeasure;import org.apache.mahout.common.distance.ManhattanDistanceMeasure;import org.apache.mahout.math.RandomAccessSparseVector;import org.apache.mahout.math.Vector;import org.apache.mahout.math.VectorWritable;import com.google.common.collect.Lists;public class CanopyDebug {/** * canopy 測(cè)試程序 * @author fansy * time 2013/7/21 22:59 * edited at 2013/7/22 21:38 */private static final Collection canopies = Lists.newArrayList();private static CanopyClusterer canopyClusterer;private static int clusterFilter;private static int num=0;private static Map center=new HashMap();public static void main(String[] args) { // 初始化DistanceMeasure measure=new ManhattanDistanceMeasure();double t1=30.2;double t2=5.3;int clusterFilter =1;Configuration conf=init(measure,t1,t2,clusterFilter );// setup() 函數(shù)setup(conf);// map()函數(shù) map();// cleanup() 函數(shù)Map result=cleanup();System.out.println("done..."+result); }/** * 初始化 * @param measure * @param t1 * @param t2 * @param clusterFileter：每個(gè)canopy至少含有的樣本數(shù) * @return */public static Configuration init(DistanceMeasure measure,double t1,double t2,int clusterFileter){Configuration conf=new Configuration();double t3=t1;double t4=t2; conf.set(CanopyConfigKeys.DISTANCE_MEASURE_KEY, measure.getClass() .getName()); conf.set(CanopyConfigKeys.T1_KEY, String.valueOf(t1)); conf.set(CanopyConfigKeys.T2_KEY, String.valueOf(t2)); conf.set(CanopyConfigKeys.T3_KEY, String.valueOf(t3)); conf.set(CanopyConfigKeys.T4_KEY, String.valueOf(t4)); conf.set(CanopyConfigKeys.CF_KEY, String.valueOf(clusterFilter)); return conf;}/** * map setup() 函數(shù) * @param conf 輸入初始參數(shù) */public static void setup(Configuration conf){canopyClusterer = new CanopyClusterer(conf); clusterFilter = Integer.parseInt(conf.get( CanopyConfigKeys.CF_KEY));}/** * 獲得初始數(shù)據(jù) --> * @return */public static List makeInData(){List list=new ArrayList();VectorWritable vw=null;Vector vector=null;for(int i=0;i<10;i++){vw=new VectorWritable();vector=new RandomAccessSparseVector(3);vector.set(0, i%3*(i%3)*(i%3)*(i%3)+Math.random());vector.set(1, i%3*(i%3)*(i%3)*(i%3)+Math.random());vector.set(2, i%3*(i%3)*(i%3)*(i%3)+Math.random());vw.set(vector);list.add(vw);}return list;}/** * 模仿Mapper的map函數(shù) */public static void map(){List vwList=makeInData2();for(VectorWritable point: vwList){canopyClusterer.addPointToCanopies(point.get(), canopies);}}/** * 模仿Mapper的cleanup函數(shù) * @return */public static Map cleanup(){for (Canopy canopy : canopies) { canopy.computeParameters(); if (canopy.getNumObservations() > clusterFilter) { center.put(new Text("centroid"+num++), new VectorWritable(canopy .getCenter())); } }return center;}/** * 固定輸入數(shù)據(jù)，產(chǎn)生輸入數(shù)據(jù)的第二種方式，方便調(diào)試 * @return */public static List makeInData2(){List list=new ArrayList();VectorWritable vw=null;Vector vector=null;for(int i=0;i<10;i++){vw=new VectorWritable();vector=new RandomAccessSparseVector(3);vector.set(0, i%3*(i%3)*(i%3)*(i%3)+1);vector.set(1, i%3*(i%3)*(i%3)*(i%3)+1);vector.set(2, i%3*(i%3)*(i%3)*(i%3)+1);vw.set(vector);list.add(vw);}return list;}}

利用上面的代碼可以直接進(jìn)行調(diào)試，調(diào)試可以看到每一步運(yùn)行的結(jié)果，很方便理解該算法。下面就結(jié)合實(shí)例進(jìn)行說(shuō)明：

輸入數(shù)據(jù)(VectorWritable格式)：

[1.0,1.0,1.0][2.0,2.0,2.0][17.0,17.0,17.0][1.0,1.0,1.0][2.0,2.0,2.0][17.0,17.0,17.0][1.0,1.0,1.0][2.0,2.0,2.0][17.0,17.0,17.0][1.0,1.0,1.0]

通過(guò)makeData2()函數(shù)即可獲得上面轉(zhuǎn)換成VectorWritable的數(shù)據(jù)，然后進(jìn)入map函數(shù)，其中的for循環(huán)即是仿造CanopyMapper中的map函數(shù)了。map函數(shù)里面使用到的函數(shù)為：addPointToCanopies，打開(kāi)CanopyCluster類(lèi)可以看到這個(gè)函數(shù)的主要內(nèi)容如下;public void addPointToCanopies(Vector point, Collection canopies) { boolean pointStronglyBound = false; for (Canopy canopy : canopies) { double dist = measure.distance(canopy.getCenter().getLengthSquared(), canopy.getCenter(), point); if (dist < t1) { if (log.isDebugEnabled()) { log.debug("Added point: {} to canopy: {}", AbstractCluster.formatVector(point, null), canopy.getIdentifier()); } canopy.observe(point); } pointStronglyBound = pointStronglyBound || dist < t2; } if (!pointStronglyBound) { if (log.isDebugEnabled()) { log.debug("Created new Canopy:{} at center:{}", nextCanopyId, AbstractCluster.formatVector(point, null)); } canopies.add(new Canopy(point, nextCanopyId++, measure)); } }

針對(duì)第一行的樣本[1.0,1.0,1.0]因?yàn)閏anopies為空，所以直接 執(zhí)行 canopies.add(new Canopy(point,nextCanopyId++,measure))；這樣就生成了第一個(gè)canopy，先看下Canopy這個(gè)類(lèi)，其父類(lèi)為DistanceMeasureCluster，父類(lèi)為AbstractCluster，其屬性有：private int id; private long numObservations; private long totalObservations; private Vector center; private Vector radius; // the observation statistics private double s0; private Vector s1; private Vector s2;

在這個(gè)函數(shù)里面canopy變化的屬性只有 s0,s1,s2 ，s0代表這個(gè)canopy里面的樣本數(shù)，s1代表所有樣本的對(duì)應(yīng)維度相加，s2代表所有樣本所有維度先平方再對(duì)應(yīng)相加；

針對(duì)第二行樣本[2.0,2.0,2.0],dist=3(曼哈頓距離)

針對(duì)第三行樣本[17,17,17]因?yàn)閐ist=16*3>t1=30.2所以直接執(zhí)行canopies.add()方法，即又添加了一個(gè)canopy；后面的樣本以此類(lèi)推；

在CanopMapper類(lèi)中最后執(zhí)行的方法為cleanup函數(shù)，這里也仿造了一個(gè)cleanup函數(shù)，其實(shí)cleanup函數(shù)的主要作用就是一個(gè)過(guò)濾和設(shè)置值的過(guò)程函數(shù)對(duì)應(yīng)為canopy.computeParameters()，過(guò)濾即是說(shuō)一個(gè)canopy中的樣本數(shù)必須要>clusterFilter才行，設(shè)置值即為設(shè)置radius和center值，具體代碼參見(jiàn)：

setNumObservations((long) getS0()); setTotalObservations(getTotalObservations() + getNumObservations()); setCenter(getS1().divide(getS0())); // compute the component stds if (getS0() > 1) { setRadius(getS2().times(getS0()).minus(getS1().times(getS1())).assign(new SquareRootFunction()).divide(getS0())); } setS0(0); setS1(center.like()); setS2(center.like());

center的值設(shè)置為s1/s0，radius的值：A=[s2*s0-s1.*s1]，(前一個(gè)是乘以，后面是點(diǎn)乘)然后把A的每一個(gè)維度的值開(kāi)方并除以s0即可得到radius。

分享，快樂(lè)，成長(zhǎng)

本文轉(zhuǎn)載自：CSDN博客

歡迎加入我愛(ài)機(jī)器學(xué)習(xí)QQ14群：336582044

微信掃一掃，關(guān)注我愛(ài)機(jī)器學(xué)習(xí)公眾號(hào)

總結(jié)

以上是生活随笔為你收集整理的canopy算法 java_mahout源码canopy算法分析之二CanopyMapper的全部?jī)?nèi)容，希望文章能夠幫你解決所遇到的問(wèn)題。

如果覺(jué)得生活随笔網(wǎng)站內(nèi)容還不錯(cuò)，歡迎將生活随笔推薦給好友。