目錄

文章目錄

• 目錄
• 前言
• 1.CART回歸樹簡介
• 2.剪枝策略
• 3.模型樹
• 4.線性回歸 回歸樹 模型樹比較

前言

雖然許多問題都可以用線性方法取得良好的結果，但現(xiàn)實中也有許多問題是非線性的，用線性模型并不能很好的擬合數(shù)據(jù)，這種情況下可以使用樹回歸來擬合數(shù)據(jù)。因此本文特別介紹一下CART, 樹剪枝，模型樹等等算法。

1.CART回歸樹簡介

傳統(tǒng)決策樹是一種貪心算法，在給定時間內(nèi)做出最佳選擇，不關心是否達到全局最優(yōu)。切分過于迅速，特征一旦使用后面將不再使用。不能處理連續(xù)型特征，進行離散化可能會破壞連續(xù)變量的內(nèi)在特征。

CART 分類回歸樹，既能分類又能回歸。CRAT來進行節(jié)點決策時，使用二元切分來處理連續(xù)型變量，給定特征屬性以及特征值，若大于該值則執(zhí)行左子樹，相反則放入右子樹。當某個節(jié)點不能再切分時，節(jié)點值是單個值（CART）,也可以是一個線性方程（模型樹）。

加載數(shù)據(jù)集 按行加載到矩陣中：

def loadDataSet(fileName): #general function to parse tab -delimited floatsdataMat = [] #assume last column is target valuefr = open(fileName)for line in fr.readlines():curLine = line.strip().split('\t')fltLine = list(map(float,curLine)) #map all elements to float()dataMat.append(fltLine)return dataMat

按照某一特征以及響應特征值來對數(shù)據(jù)集進行劃分：

feature是特征屬性的索引 即列數(shù) value劃分閾值 如果value大于閾值則放入mat0 否則放入mat1

def binSplitDataSet(dataSet, feature, value):mat0 = dataSet[nonzero(dataSet[:,feature] > value)[0],:]mat1 = dataSet[nonzero(dataSet[:,feature] <= value)[0],:]return mat0,mat1

創(chuàng)建回歸樹：

找到最佳待切分屬性;

如果該節(jié)點不能切分，則該節(jié)點存為子節(jié)點

執(zhí)行二元切分

右子樹調(diào)用createTree()

左子樹調(diào)用createTree()

leafType 創(chuàng)建葉節(jié)點的函數(shù) errType代表誤差計算函數(shù) 每一個節(jié)點使用字典來存儲，分別包含spInd spVal left right等key值。

def createTree(dataSet, leafType=regLeaf, errType=regErr, ops=(1,4)):#assume dataSet is NumPy Mat so we can array filteringfeat, val = chooseBestSplit(dataSet, leafType, errType, ops)#choose the best splitif feat == None: return val #if the splitting hit a stop condition return valretTree = {}retTree['spInd'] = featretTree['spVal'] = vallSet, rSet = binSplitDataSet(dataSet, feat, val)retTree['left'] = createTree(lSet, leafType, errType, ops)retTree['right'] = createTree(rSet, leafType, errType, ops)return retTree

樹節(jié)點劃分的度量，計算連續(xù)函數(shù)的混亂度（決策樹使用信息熵以及基尼純等），這里可以采用數(shù)據(jù)的總方差來計算數(shù)據(jù)的混亂度，均方差乘以數(shù)據(jù)集的樣本數(shù)。

遍歷所有特征以及所有特征值使總方差最小的值即為劃分特征以及劃分閾值。

def chooseBestSplit(dataSet, leafType=regLeaf, errType=regErr, ops=(1,4)):tolS = ops[0]; tolN = ops[1]#if all the target variables are the same value: quit and return valueif len(set(dataSet[:,-1].T.tolist()[0])) == 1: #exit cond 1return None, leafType(dataSet)m,n = shape(dataSet)#the choice of the best feature is driven by Reduction in RSS error from meanS = errType(dataSet)bestS = inf; bestIndex = 0; bestValue = 0for featIndex in range(n-1):for splitVal in set((dataSet[:,featIndex].T.A.tolist())[0]):mat0, mat1 = binSplitDataSet(dataSet, featIndex, splitVal)if (shape(mat0)[0] < tolN) or (shape(mat1)[0] < tolN): continuenewS = errType(mat0) + errType(mat1)if newS < bestS: bestIndex = featIndexbestValue = splitValbestS = newS#if the decrease (S-bestS) is less than a threshold don't do the splitif (S - bestS) < tolS: return None, leafType(dataSet) #exit cond 2mat0, mat1 = binSplitDataSet(dataSet, bestIndex, bestValue)if (shape(mat0)[0] < tolN) or (shape(mat1)[0] < tolN): #exit cond 3return None, leafType(dataSet)return bestIndex,bestValue#returns the best feature to split on#and the value used for that split

tolS為容許的誤差最小下降值，當劃分一次誤差小于該值時，提升效果不大，直接返回。

tolN為切分的最少樣本數(shù)，當切分之后，左右子數(shù)量小于tolN，說明切分字節(jié)過小，直接返回。

leafType為葉子結點的創(chuàng)建函數(shù)，采用均值方式

def regLeaf(dataSet):#returns the value used for each leafreturn mean(dataSet[:,-1])

errType為誤差估計函數(shù)，這里使用總方差，即均方差乘以樣本總數(shù)

def regErr(dataSet):return var(dataSet[:,-1]) * shape(dataSet)[0]

如果某個節(jié)點數(shù)據(jù)特征值都相同，則無法繼續(xù)劃分，直接返回葉子結點。

if len(set(dataSet[:,-1].T.tolist()[0])) == 1:

遍歷每一個特征以及相應的特征值來進行劃分，計算每一種劃分的總方差，返回最優(yōu)的特征屬性以及特征閾值：

for featIndex in range(n-1):for splitVal in set(dataSet[:,featIndex]):

繪出樣本集的分布圖：

def plotarr(arr):import matplotlib.pyplot as pltfig = plt.figure()ax = fig.add_subplot(111)ax.scatter(arr[:,0].flatten().A[0], arr[:,1].flatten().A[0])plt.show()

運行測試如下：

加載另一數(shù)據(jù)集：

得到CART回歸結果：

2.剪枝策略

當回歸樹葉子結點過多時，容易發(fā)生過擬合，導致泛化性能降低。可以采取剪枝來防止過擬合，有預剪枝以及后剪枝。

def chooseBestSplit(dataSet, leafType=regLeaf, errType=regErr, ops=(1,4)):tolS = ops[0]; tolN = ops[1]#if all the target variables are the same value: quit and return valueif len(set(dataSet[:,-1].T.tolist()[0])) == 1: #exit cond 1return None, leafType(dataSet)m,n = shape(dataSet)#the choice of the best feature is driven by Reduction in RSS error from meanS = errType(dataSet)bestS = inf; bestIndex = 0; bestValue = 0for featIndex in range(n-1):for splitVal in set((dataSet[:,featIndex].T.A.tolist())[0]):mat0, mat1 = binSplitDataSet(dataSet, featIndex, splitVal)if (shape(mat0)[0] < tolN) or (shape(mat1)[0] < tolN): continuenewS = errType(mat0) + errType(mat1)if newS < bestS: bestIndex = featIndexbestValue = splitValbestS = newS#if the decrease (S-bestS) is less than a threshold don't do the splitif (S - bestS) < tolS: return None, leafType(dataSet) #exit cond 2mat0, mat1 = binSplitDataSet(dataSet, bestIndex, bestValue)if (shape(mat0)[0] < tolN) or (shape(mat1)[0] < tolN): #exit cond 3return None, leafType(dataSet)return bestIndex,bestValue#returns the best feature to split on#and the value used for that split

其中tolS與tolN就能在一定程度上防止過擬合，主要采用預剪枝。通過tolS如果剪枝對于數(shù)據(jù)集的誤差降低不大則可以不劃分節(jié)點，tolN如果剪枝之后葉子結點數(shù)據(jù)過少，則也可以預剪枝處理。這對參數(shù)tolS,tolN的取值提出了很高的要求，往往難以設置求解。

后剪枝：將數(shù)據(jù)分為訓練集與測試集，首先構建一顆完整樹，然后依次尋找葉子結點，用測試集來判斷將葉子結點合并是否能降低測試誤差，若能則采取后剪枝。

基于已有的樹切分測試數(shù)據(jù)：
如果存在任一子集是一棵樹，在該子集繼續(xù)剪枝過程。
計算將兩個葉子結點合并后的誤差
計算不合并的誤差
若合并會降低誤差，則合并兩個葉子結點

判斷某一節(jié)點是否是一棵樹，及判斷是否為字典類型：

def isTree(obj):return (type(obj).__name__=='dict')

執(zhí)行樹坍塌過程，返回樹的平均值

def getMean(tree):if isTree(tree['right']): tree['right'] = getMean(tree['right'])if isTree(tree['left']): tree['left'] = getMean(tree['left'])return (tree['left']+tree['right'])/2.0

進行后剪枝處理：

def prune(tree, testData):if shape(testData)[0] == 0: return getMean(tree) #if we have no test data collapse the tree#如果該樹是子集，則劃分測試數(shù)據(jù)，繼續(xù)后剪枝if (isTree(tree['right']) or isTree(tree['left'])):#if the branches are not trees try to prune themlSet, rSet = binSplitDataSet(testData, tree['spInd'], tree['spVal'])if isTree(tree['left']): tree['left'] = prune(tree['left'], lSet)if isTree(tree['right']): tree['right'] = prune(tree['right'], rSet)#if they are now both leafs, see if we can merge them#如果節(jié)點是葉子結點if not isTree(tree['left']) and not isTree(tree['right']):#劃分測試數(shù)據(jù)lSet, rSet = binSplitDataSet(testData, tree['spInd'], tree['spVal'])#沒有合并前的誤差errorNoMerge = sum(power(lSet[:,-1] - tree['left'],2)) +\sum(power(rSet[:,-1] - tree['right'],2))#合并后的誤差 合并后 節(jié)點值變?yōu)閮蓚€子節(jié)點的平均值treeMean = (tree['left']+tree['right'])/2.0#取出最后一列y的值真實值與預測值求總方差errorMerge = sum(power(testData[:,-1] - treeMean,2))if errorMerge < errorNoMerge: print ("merging")return treeMeanelse: return treeelse: return tree

加載數(shù)據(jù)集，創(chuàng)建一顆最完整的分類回歸樹， 設置tolS=0, tolN=1

>>> dd = mat(regTrees.loadDataSet('ex2.txt')) >>> mt = regTrees.createTree(dd, ops(0,1))

此時回歸樹

剪枝之后：

有一部分節(jié)點被剪掉。

3.模型樹

前面CART葉子結點為某個值，現(xiàn)在可以把葉子結點變?yōu)橐粋€分段函數(shù)，即某一個葉子結點下面允許分段函數(shù)形式的數(shù)據(jù)存在。

將一個數(shù)據(jù)集求出線性擬合函數(shù)：

def linearSolve(dataSet): #helper function used in two placesm,n = shape(dataSet)X = mat(ones((m,n))); Y = mat(ones((m,1)))#create a copy of data with 1 in 0th postionX[:,1:n] = dataSet[:,0:n-1]; Y = dataSet[:,-1]#and strip out YxTx = X.T*Xif linalg.det(xTx) == 0.0:raise NameError('This matrix is singular, cannot do inverse,\n\try increasing the second value of ops')ws = xTx.I * (X.T * Y)return ws,X,Y

首先進行數(shù)據(jù)矩陣變換，利用線性模型直接求解回歸系數(shù)ws

如果一個節(jié)點是葉子結點時，需要存儲ws系數(shù)權向量

def modelLeaf(dataSet):#create linear model and return coeficientsws,X,Y = linearSolve(dataSet)return ws

當采用線性模型時，使用平方誤差和來計算總誤差：

def modelErr(dataSet):ws,X,Y = linearSolve(dataSet)yHat = X * wsreturn sum(power(Y - yHat,2))

加載數(shù)據(jù)集進行測試：

def testmodel():tt = mat(loadDataSet('exp2.txt'))return createTree(tt, modelLeaf, modelErr, (1, 10))

數(shù)據(jù)集分布：

4.線性回歸 回歸樹 模型樹比較

通過對于同一份數(shù)據(jù)進行訓練模型，在通過測試集比較不同模型之間的性能差異。

模型樹與回歸樹預測值的輸出：

def treeForeCast(tree, inData, modelEval=regTreeEval):if not isTree(tree): return modelEval(tree, inData)if inData[tree['spInd']] > tree['spVal']:if isTree(tree['left']): return treeForeCast(tree['left'], inData, modelEval)else: return modelEval(tree['left'], inData)else:if isTree(tree['right']): return treeForeCast(tree['right'], inData, modelEval)else: return modelEval(tree['right'], inData)

tree訓練樹所得，inData為待預測的樣本行向量，modelEval表示節(jié)點類型，當modelEval=regTreeEval說明葉子節(jié)點為分類類型，節(jié)點值為具體的分類值，即預測值值直接返回節(jié)點值即可，當modelEval=modelTreeEval時，說明葉子結點為回歸類型，節(jié)點值為線性權向量，返回值應該與測試數(shù)據(jù)相乘得到最終預測值。

def regTreeEval(model, inDat):return float(model)def modelTreeEval(model, inDat):n = shape(inDat)[1]X = mat(ones((1,n+1)))X[:,1:n+1]=inDatreturn float(X*model)

返回測試集的預測值，列向量：

def createForeCast(tree, testData, modelEval=regTreeEval):m=len(testData)yHat = mat(zeros((m,1)))for i in range(m):yHat[i,0] = treeForeCast(tree, mat(testData[i]), modelEval)return yHat

利用相關系數(shù)來衡量數(shù)據(jù)擬合情況：

def regtree():traindata = mat(loadDataSet('bikeSpeedVsIq_train.txt'))testdata = mat(loadDataSet('bikeSpeedVsIq_test.txt'))mt = createTree(traindata, ops=(1, 20))yHat = createForeCast(mt, testdata[:,0])return corrcoef(yHat, testdata[:,1], rowvar=0)[0,1]def modeltree():traindata = mat(loadDataSet('bikeSpeedVsIq_train.txt'))testdata = mat(loadDataSet('bikeSpeedVsIq_test.txt'))mt = createTree(traindata, modelLeaf, modelErr, ops=(1, 20))yHat = createForeCast(mt, testdata[:,0], modelTreeEval)return corrcoef(yHat, testdata[:,1], rowvar=0)[0,1]def reg():traindata = mat(loadDataSet('bikeSpeedVsIq_train.txt'))testdata = mat(loadDataSet('bikeSpeedVsIq_test.txt'))ws, x, y = linearSolve(traindata)yHat=[0]*shape(testdata)[0]for i in range(shape(testdata)[0]):yHat[i] = testdata[i,0]*ws[1,0]+ws[0,0]return corrcoef(yHat, testdata[:,1], rowvar=0)[0,1]

可知模型樹擬合效果最好
5.Tkinter庫圖形化

使用tkinter庫來實現(xiàn)是圖畫化展示數(shù)據(jù)擬合

from numpy import *#python3導入方式不變from tkinter import *import regTreesimport matplotlibmatplotlib.use('TkAgg')from matplotlib.backends.backend_tkagg import FigureCanvasTkAggfrom matplotlib.figure import Figuredef reDraw(tolS,tolN):reDraw.f.clf() # clear the figurereDraw.a = reDraw.f.add_subplot(111)if chkBtnVar.get():if tolN < 2: tolN = 2#繪出模型樹myTree=regTrees.createTree(reDraw.rawDat, regTrees.modelLeaf,\regTrees.modelErr, (tolS,tolN))yHat = regTrees.createForeCast(myTree, reDraw.testDat, \regTrees.modelTreeEval)else:#繪出回歸樹myTree=regTrees.createTree(reDraw.rawDat, ops=(tolS,tolN))yHat = regTrees.createForeCast(myTree, reDraw.testDat)#繪出數(shù)據(jù)分布時，矩陣和一位數(shù)組之間的轉(zhuǎn)換reDraw.a.scatter(reDraw.rawDat[:,0].flatten().A[0], reDraw.rawDat[:,1].flatten().A[0], s=5) #use scatter for data setreDraw.a.plot(reDraw.testDat, yHat, linewidth=2.0) #use plot for yHat#修改draw()為show()reDraw.canvas.draw()def getInputs():try: tolN = int(tolNentry.get())except: tolN = 10 print ("enter Integer for tolN")tolNentry.delete(0, END)tolNentry.insert(0,'10')try: tolS = float(tolSentry.get())except: tolS = 1.0 print ("enter Float for tolS")tolSentry.delete(0, END)tolSentry.insert(0,'1.0')return tolN,tolSdef drawNewTree():tolN,tolS = getInputs()#get values from Entry boxesreDraw(tolS,tolN)root=Tk()reDraw.f = Figure(figsize=(5,4), dpi=100) #create canvasreDraw.canvas = FigureCanvasTkAgg(reDraw.f, master=root)# show()方法應該修改為draw()reDraw.canvas.draw()reDraw.canvas.get_tk_widget().grid(row=0, columnspan=3)Label(root, text="tolN").grid(row=1, column=0)tolNentry = Entry(root)tolNentry.grid(row=1, column=1)tolNentry.insert(0,'10')Label(root, text="tolS").grid(row=2, column=0)tolSentry = Entry(root)tolSentry.grid(row=2, column=1)tolSentry.insert(0,'1.0')Button(root, text="ReDraw", command=drawNewTree).grid(row=1, column=2, rowspan=3)chkBtnVar = IntVar()chkBtn = Checkbutton(root, text="Model Tree", variable = chkBtnVar)chkBtn.grid(row=3, column=0, columnspan=2)reDraw.rawDat = mat(regTrees.loadDataSet('sine.txt'))reDraw.testDat = arange(min(reDraw.rawDat[:,0]),max(reDraw.rawDat[:,0]),0.01)reDraw(1.0, 10)root.mainloop()

由于python3的變化，代碼需要改變?nèi)缦?#xff1a;

1.from tkinter import * 庫導入庫名變?yōu)樾?/p>

2.reDraw.canvas.draw() FigureCanvasTkAgg對象draw方法而不是show()

3.reDraw.a.scatter(reDraw.rawDat[:,0].flatten().A[0], reDraw.rawDat[:,1].flatten().A[0], s=5)進行數(shù)據(jù)分布繪制時需要轉(zhuǎn)換矩陣為一維數(shù)組。

改變tolS tolN的值，繪制如下：

總結

以上是生活随笔為你收集整理的机器学习--CART分类回归树的全部內(nèi)容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網(wǎng)站內(nèi)容還不錯，歡迎將生活随笔推薦給好友。