• TensorFlow是一個(gè)面向數(shù)值計(jì)算的通用平臺(tái)，可以方便地訓(xùn)練線性模型。下面采用TensorFlow完成Andrew Ng主講的Deep Learning課程練習(xí)題，提供了整套源碼。

  線性回歸
  多元線性回歸
  邏輯回歸

線性回歸

# -*- coding: utf-8 -*- """ Created on Wed Sep 6 19:46:04 2017@author: Administrator """#!/usr/bin/env python # -*- coding=utf-8 -*- # @author: ranjiewen # @date: 2017-9-6 # @description: compare scikit-learn and tensorflow, using linear regression data from deep learning course by Andrew Ng. # @ref: http://openclassroom.stanford.edu/MainFolder/DocumentPage.php?course=DeepLearning&doc=exercises/ex2/ex2.htmlimport tensorflow as tf import numpy as np from sklearn import linear_model# Read x and y #x_data = np.loadtxt("ex2x.dat") #y_data = np.loadtxt("ex2y.dat")x_data = np.random.rand(100).astype(np.float32) y_data = x_data * 0.1 + 0.3+np.random.rand(100)# We use scikit-learn first to get a sense of the coefficients reg = linear_model.LinearRegression() reg.fit(x_data.reshape(-1, 1), y_data)print ("Coefficient of scikit-learn linear regression: k=%f, b=%f" % (reg.coef_, reg.intercept_))# Then we apply tensorflow to achieve the similar results # The structure of tensorflow code can be divided into two parts:# First part: set up computation graph W = tf.Variable(tf.random_uniform([1], -1.0, 1.0)) b = tf.Variable(tf.zeros([1])) y = W * x_data + bloss = tf.reduce_mean(tf.square(y - y_data)) / 2 # 對(duì)于tensorflow，梯度下降的步長(zhǎng)alpha參數(shù)需要很仔細(xì)的設(shè)置，步子太大容易扯到蛋導(dǎo)致無(wú)法收斂；步子太小容易等得蛋疼。迭代次數(shù)也需要細(xì)致的嘗試。 optimizer = tf.train.GradientDescentOptimizer(0.07) # Try 0.1 and you will see unconvergency train = optimizer.minimize(loss)init = tf.initialize_all_variables()# Second part: launch the graph sess = tf.Session() sess.run(init)for step in range(1500):sess.run(train)if step % 100 == 0:print (step, sess.run(W), sess.run(b)) print ("Coeeficient of tensorflow linear regression: k=%f, b=%f" % (sess.run(W), sess.run(b)))

• 思考：對(duì)于tensorflow，梯度下降的步長(zhǎng)alpha參數(shù)需要很仔細(xì)的設(shè)置，步子太大容易扯到蛋導(dǎo)致無(wú)法收斂；步子太小容易等得蛋疼。迭代次數(shù)也需要細(xì)致的嘗試。

多元線性回歸

# -*- coding: utf-8 -*- """ Created on Wed Sep 6 19:53:24 2017@author: Administrator """import numpy as np import tensorflow as tf from numpy import mat from sklearn import linear_model from sklearn import preprocessing# Read x and y #x_data = np.loadtxt("ex3x.dat").astype(np.float32) #y_data = np.loadtxt("ex3y.dat").astype(np.float32)x_data = [np.random.rand(100).astype(np.float32),np.random.rand(100).astype(np.float32)+10] x_data=mat(x_data).T y_data = 5.3+np.random.rand(100)# We evaluate the x and y by sklearn to get a sense of the coefficients. reg = linear_model.LinearRegression() reg.fit(x_data, y_data) print ("Coefficients of sklearn: K=%s, b=%f" % (reg.coef_, reg.intercept_))# Now we use tensorflow to get similar results.# Before we put the x_data into tensorflow, we need to standardize it # in order to achieve better performance in gradient descent; # If not standardized, the convergency speed could not be tolearated. # Reason: If a feature has a variance that is orders of magnitude larger than others, # it might dominate the objective function # and make the estimator unable to learn from other features correctly as expected. # 對(duì)于梯度下降算法，變量是否標(biāo)準(zhǔn)化很重要。在這個(gè)例子中，變量一個(gè)是面積，一個(gè)是房間數(shù)，量級(jí)相差很大，如果不歸一化，面積在目標(biāo)函數(shù)和梯度中就會(huì)占據(jù)主導(dǎo)地位，導(dǎo)致收斂極慢。 scaler = preprocessing.StandardScaler().fit(x_data) print (scaler.mean_, scaler.scale_) x_data_standard = scaler.transform(x_data)W = tf.Variable(tf.zeros([2, 1])) b = tf.Variable(tf.zeros([1, 1])) y = tf.matmul(x_data_standard, W) + bloss = tf.reduce_mean(tf.square(y - y_data.reshape(-1, 1)))/2 optimizer = tf.train.GradientDescentOptimizer(0.3) train = optimizer.minimize(loss)init = tf.initialize_all_variables()sess = tf.Session() sess.run(init) for step in range(100):sess.run(train)if step % 10 == 0:print (step, sess.run(W).flatten(), sess.run(b).flatten())print ("Coefficients of tensorflow (input should be standardized): K=%s, b=%s" % (sess.run(W).flatten(), sess.run(b).flatten())) print ("Coefficients of tensorflow (raw input): K=%s, b=%s" % (sess.run(W).flatten() / scaler.scale_, sess.run(b).flatten() - np.dot(scaler.mean_ / scaler.scale_, sess.run(W))))

• 思路：對(duì)于梯度下降算法，變量是否標(biāo)準(zhǔn)化很重要。在這個(gè)例子中，變量一個(gè)是面積，一個(gè)是房間數(shù)，量級(jí)相差很大，如果不歸一化，面積在目標(biāo)函數(shù)和梯度中就會(huì)占據(jù)主導(dǎo)地位，導(dǎo)致收斂極慢。

邏輯回歸

• 數(shù)據(jù)下載：Exercise: Logistic Regression and Newton's Method

# -*- coding: utf-8 -*- """ Created on Wed Sep 6 20:13:15 2017 數(shù)據(jù)下載：http://openclassroom.stanford.edu/MainFolder/DocumentPage.php?course=DeepLearning&doc=exercises/ex4/ex4.html@author: Administrator """import tensorflow as tf import numpy as np from numpy import mat from sklearn.linear_model import LogisticRegression from sklearn import preprocessing# Read x and y x_data = np.loadtxt("ex4Data/ex4x.dat").astype(np.float32) y_data = np.loadtxt("ex4Data/ex4y.dat").astype(np.float32)#x_data = [np.random.rand(100).astype(np.float32),np.random.rand(100).astype(np.float32)+10] #x_data=mat(x_data).T #y_data = 5.3+np.random.rand(100)scaler = preprocessing.StandardScaler().fit(x_data) x_data_standard = scaler.transform(x_data)# We evaluate the x and y by sklearn to get a sense of the coefficients. reg = LogisticRegression(C=999999999, solver="newton-cg") # Set C as a large positive number to minimize the regularization effect reg.fit(x_data, y_data) print ("Coefficients of sklearn: K=%s, b=%f" % (reg.coef_, reg.intercept_))# Now we use tensorflow to get similar results. W = tf.Variable(tf.zeros([2, 1])) b = tf.Variable(tf.zeros([1, 1])) y = 1 / (1 + tf.exp(-tf.matmul(x_data_standard, W) + b)) loss = tf.reduce_mean(- y_data.reshape(-1, 1) * tf.log(y) - (1 - y_data.reshape(-1, 1)) * tf.log(1 - y))optimizer = tf.train.GradientDescentOptimizer(1.3) train = optimizer.minimize(loss)init = tf.initialize_all_variables()sess = tf.Session() sess.run(init) for step in range(100):sess.run(train)if step % 10 == 0:print (step, sess.run(W).flatten(), sess.run(b).flatten())print ("Coefficients of tensorflow (input should be standardized): K=%s, b=%s" % (sess.run(W).flatten(), sess.run(b).flatten())) print ("Coefficients of tensorflow (raw input): K=%s, b=%s" % (sess.run(W).flatten() / scaler.scale_, sess.run(b).flatten() - np.dot(scaler.mean_ / scaler.scale_, sess.run(W))))# Problem solved and we are happy. But... # I'd like to implement the logistic regression from a multi-class viewpoint instead of binary. # In machine learning domain, it is called softmax regression # In economic and statistics domain, it is called multinomial logit (MNL) model, proposed by Daniel McFadden, who shared the 2000 Nobel Memorial Prize in Economic Sciences.print ("------------------------------------------------") print ("We solve this binary classification problem again from the viewpoint of multinomial classification") print ("------------------------------------------------")# As a tradition, sklearn first reg = LogisticRegression(C=9999999999, solver="newton-cg", multi_class="multinomial") reg.fit(x_data, y_data) print ("Coefficients of sklearn: K=%s, b=%f" % (reg.coef_, reg.intercept_)) print ("A little bit difference at first glance. What about multiply them with 2?")# Then try tensorflow W = tf.Variable(tf.zeros([2, 2])) # first 2 is feature number, second 2 is class number b = tf.Variable(tf.zeros([1, 2])) V = tf.matmul(x_data_standard, W) + b y = tf.nn.softmax(V) # tensorflow provide a utility function to calculate the probability of observer n choose alternative i, you can replace it with `y = tf.exp(V) / tf.reduce_sum(tf.exp(V), keep_dims=True, reduction_indices=[1])`# Encode the y label in one-hot manner lb = preprocessing.LabelBinarizer() lb.fit(y_data) y_data_trans = lb.transform(y_data) y_data_trans = np.concatenate((1 - y_data_trans, y_data_trans), axis=1) # Only necessary for binary class loss = tf.reduce_mean(-tf.reduce_sum(y_data_trans * tf.log(y), reduction_indices=[1])) optimizer = tf.train.GradientDescentOptimizer(1.3) train = optimizer.minimize(loss)init = tf.initialize_all_variables()sess = tf.Session() sess.run(init) for step in range(100):sess.run(train)if step % 10 == 0:print (step, sess.run(W).flatten(), sess.run(b).flatten())print ("Coefficients of tensorflow (input should be standardized): K=%s, b=%s" % (sess.run(W).flatten(), sess.run(b).flatten())) print ("Coefficients of tensorflow (raw input): K=%s, b=%s" % ((sess.run(W) / scaler.scale_).flatten(), sess.run(b).flatten() - np.dot(scaler.mean_ / scaler.scale_, sess.run(W))))

• 思考：
• 對(duì)于邏輯回歸，損失函數(shù)比線性回歸模型復(fù)雜了一些。首先需要通過(guò)sigmoid函數(shù)，將線性回歸的結(jié)果轉(zhuǎn)化為0至1之間的概率值。然后寫(xiě)出每個(gè)樣本的發(fā)生概率（似然），那么所有樣本的發(fā)生概率就是每個(gè)樣本發(fā)生概率的乘積。為了求導(dǎo)方便，我們對(duì)所有樣本的發(fā)生概率取對(duì)數(shù)，保持其單調(diào)性的同時(shí)，可以將連乘變?yōu)榍蠛?#xff08;加法的求導(dǎo)公式比乘法的求導(dǎo)公式簡(jiǎn)單很多）。對(duì)數(shù)極大似然估計(jì)方法的目標(biāo)函數(shù)是最大化所有樣本的發(fā)生概率；機(jī)器學(xué)習(xí)習(xí)慣將目標(biāo)函數(shù)稱(chēng)為損失，所以將損失定義為對(duì)數(shù)似然的相反數(shù)，以轉(zhuǎn)化為極小值問(wèn)題。
• 我們提到邏輯回歸時(shí)，一般指的是二分類(lèi)問(wèn)題；然而這套思想是可以很輕松就拓展為多分類(lèi)問(wèn)題的，在機(jī)器學(xué)習(xí)領(lǐng)域一般稱(chēng)為softmax回歸模型。本文的作者是統(tǒng)計(jì)學(xué)與計(jì)量經(jīng)濟(jì)學(xué)背景，因此一般將其稱(chēng)為MNL模型。

Reference：

• 基礎(chǔ)練習(xí)：線性模型

總結(jié)

以上是生活随笔為你收集整理的tensorflow基础练习：线性模型的全部?jī)?nèi)容，希望文章能夠幫你解決所遇到的問(wèn)題。

如果覺(jué)得生活随笔網(wǎng)站內(nèi)容還不錯(cuò)，歡迎將生活随笔推薦給好友。