原文：http://www.jianshu.com/p/94516a58314d

• Dataset transformations| 數據轉換
• Combining estimators|組合學習器
• Feature extration|特征提取
• Preprocessing data|數據預處理

---

?

1 Dataset transformations

---

scikit-learn provides a library of transformers, which may clean (see Preprocessing data), reduce (see Unsupervised dimensionality reduction), expand (see Kernel Approximation) or generate (see Feature extraction) feature representations.

scikit-learn 提供了數據轉換的模塊，包括數據清理、降維、擴展和特征提取。

Like other estimators, these are represented by classes with fit method, which learns model parameters (e.g. mean and standard deviation for normalization) from a training set, and a transform method which applies this transformation model to unseen data. fit_transform may be more convenient and efficient for modelling and transforming the training data simultaneously.

scikit-learn模塊有3種通用的方法：fit(X,y=None)、transform(X)、fit_transform(X)、inverse_transform(newX)。fit用來訓練模型；transform在訓練后用來降維；fit_transform先用訓練模型，然后返回降維后的X；inverse_transform用來將降維后的數據轉換成原始數據。

?

1.1 combining estimators

• ?

  1.1.1 Pipeline:chaining estimators

  Pipeline 模塊是用來組合一系列估計器的。對固定的一系列操作非常便利，如：同時結合特征選擇、數據標準化、分類。
  • Usage|使用
    代碼： from sklearn.pipeline import Pipeline from sklearn.svm import SVC from sklearn.decomposition import PCA from sklearn.pipeline import make_pipeline #define estimators #the arg is a list of (key,value) pairs,where the key is a string you want to give this step and value is an estimators object estimators=[('reduce_dim',PCA()),('svm',SVC())] #combine estimators clf1=Pipeline(estimators) clf2=make_pipeline(PCA(),SVC()) #use func make_pipeline() can do the same thing print(clf1,'\n',clf2) 輸出： Pipeline(steps=[('reduce_dim', PCA(copy=True, n_components=None, whiten=False)), ('svm', SVC(C=1.0, cache_size=200, class_weight=None, coef0=0.0, decision_function_shape=None, degree=3, gamma='auto', kernel='rbf', max_iter=-1, probability=False, random_state=None, shrinking=True, tol=0.001, verbose=False))]) Pipeline(steps=[('pca', PCA(copy=True, n_components=None, whiten=False)), ('svc', SVC(C=1.0, cache_size=200, class_weight=None, coef0=0.0, decision_function_shape=None, degree=3, gamma='auto', kernel='rbf', max_iter=-1, probability=False, random_state=None, shrinking=True, tol=0.001, verbose=False))]) 可以通過set_params()方法設置學習器的屬性，參數形式為<estimator>_<parameter> clf.set_params(svm__C=10) 上面的方法在網格搜索時很重要： from sklearn.grid_search import GridSearchCV params = dict(reduce_dim__n_components=[2, 5, 10],svm__C=[0.1, 10, 100]) grid_search = GridSearchCV(clf, param_grid=params) 上面的例子相當于把pipeline生成的學習器作為一個普通的學習器，參數形式為<estimator>_<parameter>。
  • Note|說明
    1.可以使用dir()函數查看clf的所有屬性和方法。例如step屬性就是每個操作步驟的屬性。
    如
    >>> clf.steps[0] ('reduce_dim', PCA(copy=True, n_components=None, whiten=False))
    2.調用pipeline生成的學習器的fit方法相當于依次調用其包含的所有學習器的方法，transform輸入然后把結果扔向下一步驟。pipeline生成的學習器有著它包含的學習器的所有方法。如果最后一個學習器是分類，那么生成的學習器就是分類，如果最后一個是transform，那么生成的學習器就是transform，依次類推。

• ?

  1.1.2 FeatureUnion: composite feature spaces

  與pipeline不同的是FeatureUnion只組合transformer，它們也可以結合成更復雜的模型。

  FeatureUnion combines several transformer objects into a new transformer that combines their output. AFeatureUnion takes a list of transformer objects. During fitting, each of these is fit to the data independently. For transforming data, the transformers are applied in parallel, and the sample vectors they output are concatenated end-to-end into larger vectors.

  • Usage|使用
    代碼：

    from sklearn.pipeline import FeatureUnion from sklearn.decomposition import PCA from sklearn.decomposition import KernelPCA from sklearn.pipeline import make_union #define transformers #the arg is a list of (key,value) pairs,where the key is a string you want to give this step and value is an transformer object estimators=[('linear_pca)',PCA()),('Kernel_pca',KernelPCA())] #combine transformers clf1=FeatureUnion(estimators) clf2=make_union(PCA(),KernelPCA()) print(clf1,'\n',clf2) print(dir(clf1))

    輸出：

    FeatureUnion(n_jobs=1,transformer_list=[('linear_pca)', PCA(copy=True, n_components=None, whiten=False)), ('Kernel_pca', KernelPCA(alpha=1.0, coef0=1, degree=3, eigen_solver='auto', fit_inverse_transform=False, gamma=None, kernel='linear', kernel_params=None, max_iter=None, n_components=None, remove_zero_eig=False, tol=0))], transformer_weights=None) FeatureUnion(n_jobs=1, transformer_list=[('pca', PCA(copy=True, n_components=None, whiten=False)), ('kernelpca', KernelPCA(alpha=1.0, coef0=1, degree=3, eigen_solver='auto', fit_inverse_transform=False, gamma=None, kernel='linear', kernel_params=None, max_iter=None, n_components=None, remove_zero_eig=False, tol=0))], transformer_weights=None)

    可以看出FeatureUnion的用法與pipeline一致

  • Note|說明

    (A FeatureUnion has no way of checking whether two transformers might produce identical features. It only produces a union when the feature sets are disjoint, and making sure they are is the caller’s responsibility.)

    Here is a example python source code:feature_stacker.py

?

1.2 Feature extraction

The sklearn.feature_extraction module can be used to extract features in a format supported by machine learning algorithms from datasets consisting of formats such as text and image.

skilearn.feature_extraction模塊是用機器學習算法所支持的數據格式來提取數據，如將text和image信息轉換成dataset。
Note:
Feature extraction（特征提取）與Feature selection（特征選擇）不同，前者是用來將非數值的數據轉換成數值的數據，后者是用機器學習的方法對特征進行學習（如PCA降維）。

• ?

  1.2.1 Loading features from dicts

  The class DictVectorizer can be used to convert feature arrays represented as lists of standard Python dict
  objects to the NumPy/SciPy representation used by scikit-learn estimators.
  Dictvectorizer類用來將python內置的dict類型轉換成數值型的array。dict類型的好處是在存儲稀疏數據時不用存儲無用的值。

  代碼：

  measurements=[{'city': 'Dubai', 'temperature': 33.} ,{'city': 'London', 'temperature':12.} ,{'city':'San Fransisco','temperature':18.},] from sklearn.feature_extraction import DictVectorizer vec=DictVectorizer() x=vec.fit_transform(measurements).toarray() print(x) print(vec.get_feature_names())

  輸出：

  [[ 1. 0. 0. 33.] [ 0. 1. 0. 12.] [ 0. 0. 1. 18.]] ['city=Dubai', 'city=London', 'city=San Fransisco', 'temperature'] [Finished in 0.8s]

• ?

  1.2.2 Feature hashing

• ?

  1.2.3 Text feature extraction

• ?

  1.2.4 Image feature extraction

  以上三小節暫未考慮（設計到語言處理及圖像處理）見官方文檔

?

1.3 Preprogressing data

The sklearn.preprocessing
package provides several common utility functions and transformer classes to change raw feature vectors into a representation that is more suitable for the downstream estimators

sklearn.preprogressing模塊提供了幾種常見的數據轉換，如標準化、歸一化等。

• ?

  1.3.1 Standardization, or mean removal and variance scaling

  Standardization of datasets is a common requirement for many machine learning estimators implemented in the scikit; they might behave badly if the individual features do not more or less look like standard normally distributed data: Gaussian with zero mean and unit variance.

  很多學習算法都要求事先對數據進行標準化，如果不是像標準正太分布一樣0均值1方差就可能會有很差的表現。

  • Usage|用法

  代碼：

  from sklearn import preprocessing import numpy as np X = np.array([[1.,-1., 2.], [2.,0.,0.], [0.,1.,-1.]]) Y=X Y_scaled = preprocessing.scale(Y) y_mean=Y_scaled.mean(axis=0) #If 0, independently standardize each feature, otherwise (if 1) standardize each sample|axis=0 時求每個特征的均值，axis=1時求每個樣本的均值 y_std=Y_scaled.std(axis=0) print(Y_scaled) scaler= preprocessing.StandardScaler().fit(Y)#用StandardScaler類也能完成同樣的功能 print(scaler.transform(Y))

  輸出：

  [[ 0. -1.22474487 1.33630621] [ 1.22474487 0. -0.26726124] [-1.22474487 1.22474487 -1.06904497]] [[ 0. -1.22474487 1.33630621] [ 1.22474487 0. -0.26726124] [-1.22474487 1.22474487 -1.06904497]] [Finished in 1.4s]
  • Note|說明
    1.func scale
    2.class StandardScaler
    3.StandardScaler 是一種Transformer方法，可以讓pipeline來使用。
    MinMaxScaler （min-max標準化[0,1]）類和MaxAbsScaler（[-1,1]）類是另外兩個標準化的方式，用法和StandardScaler類似。
    4.處理稀疏數據時用MinMax和MaxAbs很合適
    5.魯棒的數據標準化方法（適用于離群點很多的數據處理）：

    the median and the interquartile range often give better results

  用中位數代替均值（使均值為0），用上四分位數-下四分位數代替方差（IQR為1？）。

• ?

  1.3.2 Impution of missing values|缺失值的處理

  • Usage
    代碼： import scipy.sparse as sp from sklearn.preprocessing import Imputer X=sp.csc_matrix([[1,2],[0,3],[7,6]]) imp=preprocessing.Imputer(missing_value=0,strategy='mean',axis=0) imp.fit(X) X_test=sp.csc_matrix([[0, 2], [6, 0], [7, 6]]) print(X_test) print(imp.transform(X_test)) 輸出： (1, 0) 6 (2, 0) 7 (0, 1) 2 (2, 1) 6 [[ 4. 2. ] [ 6. 3.66666675] [ 7. 6. ]] [Finished in 0.6s]
  • Note
    1.scipy.sparse是用來存儲稀疏矩陣的
    2.Imputer可以用來處理scipy.sparse稀疏矩陣

• ?

  1.3.3 Generating polynomial features

  • Usage
    代碼：

    import numpy as np from sklearn.preprocessing import PolynomialFeatures X=np.arange(6).reshape(3,2) print(X) poly=PolynomialFeatures(2) print(poly.fit_transform(X))

    輸出：

    [[0 1] [2 3] [4 5]] [[ 1. 0. 1. 0. 0. 1.] [ 1. 2. 3. 4. 6. 9.] [ 1. 4. 5. 16. 20. 25.]] [Finished in 0.8s]

  • Note
    生成多項式特征用在多項式回歸中以及多項式核方法中 。

• ?

  1.3.4 Custom transformers

  這是用來構造transform方法的函數

  • Usage:
    代碼： import numpy as np from sklearn.preprocessing import FunctionTransformer transformer = FunctionTransformer(np.log1p) x=np.array([[0,1],[2,3]]) print(transformer.transform(x)) 輸出： [[ 0. 0.69314718] [ 1.09861229 1.38629436]] [Finished in 0.8s]
  • Note

    For a full code example that demonstrates using a FunctionTransformer to do custom feature selection, see Using FunctionTransformer to select columns

文／houhzize（簡書作者）
原文鏈接：http://www.jianshu.com/p/94516a58314d
著作權歸作者所有，轉載請聯系作者獲得授權，并標注“簡書作者”。

轉載于:https://www.cnblogs.com/zhizhan/p/5557357.html

與50位技術專家面對面20年技術見證，附贈技術全景圖

總結

以上是生活随笔為你收集整理的python data analysis | python数据预处理（基于scikit-learn模块）的全部內容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網站內容還不錯，歡迎將生活随笔推薦給好友。