文章目錄

• • 1. 從類別變量中提取特征
  • 2. 特征標準化
• • 3. 從文本中提取特征
  • • 3.1 詞袋模型
    • 3.2 停用詞過濾
    • 3.3 詞干提取和詞形還原
    • 3.4 TF-IDF 權重擴展詞包
    • 3.5 空間有效特征向量化與哈希技巧
    • 3.6 詞向量
  • 4. 從圖像中提取特征
  • • 4.1 從像素強度中提取特征
    • 4.2 使用卷積神經網絡激活項作為特征

本文為 scikit-learn機器學習（第2版）學習筆記

許多機器學習問題需要從 類別變量、文本、圖片中學習，需要從中提取出數字特征

1. 從類別變量中提取特征

通常使用 one-hot 編碼，產生2進制的編碼，會擴展數據，當數據值種類多時，不宜使用

from sklearn.feature_extraction import DictVectorizer onehot_encoder = DictVectorizer() X=[{'city':'Beijing'},{'city':'Guangzhou'},{'city':'Shanghai'} ] print(onehot_encoder.fit_transform(X).toarray()) [[1. 0. 0.][0. 1. 0.][0. 0. 1.]]

one-hot 編碼，沒有順序或大小之分，相比于用 0， 1， 2 來表示上述 3 個city，one-hot編碼更好

• DictVectorizer 只針對 string 變量，如果分類變量是數字類型，請使用 sklearn.preprocessing.OneHotEncoder

this transformer will only do a binary one-hot encoding when feature values are of type string.

If categorical features are represented as numeric values such as int, the DictVectorizer can be followed by sklearn.preprocessing.OneHotEncoder to complete binary one-hot encoding.

• DictVectorizer 對數字特征 失效案列：

X=[{'city':1},{'city':4},{'city':5} ] onehot_encoder = DictVectorizer() print(onehot_encoder.fit_transform(X).toarray()) [[1.][4.][5.]]

• OneHotEncoder 既可針對 string 類型，也可以對數字類型，進行編碼

# string 類型 from sklearn.preprocessing import OneHotEncoder import pandas as pd onehot_encoder = OneHotEncoder() X=[{'city':'Beijing'},{'city':'Guangzhou'},{'city':'Shanghai'} ] X = pd.DataFrame(X) print(onehot_encoder.fit_transform(X).toarray()) [[1. 0. 0.][0. 1. 0.][0. 0. 1.]] # 數字類型 onehot_encoder = OneHotEncoder() X=[{'city':1},{'city':4},{'city':5} ] X = pd.DataFrame(X) print(onehot_encoder.fit_transform(X).toarray()) [[1. 0. 0.][0. 1. 0.][0. 0. 1.]]

2. 特征標準化

• 防止特征淹沒，某些特征無法發揮作用
• 加快算法收斂

from sklearn import preprocessing import numpy as np X = np.array([[0., 0., 5., 13., 9., 1.],[0., 0., 13., 15., 10., 15.],[0., 3., 15., 2., 0., 11.] ]) s = preprocessing.StandardScaler() print(s.fit_transform(X))

StandardScaler 均值為0，方差為1

[[ 0. -0.70710678 -1.38873015 0.52489066 0.59299945 -1.35873244][ 0. -0.70710678 0.46291005 0.87481777 0.81537425 1.01904933][ 0. 1.41421356 0.9258201 -1.39970842 -1.4083737 0.33968311]]

RobustScaler 對異常值有更好的魯棒性，減輕異常值的影響

This Scaler removes the median and scales the data according to the quantile range (defaults to IQR: Interquartile Range).

The IQR is the range between the 1st quartile (25th quantile) and the 3rd quartile (75th quantile).

from sklearn.preprocessing import RobustScaler s = RobustScaler() print(s.fit_transform(X)) [[ 0. 0. -1.6 0. 0. -1.42857143][ 0. 0. 0. 0.30769231 0.2 0.57142857][ 0. 2. 0.4 -1.69230769 -1.8 0. ]]

3. 從文本中提取特征

文本通常為自然語言

3.1 詞袋模型

• 不會編碼任何文本句法，忽略單詞順序，忽略語法，忽略詞頻
• 可看做 one-hot 的一種擴展，會對文本中關注的每一個單詞創建一個特征
• 可用于文檔分類和檢索

corpus = ["UNC played Duke in basketball","Duke lost the basketball game" ] from sklearn.feature_extraction.text import CountVectorizer vectorizer = CountVectorizer() print(vectorizer.fit_transform(corpus).todense()) # [[1 1 0 1 0 1 0 1] # [1 1 1 0 1 0 1 0]] print(vectorizer.vocabulary_) # {'unc': 7, 'played': 5, 'duke': 1, 'in': 3, # 'basketball': 0, 'lost': 4, 'the': 6, 'game': 2}

• 注意：只會提取長度 >= 2 的單詞，添加一個句子，該句子的單詞 I，a 沒有向量化

corpus.append("I ate a sandwich and an apple") print(vectorizer.fit_transform(corpus).todense()) # [[0 0 0 0 1 1 0 1 0 1 0 0 1] # [0 0 0 0 1 1 1 0 1 0 0 1 0] # [1 1 1 1 0 0 0 0 0 0 1 0 0]] print(vectorizer.vocabulary_) # {'unc': 12, 'played': 9, 'duke': 5, 'in': 7, # 'basketball': 4, 'lost': 8, 'the': 11, 'game': 6, # 'ate': 3, 'sandwich': 10, 'and': 1, 'an': 0, 'apple': 2}

• 進行文本相似度計算，計算文本向量之間的歐氏距離（L₂范數）

from sklearn.metrics.pairwise import euclidean_distances X = vectorizer.fit_transform(corpus).todense() print("distance between doc1 and doc2 ", euclidean_distances(X[0],X[1])) print("distance between doc1 and doc3 ", euclidean_distances(X[0],X[2])) print("distance between doc2 and doc3 ", euclidean_distances(X[1],X[2])) # distance between doc1 and doc2 [[2.44948974]] # distance between doc1 and doc3 [[3.16227766]] # distance between doc2 and doc3 [[3.16227766]]

可以看出，文檔1跟文檔2更相似
真實環境中，詞匯數量相當大，需要的內存很大，為了緩和這個矛盾，采用稀疏向量
后序還有降維方法，來降低向量的維度

3.2 停用詞過濾

降維策略：

• 所有單詞轉成小寫，對單詞的意思沒有影響
• 忽略語料庫中大部分文檔中經常出現的單詞，如the\a\an\do \be\will\on\around等，稱之 stop_words
• CountVectorizer 可以通過 stop_words 關鍵詞參數，過濾停用詞，它本身也有一個基本的英語停用詞列表

vectorizer = CountVectorizer(stop_words='english') print(vectorizer.fit_transform(corpus).todense()) # [[0 0 1 1 0 0 1 0 1] # [0 0 1 1 1 1 0 0 0] # [1 1 0 0 0 0 0 1 0]] print(vectorizer.vocabulary_) # {'unc': 8, 'played': 6, 'duke': 3, 'basketball': 2, # 'lost': 5, 'game': 4, 'ate': 1, 'sandwich': 7, 'apple': 0}

我們發現 in\the\and\an不見了

3.3 詞干提取和詞形還原

停用詞列表包含的詞很少，過濾后依然包含很多單詞怎么辦？

• 詞干提取、詞形還原，進一步降維

例如，jumping\jumps\jump，一篇報道跳遠比賽的文章中，這幾個詞時分別編碼的，我們可以對他們進行統一處理，壓縮成單個特征

corpus = ['He ate the sandwiches','Every sandwich was eaten by him' ] vectorizer = CountVectorizer(binary=True, stop_words='english') print(vectorizer.fit_transform(corpus).todense()) # [[1 0 1 0] # [0 1 0 1]] print(vectorizer.vocabulary_) # {'ate': 0, 'sandwiches': 2, 'sandwishes': 3, 'eaten': 1}

我們看到這兩個句子表達的一個意思，特征向量卻沒有一個共同元素

• Lemmatizer 詞性還原
  注：NLTK WordNet 安裝 參考，解壓、添加路徑、重新打開python即可

corpus = ['I am gathering ingredients for the sandwich.','There were many peoples at the gathering.' ] from nltk.stem.wordnet import WordNetLemmatizer # help(WordNetLemmatizer) lemmatizer = WordNetLemmatizer() print(lemmatizer.lemmatize('gathering','v')) # gather，動詞 print(lemmatizer.lemmatize('gathering','n')) # gathering，名詞

• PorterStemmer 詞干提取

from nltk.stem import PorterStemmer # help(PorterStemmer) stemmer = PorterStemmer() print(stemmer.stem('gathering')) # gather

小例子：

from nltk import word_tokenize # 取詞 from nltk.stem import PorterStemmer # 詞干提取 from nltk.stem.wordnet import WordNetLemmatizer # 詞性還原 from nltk import pos_tag # 詞性標注wordnet_tags = ['n','v'] corpus = ['He ate the sandwiches','Every sandwich was eaten by him' ] stemmer = PorterStemmer() print("詞干：", [[stemmer.stem(word) for word in word_tokenize(doc)] for doc in corpus])# 詞干： [['He', 'ate', 'the', 'sandwich'], # ['everi', 'sandwich', 'wa', 'eaten', 'by', 'him']] def lemmatize(word, tag):if tag[0].lower() in ['n','v']:return lemmatizer.lemmatize(word, tag[0].lower())return word lemmatizer = WordNetLemmatizer() tagged_corpus = [pos_tag(word_tokenize(doc)) for doc in corpus]print(tagged_corpus) # [[('He', 'PRP'), ('ate', 'VBD'), ('the', 'DT'), ('sandwiches', 'NNS')], # [('Every', 'DT'), ('sandwich', 'NN'), ('was', 'VBD'), # ('eaten', 'VBN'), ('by', 'IN'), ('him', 'PRP')]]print('詞性還原：',[[lemmatize(word,tag) for word, tag in doc] for doc in tagged_corpus]) # 詞性還原： [['He', 'eat', 'the', 'sandwich'], # ['Every', 'sandwich', 'be', 'eat', 'by', 'him']]對 n,v 開頭的詞性的單詞進行了詞性還原

3.4 TF-IDF 權重擴展詞包

詞頻是很重要的，創建編碼單詞頻數的特征向量

import numpy as np from sklearn.feature_extraction.text import CountVectorizercorpus = ["The dog ate a sandwich, the people manufactured many sandwiches,\and I ate a sandwich"]vectorizer = CountVectorizer(stop_words='english') freq = np.array(vectorizer.fit_transform(corpus).todense()) freq # array([[2, 1, 1, 3]], dtype=int64) vectorizer.vocabulary_ # {'dog': 1, 'ate': 0, 'sandwich': 3, 'people': 2} for word, idx in vectorizer.vocabulary_.items():print(word, " 出現了 ", freq[0][idx]," 次") dog 出現了 1 次 ate 出現了 2 次 sandwich 出現了 2 次 people 出現了 1 次 manufactured 出現了 1 次 sandwiches 出現了 1 次

• sklearn 的TfidfVectorizer 可以統計單詞的權值：單詞頻率-逆文本頻率 TF-IDF
  

from sklearn.feature_extraction.text import TfidfVectorizer corpus = ["The dog ate a sandwich, and I ate a sandwich","the people manufactured a sandwich"] vectorizer = TfidfVectorizer(stop_words='english') print(vectorizer.fit_transform(corpus).todense()) print(vectorizer.vocabulary_) [[0.75458397 0.37729199 0. 0. 0.53689271][0. 0. 0.6316672 0.6316672 0.44943642]] {'dog': 1, 'ate': 0, 'sandwich': 4, 'people': 3, 'manufactured': 2}

3.5 空間有效特征向量化與哈希技巧

• 書上大概意思是說可以省內存，可以用于在線流式任務創建特征向量

from sklearn.feature_extraction.text import HashingVectorizer # help(HashingVectorizer) corpus = ['This is the first document.','This document is the second document.'] vectorizer = HashingVectorizer(n_features=2**4) X = vectorizer.fit_transform(corpus).todense() print(X) x = vectorizer.transform(['This is the first document.']).todense() print(x) x in X # True [[-0.57735027 0. 0. 0. 0. 0.0. 0. -0.57735027 0. 0. 0.0. 0.57735027 0. 0. ][-0.81649658 0. 0. 0. 0. 0.0. 0. 0. 0. 0. 0.408248290. 0.40824829 0. 0. ]] [[-0.57735027 0. 0. 0. 0. 0.0. 0. -0.57735027 0. 0. 0.0. 0.57735027 0. 0. ]]

3.6 詞向量

詞向量模型相比于詞袋模型更好些。

詞向量模型在類似的詞語上產生類似的詞向量（如，small、tiny都表示小），反義詞的向量則只在很少的幾個維度類似

# google colab 運行以下代碼 import gensim from google.colab import drive drive.mount('/gdrive') # !git clone https://github.com/mmihaltz/word2vec-GoogleNews-vectors.git ! wget -c "https://s3.amazonaws.com/dl4j-distribution/GoogleNews-vectors-negative300.bin.gz"!cd /content !gzip -d /content/GoogleNews-vectors-negative300.bin.gzmodel = gensim.models.KeyedVectors.load_word2vec_format('/content/GoogleNews-vectors-negative300.bin', binary=True) embedding = model.word_vec('cat') embedding.shape # (300,)相似度 print(model.similarity('cat','dog')) # 0.76094574 print(model.similarity('cat','sandwich')) # 0.17211203最相似的n個單詞 print(model.most_similar(positive=['good','ok'],negative=['bad'],topn=3)) # [('okay', 0.7390689849853516), # ('alright', 0.7239435911178589), # ('OK', 0.5975555777549744)]

4. 從圖像中提取特征

4.1 從像素強度中提取特征

將圖片的矩陣展平后作為特征向量

• 有缺點，產出的模型對縮放、旋轉、平移很敏感，對光照強度變化也很敏感

from sklearn import datasets digits = datasets.load_digits() print(digits.images[0].reshape(-1,64)) 圖片特征向量 [[ 0. 0. 5. 13. 9. 1. 0. 0. 0. 0. 13. 15. 10. 15. 5. 0. 0. 3.15. 2. 0. 11. 8. 0. 0. 4. 12. 0. 0. 8. 8. 0. 0. 5. 8. 0.0. 9. 8. 0. 0. 4. 11. 0. 1. 12. 7. 0. 0. 2. 14. 5. 10. 12.0. 0. 0. 0. 6. 13. 10. 0. 0. 0.]]

4.2 使用卷積神經網絡激活項作為特征

不懂，暫時跳過。

總結

以上是生活随笔為你收集整理的[scikit-learn 机器学习] 4. 特征提取的全部內容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網站內容還不錯，歡迎將生活随笔推薦給好友。