PCA（Principal Component Analysis） 是一種常見的數據分析方式，常用于高維數據的降維，可用于提取數據的主要特征分量。

首先介紹一些關于線性降維的基本思想，用于線性PCA的實現。在文章后半部分會總結非線性PCA的實現方法，即Kernel PCA。

Linear PCA

Linear dimensionality reduction

Dimentionality reduction

Map the dataset with dimension d to the dataset with dimension k, and d > k.

$$x={?\begin{array}{c}{x}_1\\ ?\\ x_d\end{array}?}_{d\times 1}\to {?\begin{array}{c}{\hat{y}}_1\\ ?\\ {\hat{y}}_k\end{array}?}_{k\times 1}=\hat{y},k?d$$

Linear Encoding / Embedding

$$\begin{gathered} \hat{y}=b+W^{T}x \\ b={?\begin{array}{c}{b}_1\\ ?\\ b_k\end{array}?}_{k\times 1}W={?\begin{array}{ccc}|& ...& |\\ w_1& ...& w_k\\ |& ...& |\end{array}?}_{d\times k} \end{gathered}$$
將 X 數據集使用線性關系映射到 Y上，使維度降低。

Linear Decoding / Reconstruction

$$\hat{y}={?\begin{array}{c}{\hat{y}}_1\\ ?\\ {\hat{y}}_k\end{array}?}_{k\times 1}\to {?\begin{array}{c}{\hat{x}}_1\\ ?\\ {\hat{x}}_d\end{array}?}_{d\times 1}=\hat{\mathbf{x}},d?k$$

$$\hat{\mathbf{x}}=\mathbf{a}+\mathbf{V}\hat{\mathbf{y}}$$

$$\mathbf{a}={?\begin{array}{c}{a}_1\\ ?\\ a_d\end{array}?}_{d\times 1}\mathbf{V}={?\begin{array}{ccccc}|& & ?& & |\\ v_1& & ?& & v_k\\ |& & ?& & |\end{array}?}_{d\times k}$$

將 Y 用線性關系映射回 X，使得數據集重構。

PCA Objective

• Reconstruction MSE:
  $$g(\mathbf{b},\mathbf{W},\mathbf{a},\mathbf{V})=\frac{1}{n}\sum _{j=1}^n\parallel {\mathbf{x}}_{\mathbf{j}}?{\hat{\mathbf{x}}}_{\mathbf{j}}{\parallel }^2$$

  $$=\frac{1}{n}\sum _{j=1}^n\parallel {\mathbf{x}}_{\mathbf{j}}?\mathbf{a}?\mathbf{V}{\mathbf{y}}_{\mathbf{j}}{\parallel }^2$$

  $$=\frac{1}{n}\sum _{j=1}^n\parallel {\mathbf{x}}_{\mathbf{j}}?\mathbf{a}?\mathbf{V}(\mathbf{b}+{\mathbf{W}}^{\mathbf{T}}{\mathbf{x}}_{\mathbf{j}}){\parallel }^2$$

• Learning problem: given training feature vectors
  $$x_1,...x_n$$
  Find b, W, a, V, to minimize reconstruciton MSE:
  $$({\mathbf{b}}_{\mathbf{P}\mathbf{C}\mathbf{A}},{\mathbf{W}}_{\mathbf{P}\mathbf{C}\mathbf{A}},{\mathbf{a}}_{\mathbf{P}\mathbf{C}\mathbf{A}},{\mathbf{V}}_{\mathbf{P}\mathbf{C}\mathbf{A}})=argming(\mathbf{b},\mathbf{W},\mathbf{a},\mathbf{V})$$

PCA terminology

• Principal directions:
  $${\mathbf{w}}_{1,PCA},?,{\mathbf{w}}_{k,PCA}$$

• Principal components:

  • coordinates:
    $${\hat{y}}_{1,PCA},?,{\hat{y}}_{k,PCA}$$

  • Principal directions scaled by coordinates
    $$({\hat{y}}_{1,PCA}?{\mathbf{w}}_{1,PCA}),?,({\hat{y}}_{k,PCA}?{\mathbf{w}}_{k,PCA})$$

  • Principal subspace:
    $$Span({\mathbf{w}}_{1,PCA},?,{\mathbf{w}}_{k,PCA})$$

PCA classical solution

$$W_{PCA}=V_{PCA}={?\begin{array}{ccccc}|& & ?& & |\\ u_1& & ?& & u_k\\ |& & ?& & |\end{array}?}_{d\times k}$$

$$,a_{PCA}={\hat{\mu }}_x,b_{PCA}=?V_{PCA}^T{\hat{\mu }}_x$$

where,
$${\hat{\mu }}_x=\frac{1}{n}\sum _{j=1}^n{x}_j$$
is the d x 1 empirical mean feature vector
$${\hat{S}}_x=\frac{1}{n}\sum _{j=1}^n(x_j?{\hat{\mu }}_x)(x_j?{\hat{\mu }}_x{)}^T$$
is the d x d empirical feature covariance matrix

and
$$u_1,?,u_k$$
are the orthonormal eigenvectors of Sx corresponding to its top k eigenvalues
$${\lambda }_1,?,{\lambda }_k$$

• Need to re-arrange the eigenvectors and corresponding eigenvalues of Sx such that eigenvalues appear in a non-increasing order.

• If Sx has rank r, then
  $${\lambda }_r>0,and{\lambda }_{r+1}=?{\lambda }_d=0$$

Geometric visualization

選取方向向量，使數據集投射到該方向向量上時投射誤差盡可能小。比如選擇Top K個方向向量即是選擇k個principal vectors，使得總的投射誤差最小（最能反映數據集的特征）。

此時特征值eigenvalue => variance of data on each direction

**投射誤差：**數據點到方向向量作垂線的長度

以三維的數據集為例

數據集在三個方向向量的投影，投射誤差按u1， u2， u3遞增。

PCA Algorithm

Step 1: Compute d x 1 empirical mean feature vector:
$${\hat{\mu }}_{\mathbf{x}}=\frac{1}{n}\sum _{j=1}^n{x}_j$$
Step 2: Compute d x d empirical feature covariance matrix:
$${\hat{S}}_x=\frac{1}{n}\sum _{j=1}^n(x_j?{\hat{\mu }}_x)(x_j?{\hat{\mu }}_x{)}^T$$
Step 3: Compute eigen-decompostion of Sx:

• Sort the eigenvalues of Sx and re-arrange the eigenvectors corresponding to eigenvalues.

• u1, … uk are the corresponding orthonormal eigenvectors

• $$W_{PCA}={?\begin{array}{ccccc}|& & ?& & |\\ u_1& & ?& & u_k\\ |& & ?& & |\end{array}?}_{d\times k}$$

Step 4: Encode/ embeded any feature vector x(training or new unseen test) from d-dimension to k-dimention as follows:
$${\hat{y}}_{PCA}=W_{PCA}^T?(x?{\hat{\mu }}_x)$$
Step 5: Decode/ Reconstruct any label vector y from k-dimension to d-dimention as follows:
$${\hat{x}}_{PCA}={\hat{\mu }}_x+W_{PCA}?y$$

Dual PCA

Eigen-decomposition of d x d empirical covariance matrix Sx is computationally challenging if d >> n.

如果數據維度很大時（比如圖像，每個像素點都是一個維度, 256*256的圖片就有65536個像素），empirical covariance matrix 是d x d的矩陣，進行特征值分解會很慢，使用dual PCA就非常有必要。

與50位技術專家面對面20年技術見證，附贈技術全景圖

總結

以上是生活随笔為你收集整理的【ML】Principle Component Analysis 主成分分析的全部內容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網站內容還不錯，歡迎將生活随笔推薦給好友。