Paper Review: Bayesian Regularization and Prediction

One-group Answers to Two-group questions

Two-group questions: I think this means two alternatives of ${\beta }_i=0$ or ${\beta }_i=0$.
Two-group answers: decrete mixture priors of ${\beta }_i$

Multiple Testing

$$y|\beta ～N(\beta ,{\sigma }^{2}I),{\beta }_p=({\beta }_1,?,{\beta }_p{)}^{\prime }$$

Decrete mixture priors of ${\beta }_i$:
$${\beta }_i～wg({\beta }_i)+(1?w){\delta }_0$$

Marginal density of $y$ under $\beta =0$ and $\beta =0$,
$$f_0(y)=N(y|0,{\sigma }^2),f_1(y)=\int N(y|\beta ,{\sigma }^2)g(\beta )d\beta$$

Interpret $w(y)$ as posterior probability like $P(\beta =0)$ ($y$ is a signal):
$$w(y)=P(\beta =0|y)=\frac{P(y,\beta =0)}{P(y)}=\frac{w{f}_1(y)}{w{f}_1(y)+(1?w)f_0(y)}$$

Sparse Regression

Note that

Ridge regression: $l_2$-penalty

LASSO: $l_1$-penalty

Cauchy prior: ${\beta }_i{～}_{iid}C(0,{\sigma }_i),i=1,?,p$

Horseshoe prior (Handling Sparsity via the Horseshoe):

Loss function of penalized regression:
$$l(\beta )={\Vert y?X\beta \Vert }_2^2+\nu \sum _{i=1}^p\psi ({\beta }_i^2)$$

Equivalent to $Y～N(X\beta ,{\sigma }^{2}I)$ with prior $\pi ({\beta }_i|\nu )\propto \exp \left(\nu \psi ({\beta }_i^2)\right)$. See posterior probability of $\beta$:
$$\pi (\beta |y,{\sigma }^2,\nu )\propto \frac{1}{\sigma }\exp \left(?\frac{{\Vert y?X\beta \Vert }_2^2}{2{\sigma }^2}+\nu \sum _{i=1}^p\psi ({\beta }_i^2)\right)$$

So maximization of loss function is equivalent to MAP pf posterior probability.

Global-local Shrinkage

Framework

Consider $Y～N(X\beta ,{\sigma }^{2}I)$ with priors
$$\begin{gathered} {\beta }_i|{\tau }^2,{\lambda }_i^2～N(0,{\tau }^2{\lambda }_i^2) \\ {\lambda }_i^2～\pi ({\lambda }_i^2) \\ ({\tau }^2,{\sigma }^2)～\pi ({\tau }^2,{\sigma }^2) \end{gathered}$$

Joint priors:
$$\pi (\beta ,\Lambda ,{\tau }^2,{\sigma }^2)=\prod _{i=1}^{p}N(0,{\tau }^2{\lambda }_i^2)\pi ({\lambda }_i^2)\pi ({\tau }^2,{\sigma }^2)$$

Question: why (3)?

Transformation to orthogonal scheme under $U$: $Z=XU,Z^{\prime }Z=D,\alpha =U^{\prime }\beta$ and set $\alpha |\Lambda ,{\tau }^2,{\sigma }^2～N(0,{\sigma }^2{\tau }^{2}n{D}^{?1}\Lambda )$ so $\beta |\Lambda ,{\tau }^2,{\sigma }^2～N(0,{\sigma }^2{\tau }^{2}nU{D}^{?1}\Lambda U^{\prime })$.

Question: how to understand this?

Question: how to get this?

To squelch the noise and shrink coefficients

$small\tau$: $\pi ({\tau }^2)$ concentration on zero

${\lambda }_i^{2}large$: $\tau ({\lambda }_i^2)$ heavy tail

Properties: why good performance?

Robust tail

Question: how to understand $\eta$?

Efficiency

Global Variance Component

Never choose $\pi ({\tau }^2,{\sigma }^2)$ that forces ${\sigma }^2$ away from zero.

Numerical Examples

Regularized Regression

Wavelet denoising

總結

以上是生活随笔為你收集整理的Paper Review: Bayesian Regularization and Prediction的全部內容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網站內容還不錯，歡迎將生活随笔推薦給好友。