Paper Review: Bayesian Shrinkage towards Sharp Minimaxity

Motivation and Conclusion

Sparse normal mean model ($?～N({\sigma }^2{I}_n)$ but set ${\sigma }^2=1$):
$$y=\theta +?,?～N(0,I_n)$$

A general form of shrinkage prior:
$$\pi (\theta |\tau )=\prod _{i=1}^n\frac{1}{\tau }{\pi }_0\left(\frac{{\theta }_i}{\tau }\right),\tau ～\pi (\tau )$$

If ${\pi }_0$ is mixture of Gaussian, the general form leads to local-global shrinkage:
$${\theta }_i～N({\lambda }_i^2{\tau }^2),{\lambda }_i^2～\pi ({\lambda }_i^2),\tau ～\pi (\tau )$$

Observation: about contraction rate
Let ${\theta }^{?}$ be true parameter, $s$ be the number of nonzero entries in $\theta$, $r_n$ denote the contraction rate.

Frequentest:
$$\underset{\hat{\theta }}{\min }\underset{{\theta }^{?}}{\max }\Vert \hat{\theta }?{\theta }^{?}\Vert =\sqrt{(2+o(1))s\log \frac{n}{s}}$$
Bayesian:

Dirichlet-Laplace prior: $r_n?\sqrt{s\log \frac{n}{s}}$ when $\Vert {\theta }^{?}\Vert \le \sqrt{s}{\log }^2\frac{n}{s}$

Horseshoe prior: $r_n=M_n\sqrt{s\log \frac{n}{s}},as{M}_n\to \infty$

In general, polynomial decaying ${\pi }_0$ leads to near optimal rate

Further questions: How order of polynomial decaying ${\pi }_0$ affects contraction rate? How to choose $\tau$ to achieve (near-) optimal contraction rate given polynomial decaying ${\pi }_0$?

Contribution of this paper:

If order of ${\pi }_0$, say $\alpha \approx 1$, $r_n/\sqrt{2s\log \frac{n}{s}}\approx 1$ (Bayesian sharp minimaxity, Thm 2.1)

Choosing $\tau$ requires knowledge on $s/n$, so the author proposed a Beta modeling on $\tau$ to avoid unknown information.

Questions not been covered

How $r_n$ changes w.r.t ${\lim }_n{\alpha }_n\to 1$?

How about $\alpha =1$ (Thm 2.1 breaks down)?

Beyond contraction rate, how $\alpha$ affects model selection?

How $\alpha$ affects contraction rate in linear regression setting?

Bayesian sharp minimaxity

Import conditions on model sparsity and ${\pi }_0$

For simplicity, $\tau$ is a deterministic value and ${\theta }_i$s are mutually independent.

Remark 1: Conditions for $\tau$,

${\tau }^{\alpha ?1}\ge (s/n{)}^c\sqrt{\log (n/s)}$, for some $c\in (0,1+w/2)$. $\tau$ cannot be too small, or $\theta$ will be over-shrunk.

${\tau }^{\alpha ?1}?(s/n{)}^{\alpha }[\log (n/s){]}^{\alpha }$. $\tau$ cannot be too large, or $\theta$ will be insufficient shrunk.

${\tau }^{\alpha ?1}?(s/n{)}^{\alpha }[\log (n/s){]}^{(1+\alpha )/2}$. This is the condition for $L_1$ contraction rate.

These conditions indicate $\alpha \in (1,1+w/2)$ and $w$ should be as small as possible.

Remark 2: $E^{?}$ is the expectation under true parameter ${\theta }^{?}$. Theoretical results indicate $L_2$ contraction rate is not greater than $O(\sqrt{s\log (n/s)})$ and $L_1$ contraction rate is not greater than $O(s\sqrt{\log (n/s)})$.

Remark 3: Note that $\log (n/s)?(n/s{)}^c,?c>0$. This observation leads to Corollary 2.1 which unifies (2.1) and (2.2).

Remark 4: Corollary 2.1 indicates $\tau ?(s/n{)}^{c/(\alpha ?1)}$. Select $c=\alpha +\delta$ for very small $\delta >0$. So a good choice would be $\tau ?(s/n{)}^{(\alpha +\delta )/(\alpha ?1)}$. However, we don’t know $s$. An alternative is $\tau ?(1/n{)}^{(\alpha +\delta )/(\alpha ?1)}$. Theorem 2.2 considers the properties of this alternative.

Remark 5: Conditions for $\tau$,

${\tau }^{\alpha ?1}\ge (1/n{)}^c\sqrt{\log (n/s)}$, replace $s$ with 1

${\tau }^{\alpha ?1}?(s/n{)}^{\alpha }[\log (n/s){]}^{(1+\alpha )/2}$

Theoretical results indicate $L_2$ contraction rate is not greater than $O(\sqrt{s\log (n)})$ (sub-optimal) and $L_1$ contraction rate is not greater than $O(s\sqrt{\log (n)})$. If $\log (s)?\log (n)$, sub-optimal is asymptotically non-different from optimal. If $s?n^c,c\in (0,1)$, sub-optimal has the same order as optimal. If $\log (s)～\log (n)$, sub-optimal is of greater order.

Remark 6: Theorems above are derived based on deterministic $\tau$. Now consider $\pi (\tau )$. $\pi (\tau )$ should shrink to zero but should not shrink to zero so fast because $\pi (\tau )$ needs to assign a little density to $(s/n{)}^{(\alpha +\delta )/(\alpha ?1)}$. Theorem 3.1 provides sufficient conditions on $\tau$ to guarantee (2.1) and (2.2).

Remark 7: The prior density of $\tau$ is split into three parts: around zero, $(s/n{)}^{(1+w/2)/(\alpha ?1)}$ to $(s/n{)}^{\alpha /(\alpha ?1)}$, and greater than $(s/n{)}^{\alpha /(\alpha ?1)}$. The first part is very huge and the second part is minor. Assume the third part is decay to zero.

Remark 8: A possible choice of $\pi (\tau )$ is Beta (which may be multi-modal), i.e. $\tau ～[Beta(1,n){]}^c,c\in (\alpha /(\alpha ?1),(1+w/2)/(\alpha ?1))$.

Remark 9: Note that the restriction on ${\theta }^{?}$ is a technique assumption. Without this assumption, it’s possible to achieve sub-optimal. See Theorem 3.2.

總結

以上是生活随笔為你收集整理的Paper Review: Bayesian Shrinkage towards Sharp Minimaxity的全部內容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網站內容還不錯，歡迎將生活随笔推薦給好友。