Adadelta論文原文是:
《Adadelta:An adaptive learning rate method》

論文的重點是Section3，我們重點對Section3進(jìn)行解讀

section 3.Adadelta Method

the continual decay of learning rates throughout training, and 2) the need for a manually selected
global learning rate.
意思是Adadelta是為了:
1.學(xué)習(xí)率衰退問題,2.學(xué)習(xí)率自動選擇的問題

In the ADAGRAD method the denominator accumulates the squared gradients from each iteration starting at the beginning of training. Since each term is positive, this accumulated sum continues to grow throughout training, effectively shrinking the learning rate on each dimension. After many iterations, this learning rate will become infinitesimally small.
這段話的意思是ADAGRAD會隨著訓(xùn)練的進(jìn)行，導(dǎo)致學(xué)習(xí)率逐漸變成０．

3.1.Idea1:Accumulate Over Window
Instead of accumulating the sum of squared gradients over all time, we restricted the window of past gradients that are accumulated to be some fixed size $w$ (instead of size $t$ where $t$ is the current iteration as in ADAGRAD). With this windowed accumulation the denominator of ADAGRAD cannot accumulate to infinity and instead becomes a local estimate using recent gradients. This ensures that learning continues to make progress even after many iterations of updates have been done.
意思是用一個窗口w,而不是像adagrad那樣累積之前t輪所有的權(quán)重.
$$E[g^2{]}_t=\rho E[g^2{]}_{t?1}+(1?\rho )g_t^2(8)$$
$$RMS[g{]}_t=\sqrt{E[g^2{]}_t+?}(9)$$
$$△x_t=?\frac{\eta }{RMS[g{]}_t}{g}_t(10)$$

上面的式子中，(8)代入(9),(9)代入(10)，即為最終偽代碼的一部分
然后，因為式子中$\eta$是需要手工設(shè)定的，所以下面有了3.2

3.2.Idea2:Correct Units with Hessian Approximation
二階牛頓法可以寫成：
$$x_{t+1}=x_t?\frac{f^{\prime }(x)}{f^{\prime \prime }(x)}$$
所以二階牛頓法中，我們可以把$\frac{1}{f^{\prime \prime }(x)}$視為學(xué)習(xí)率。

在二階牛頓法中，有:
$$△x=\frac{\frac{?f}{?x}}{\frac{{?}^{2}f}{?x^2}}$$
可以推導(dǎo)出：
$$\frac{1}{\frac{{?}^{2}f}{?x^2}}=\frac{△x}{\frac{?f}{?x}}$$（這個步驟我認(rèn)為沒啥用，就是在論文里面湊字?jǐn)?shù)逼叨幾句）

Since the RMS of the previous gradients is already represented in the denominator in Eqn. 10 we considered a measure of the $△x$ quantity in the numerator.
這里的意思是已經(jīng)把式子(10)的分母處理完了(這是廢話，這里是為了增加字?jǐn)?shù))

$△x_t$ for the current time step is not known, so we assume the curvature is locally smooth and approximate $△x_t$ by compute the exponentially decaying RMS over a window of size w of previous $△x$　to give the ADADELTA method.
這段話什么意思呢？
意思是說:
我們同樣對$△x$使用一個窗口來計算合理的值，講人話就是：我們腦袋一拍，覺得這里就用均方根吧。
然后就有了分子中中的$RMS[△x{]}_{t?1}$

最終算法如下：

Note:
算法中的第4步和第6步代入第5步，然后第5步代入第7步，這樣就算完成了一次更新迭代

總結(jié)

以上是生活随笔為你收集整理的Adadelta原文解读的全部內(nèi)容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網(wǎng)站內(nèi)容還不錯，歡迎將生活随笔推薦給好友。