DL：深度学习算法(神经网络模型集合)概览之《THE NEURAL NETWORK ZOO》的中文解释和感悟(二)
DL：深度學習算法(神經網絡模型集合)概覽之《THE NEURAL NETWORK ZOO》的中文解釋和感悟(二)
?
?
?
目錄
AE
VAE
DAE
SAE
?
?
?
?
?
相關文章
DL：深度學習算法(神經網絡模型集合)概覽之《THE NEURAL NETWORK ZOO》的中文解釋和感悟(一)
DL：深度學習算法(神經網絡模型集合)概覽之《THE NEURAL NETWORK ZOO》的中文解釋和感悟(二)
DL：深度學習算法(神經網絡模型集合)概覽之《THE NEURAL NETWORK ZOO》的中文解釋和感悟(三)
DL：深度學習算法(神經網絡模型集合)概覽之《THE NEURAL NETWORK ZOO》的中文解釋和感悟(四)
DL：深度學習算法(神經網絡模型集合)概覽之《THE NEURAL NETWORK ZOO》的中文解釋和感悟(五)
DL：深度學習算法(神經網絡模型集合)概覽之《THE NEURAL NETWORK ZOO》的中文解釋和感悟(六)
?
AE
? ? ??Autoencoders (AE)?are somewhat similar to FFNNs as AEs are more like a different use of FFNNs than a fundamentally different architecture. The basic idea behind autoencoders is to encode information (as in compress, not encrypt) automatically, hence the name. The entire network always resembles an hourglass like shape, with smaller hidden layers than the input and output layers. AEs are also always symmetrical around the middle layer(s) (one or two depending on an even or odd amount of layers). The smallest layer(s) is|are almost always in the middle, the place where the information is most compressed (the chokepoint of the network). Everything up to the middle is called the encoding part, everything after the middle the decoding and the middle (surprise) the code. One can train them using backpropagation by feeding input and setting the error to be the difference between the input and what came out. AEs can be built symmetrically when it comes to weights as well, so the encoding weights are the same as the decoding weights.
? ? ??自動編碼器(AE)有點類似于FFNNs，因為AEs更像是FFNNs的另一種用法，而不是一種根本不同的體系結構。自動編碼器的基本思想是自動編碼信息(如壓縮，而不是加密)，因此得名。整個網絡總是像一個沙漏形狀，隱藏層比輸入層和輸出層更小。AEs也總是圍繞中間層對稱(一層或兩層取決于偶數層或奇數層)。最小的層是|，它幾乎總是在中間，這是信息最壓縮的地方(網絡的阻塞點)。直到中間的所有內容都稱為編碼部分，中間之后的所有內容都稱為解碼和(令人驚訝的)中間代碼。可以使用反向傳播訓練它們，方法是輸入并將錯誤設置為輸入和輸出之間的差。當涉及到權值時，AEs也可以對稱地構建，因此編碼權值與解碼權值相同。
Bourlard, Hervé, and Yves Kamp. “Auto-association by multilayer perceptrons and singular value decomposition.” Biological cybernetics 59.4-5 (1988): 291-294.
Original Paper PDF
?
VAE
? ? ??Variational autoencoders (VAE)?have the same architecture as AEs but are “taught” something else: an approximated probability distribution of the input samples. It’s a bit back to the roots as they are bit more closely related to BMs and RBMs. They do however rely on Bayesian mathematics regarding probabilistic inference and independence, as well as a?re-parametrisation trick to achieve this different representation. The inference and independence parts make sense intuitively,?but they rely on somewhat?complex mathematics. The basics come down to this: take influence into account. If one thing happens in one place and something else happens somewhere else, they are not necessarily related. If they are not related, then the error propagation should consider that. This is a useful approach because neural networks are large graphs (in a way), so it helps if you can rule out influence from some nodes to other nodes as you dive into deeper layers.
變分自編碼器(VAE)具有與AEs相同的體系結構，但是“被教授”了其他一些東西:輸入樣本的近似概率分布。這有點回到根源，因為它們與BMs和RBMs的關系更密切。然而，他們確實依賴貝葉斯數學的概率推理和獨立性，以及重新參數化的技巧來實現這種不同的表示。
?? ? ??推理和獨立部分在直覺上是有意義的，但是它們依賴于一些復雜的數學。最基本的是:考慮影響力。如果一件事發生在一個地方，另一件事發生在另一個地方，它們不一定相關。如果它們不相關，那么錯誤傳播應該考慮這一點。這是一種有用的方法，因為神經網絡是大型圖(在某種程度上)，所以當您深入到更深的層次時，如果能夠排除某些節點對其他節點的影響，這將會有所幫助。
Kingma, Diederik P., and Max Welling. “Auto-encoding variational bayes.” arXiv preprint arXiv:1312.6114 (2013).
Original Paper PDF
?
DAE
??????? ? ? ? Denoising autoencoders (DAE)?are AEs where we don’t feed just the input data, but we feed the input data with noise (like making an image more grainy). We compute the error the same way though, so the output of the network is compared to the original input without noise. This encourages the network not to learn details but broader features, as learning smaller features often turns out to be “wrong” due to it constantly changing with noise.
??????? ? ? ? 去噪自編碼器(DAE)是一種AEs，我們不僅向輸入數據提供數據，還向輸入數據提供噪聲(比如使圖像更顆粒化)。我們用同樣的方法計算誤差，所以網絡的輸出與原始輸入相比沒有噪聲。這鼓勵網絡不要學習細節，而是學習更廣泛的特性，因為學習較小的特性往往是“錯誤的”，因為它會隨著噪聲不斷變化。
Vincent, Pascal, et al. “Extracting and composing robust features with denoising autoencoders.” Proceedings of the 25th international conference on Machine learning. ACM, 2008.
Original Paper PDF
?
SAE
??????? ? ? ? Sparse autoencoders (SAE)?are in a way the opposite of AEs. Instead of teaching a network to represent a bunch of information in less “space” or nodes, we try to encode information in more space. So instead of the network converging in the middle and then expanding back to the input size, we blow up the middle. These types of networks can be used to extract many small features from a dataset. If one were to train a SAE the same way as an AE, you would in almost all cases end up with a pretty useless identity network (as in what comes in is what comes out, without any transformation or decomposition). To prevent?this, instead of feeding back the input, we feed back the input plus a sparsity driver. This sparsity driver can take the form of a threshold filter, where only a certain error is passed back and trained, the other error will be “irrelevant” for that pass and set to zero. In a way this resembles spiking neural networks, where not all neurons fire all the time (and points are scored for?biological plausibility).
?? ? ? 稀疏自編碼器(SAE)在某種程度上與AEs相反。我們不是教網絡在更小的“空間”或節點中表示一堆信息，而是嘗試在更大的空間中編碼信息。所以我們不是把網絡收斂到中間然后再擴展到輸入大小，而是把中間放大。這些類型的網絡可用于從數據集中提取許多小特性。如果用與AE相同的方法訓練SAE，那么幾乎在所有情況下，您都會得到一個非常無用的身份網絡(就像輸入就是輸出一樣，沒有任何轉換或分解)。為了防止這種情況發生，我們不是返回輸入，而是返回輸入加上一個稀疏驅動程序。這種稀疏驅動程序可以采用閾值篩選器的形式，在閾值篩選器中，只有一個特定的錯誤被傳遞回去并進行訓練，而另一個錯誤將與該傳遞“無關”，并被設置為零。在某種程度上，這類似于尖峰狀的神經網絡，不是所有的神經元都一直處于興奮狀態(并根據生物學上的合理性打分)。
Marc’Aurelio Ranzato, Christopher Poultney, Sumit Chopra, and Yann LeCun. “Efficient learning of sparse representations with an energy-based model.” Proceedings of NIPS. 2007.
Original Paper PDF
?
?
?
?
?
?
?
?
?
?
?
?
?
總結
以上是生活随笔為你收集整理的DL：深度学习算法(神经网络模型集合)概览之《THE NEURAL NETWORK ZOO》的中文解释和感悟(二)的全部內容，希望文章能夠幫你解決所遇到的問題。
- 上一篇： DL：深度学习算法(神经网络模型集合)概
- 下一篇： DL：深度学习算法(神经网络模型集合)概