生成模型和判別模型

Intro

介紹

Recently I gave a presentation at work, where I explained how I solved some problem using Conditional Random Fields (CRF). And, since CRF was not so much known to my colleagues, I had a couple of slides devoted to the theory behind this algorithm. As I prepared the theory slides, I felt a duty to compare CRF with a conceptually similar algorithm, Hidden Markov Model (HMM). CRF is known to be a discriminative model and HMM is a generative model. I had to refresh my knowledge about this categorisation of supervised machine learning methods, especially generative models. Now I would like to share my understanding of the difference between generative and discriminative models in simple terms.

最近，我在工作中做了一個(gè)演講，向我解釋了如何使用條件隨機(jī)場(chǎng)(CRF)解決一些問(wèn)題。 而且，由于我的同事對(duì)CRF知之甚少，所以我有幾張幻燈片專門討論了該算法背后的理論。 在準(zhǔn)備理論幻燈片時(shí)，我感到有責(zé)任將CRF與概念上類似的算法，即隱馬爾可夫模型(HMM)進(jìn)行比較 。 眾所周知，CRF是一種判別模型，而HMM是一種生成模型。 我必須刷新有關(guān)監(jiān)督式機(jī)器學(xué)習(xí)方法(尤其是生成模型)的分類的知識(shí)。 現(xiàn)在，我想簡(jiǎn)單地分享一下我對(duì)生成模型和區(qū)分模型之間差異的理解。

Generative models are a wide class of machine learning algorithms which make predictions by modelling joint distribution P(y, x).

生成模型是一類廣泛的機(jī)器學(xué)習(xí)算法，它們通過(guò)對(duì)聯(lián)合分布P(y，x)建模來(lái)進(jìn)行預(yù)測(cè)。

Discriminative models are a class of supervised machine learning models which make predictions by estimating conditional probability P(y|x).

判別模型是一類監(jiān)督的機(jī)器學(xué)習(xí)模型，通過(guò)估計(jì)條件概率P(y | x)進(jìn)行預(yù)測(cè)。

In order to use a generative model, more unknowns should be solved: one has to estimate probability of each class and probability of observation given class. These probabilities are used to compute joint probability, and finally, joint probability can be used as a substitute for conditional probability to make predictions.

為了使用生成模型，應(yīng)該解決更多的未知數(shù)：必須估計(jì)每個(gè)類別的概率和給定類別的觀察概率。 這些概率用于計(jì)算聯(lián)合概率，最后，聯(lián)合概率可以用作條件概率的替代來(lái)進(jìn)行預(yù)測(cè)。

Generative model has more steps than discriminative model in order to estimate conditional probability P(y|x)生成模型比判別模型具有更多的步驟，以便估計(jì)條件概率P(y | x)

The discriminative model takes a shorter way: it simply estimates conditional probability directly.

判別模型采用了更短的方法：它只是直接估算條件概率。

There are many pros and cons to each of the models. I just note that generative model can be used to generate new samples, but it requires more data. Discriminative model often superior than generative model, given the same amount of data, but it does not know about dependencies between features, because it is irrelevant for prediction. Therefore discriminative model can not generate new samples.

每個(gè)模型都有很多優(yōu)點(diǎn)和缺點(diǎn)。 我只是注意到生成模型可用于生成新樣本，但是它需要更多數(shù)據(jù)。 在給定相同數(shù)據(jù)量的情況下，判別模型通常優(yōu)于生成模型，但它不了解要素之間的依賴性，因?yàn)樗c預(yù)測(cè)無(wú)關(guān)。 因此，判別模型無(wú)法生成新樣本。

Now let’s take a closer look at the concept of generative models.

現(xiàn)在，讓我們仔細(xì)看看生成模型的概念。

Generative model

生成模型

As I showed earlier, to make predictions, conditional distribution P(y|x) is enough. But since P(y|x) = P(y, x) / P(x), where P(x) is constant for the given x and all possible y, it is valid to use joint distribution P(y, x) to make predictions.

如我先前所示，要進(jìn)行預(yù)測(cè)，條件分布 P(y | x) 就足夠了。 但是由于 P(y | x)= P(y，x)/ P(x) ，其中 P(x) 對(duì)于給定 x 和 所有可能的 y 都是常數(shù) ，因此使用聯(lián)合分布 P(y，x) 是有效的 做出預(yù)測(cè)。

By modelling joint distribution P(y, x) is meant that for each pair (y?, x?) a probability P(yi, xi) is known (modelled). At the beginning it was a bit difficult for me to understand how it is even possible — the range of possible values of X might be enormous, so it’s gonna be unrealistic to suggest probabilities for each xi, leave alone pair (yi, xi). How is it supposed to be done?

通過(guò)對(duì)聯(lián)合分布P(y，x)進(jìn)行建模，意味著對(duì)于每一對(duì)( y?，x?) ，已知概率P(yi，xi) (已建模)。 剛開(kāi)始時(shí)，我很難理解它的可能性-X的可能值范圍可能很大，因此建議每個(gè)xi的概率，而對(duì)( yi ， xi)單獨(dú)給出概率將是不現(xiàn)實(shí)的。 應(yīng)該怎么做？

First. Bayes theorem! It breaks computation of joint probability P(y,x) into computation of two other types of probabilities: probability of class, P(y), and probability of observation given class, P(x|y).

第一。 貝葉斯定理！ 它將聯(lián)合概率P(y，x)的計(jì)算分解為另外兩種類型的概率的計(jì)算：類概率P(y)和給定類觀察概率P(x | y)。

P(y, x) = P(y) * P(x|y)

P(y，x)= P(y)* P(x | y)

What benefits does it give? This way it is at least easier to figure out probability P(y), because it can be estimated from the dataset by computing class frequencies. P(x|y) is trickier, because usually x is not just one feature, but a set of features: x = xi, …, xn, which might have dependencies between each other.

它有什么好處？ 這樣，至少可以容易地找出概率P(y) ，因?yàn)榭梢酝ㄟ^(guò)計(jì)算類別頻率從數(shù)據(jù)集中進(jìn)行估計(jì)。 P(x | y)比較棘手，因?yàn)橥ǔ不僅是一個(gè)特征，而且是一組特征： x = xi，…，xn ，它們之間可能存在依賴關(guān)系。

P(x|y) = П P(xi|y, x1, xi-1, xi+1, xn)

P(x | y)=ПP(xi | y，x1，xi-1，xi + 1，xn)

Often the dependencies between the features are not known, especially when they appear in complex constellations (y, x1, xi-1, xi+1, xn).

通常，這些特征之間的依存關(guān)系是未知的，尤其是當(dāng)它們出現(xiàn)在復(fù)雜的星座( y，x1，xi-1，xi + 1，xn)中時(shí) 。

So what should be done to estimate P(x|y)? For this, there is the following trick:

那么，應(yīng)該怎么做才能估計(jì)P(x | y)呢？ 為此，有以下技巧：

Second. Make wild assumptions! Or just some assumptions which make estimation of P(x|y) tractable. Naive Bayes classifier can serve as a perfect example of a generative model with such assumption, which makes computation of P(x|y) easier. Namely, it has independence assumption between the features xi, …, xn.

第二。 做出瘋狂的假設(shè)！ 或者只是使P(x | y)的估算變得容易的一些假設(shè)。 在這種假設(shè)下，樸素貝葉斯分類器可以用作生成模型的完美示例，這使得P(x | y)的計(jì)算更加容易。 即，它在特征xi，…，xn之間具有獨(dú)立性假設(shè)。

P(x|y) = П P(xi|y)

P(x | y)=ПP(xi | y)

With this relaxation, estimation of P(x|y) is tractable, because every P(xi|y) can be estimated either by finding frequencies of discrete feature xi independently from other features or using Gaussian distribution, if feature xi is continuous.

通過(guò)這種放寬， P(x | y)的估計(jì)是易于處理的，因?yàn)槿绻卣鱴i是連續(xù)的，則可以通過(guò)獨(dú)立于其他特征找到離散特征xi的頻率或使用高斯分布來(lái)估計(jì)每個(gè)P(xi | y) 。

Conclusion

結(jié)論

So now you can see that in order to use generative models one should be prepared to estimate two types of probabilities P(y) and P(x|y). At the same time, discriminative models estimate conditional probability P(y|x) directly, which often is more efficient because one does not estimate dependencies between features, as these relationships don’t necessarily contribute to the prediction of the target variable.

因此，現(xiàn)在您可以看到，為了使用生成模型，應(yīng)該準(zhǔn)備一個(gè)估計(jì)兩種類型的概率P(y)和P(x | y)。 同時(shí)，判別模型直接估算條件概率P(y | x) ，這通常更為有效，因?yàn)槿藗儾还浪闾卣髦g的依賴性，因?yàn)檫@些關(guān)系不一定有助于目標(biāo)變量的預(yù)測(cè)。

翻譯自: https://medium.com/@tanyadembelova/introduction-to-generative-and-discriminative-models-9c9ef152b9af

生成模型和判別模型

總結(jié)

以上是生活随笔為你收集整理的生成模型和判别模型_生成模型和判别模型简介的全部?jī)?nèi)容，希望文章能夠幫你解決所遇到的問(wèn)題。

如果覺(jué)得生活随笔網(wǎng)站內(nèi)容還不錯(cuò)，歡迎將生活随笔推薦給好友。