線程監視器模型

建模基礎 (MODELING FUNDAMENTALS)

Once the model development steps are complete, model validation comes into the picture. In fact model validation is an important part of the overall model development process. If a developer is spending X amount of time in developing the model, most of the time they spend X or even more than the X amount of time in validating the model and making sure of its robustness and accuracy.

一旦模型開發步驟完成，模型驗證就會生效。 實際上，模型驗證是整個模型開發過程的重要組成部分。 如果開發人員在模型開發上花費了X倍的時間，則大多數時間他們在驗證模型并確保其魯棒性和準確性上花費X倍甚至更多的時間。

In this post, I’ll emphasize the importance of the Model Validation process and how is it different from the Model monitoring processes.

在這篇文章中，我將強調模型驗證過程的重要性以及它與模型監視過程的區別。

不正確的模型驗證的后果 (Consequences of improper Model Validation)

1.對看不見的數據的模型性能不佳 (1. Poor model performance on unseen data)

Photo by Mika Baumeister on Unsplash Mika Baumeister在Unsplash上拍攝的照片

If the model has not been validated properly then it would not be able to perform well on the unseen data which is the ultimate purpose of a predictive model. There are various model validation techniques, the most important categories would be In time validation and Out of time validation.

如果未正確驗證模型，則它將無法在看不見的數據上執行良好，這是預測模型的最終目的。 有多種模型驗證技術，最重要的類別是及時驗證和超時驗證。

A part of the development dataset is kept aside and the model is then tested on it to see how it is performing on the unseen data from the similar time segment using which it was built in the first place (i.e. In Time validation).

將開發數據集的一部分放在一邊，然后在其上測試模型，以查看其在最初構建時所使用的相似時間段中未見數據上的表現如何(即，實時驗證)。

A dataset is pulled from a different time segment and the model is tested on this unseen chunk of data too to judge its response to unseen data (i.e. Out of time validation).

從不同的時間段提取數據集，并在該看不見的數據塊上測試模型，以判斷其對看不見的數據的響應(即超時驗證)。

These validations ensure that the developers can be confident of the model performance.

這些驗證可確保開發人員對模型的性能充滿信心。

2.可疑的魯棒性 (2. Questionable robustness)

Photo by KirstenMarie on Unsplash KirstenMarie在Unsplash上拍攝的照片

As explained in the above point, if a model has been validated properly, the developers have confidence in its performance. A model which has completed the validation process is deemed fit to act robustly in the future scenarios. There are sensitivity analysis tests incorporated in the validation processes which ensure that by varying the independent model variables to a certain degree to account for economic ups and downs, the dependent variable is not affected to an extreme degree which might render the model unusable.

如上所述，如果模型已經正確驗證，則開發人員會對模型的性能充滿信心。 一個已經完成驗證過程的模型被認為適合在將來的場景中采取強有力的行動。 驗證過程中包含敏感性分析測試，這些測試可確保通過在一定程度上更改自模型變量以解決經濟起伏，因變量不會受到極大程度的影響，這可能會使模型無法使用。

3.無法適應壓力情景 (3. Failure to adapt to stress scenarios)

Estée Janssens on 雅詩蘭黛森斯上UnsplashUnsplash

In extreme scenarios like recession or this current pandemic situation, the predictive models have difficulty in adapting and predicting still-good-enough predictions despite the volatility. But if the model validation process included the stress testing measures as well then it helps in putting a version of a model into production that’s already been well tested for stress scenarios and will not suddenly fail when any calamity strikes.

在衰退或當前大流行等極端情況下，盡管波動性很大，但預測模型很難適應和預測足夠好的預測。 但是，如果模型驗證過程還包括壓力測試措施，那么它將有助于將已經針對壓力場景進行了良好測試的模型版本投入生產，并且在發生任何災難時不會突然失敗。

4.模型輸出不可靠 (4. Unreliable model outputs)

If the model has not gone through basic out of time validation tests, then it might be possible that the model is an overfitted model which performed well on the development sample but fails on unseen data. To avoid such a scenario, always validate!

如果模型沒有經過基本的超時驗證測試，則該模型可能是過度擬合的模型，在開發樣本上表現良好，但在看不見的數據上失敗。 為避免這種情況，請始終進行驗證！

模型驗證與模型監視 (Model Validation VS Model Monitoring)

To begin understanding this distinction, we’ll cover three-pointers: The purpose, the metrics, and the time period.

為了開始理解這種區別，我們將涵蓋三個指標：目的，指標和時間段。

時間段： (The Time period:)

Model validation is carried out in tandem with the model development process. Many times, it’s an iterative process. If a model fails to perform in the validation stage, it goes back to the development stage. This is immediately after the model development. It’s a one-time process.

模型驗證與模型開發過程一起進行。 很多時候，這是一個反復的過程。 如果模型在驗證階段無法執行，則可以返回到開發階段。 這是在模型開發之后立即進行的。 這是一個一次性的過程。

Model Monitoring comes into effect after a model has gone into the PROD (production) stage. It’s an ongoing process. A specific monitoring frequency is decided for every model and it’s evaluated then to make sure that the model is performing up to the mark and its results are reliable. Also, we check that the population distribution should not be significantly different as compared to the development time period to ensure that the model is still relevant and okay to be used.

模型進入PROD(生產)階段后，模型監視生效。 這是一個持續的過程。 為每個模型確定一個特定的監視頻率，然后對其進行評估，以確保模型達到最佳性能，并且結果可靠。 另外，我們檢查人口分布與開發時間段是否應沒有顯著差異，以確保該模型仍然適用并且可以使用。

目的： (The Purpose:)

The purpose of model validation is to check the accuracy and performance of the model basis on the past data for which we already have actuals.

模型驗證的目的是根據我們已經擁有的實際數據檢查模型的準確性和性能。

Once the model is deployed, the model monitoring processes ensure the relevance of the model by judging the population distribution and also recording back-dated error % comparisons between the model predictions and actuals data as soon as that starts coming in to make sure that the model performance is in the acceptable range.

部署模型后，模型監視過程將通過判斷總體分布并記錄模型預測和實際數據之間的回溯誤差百分比比較來確保模型的相關性，以確保模型正確性能在可接受的范圍內。

指標： (The Metrics:)

This is the most important distinction between the two stages. In the model validation stage, we focus majorly on the statistical metrics that can decode the model performance and response for us. Whereas in the model monitoring stage, we focus both on the statistical as well the business metrics to derive our conclusion of being confident in the relevance and the reliability of a particular model.

這是兩個階段之間最重要的區別。 在模型驗證階段，我們主要關注可以為我們解碼模型性能和響應的統計指標。 在模型監視階段，我們將重點放在統計數據和業務指標上，以得出對特定模型的相關性和可靠性充滿信心的結論。

This brings us to the end of this post. This is a question that’s often asked in the interviews as well as: can you explain the difference between the model validation and the monitoring stages. The content we covered in this post can suffice as the answer.

這使我們到本文的結尾。 這是訪談中經常問到的一個問題：您能否解釋模型驗證和監視階段之間的區別。 我們在這篇文章中介紹的內容可以作為答案。

I’ll be following up with more posts diving deep into the technical metrics/techniques used in both the model validation and model monitoring stages.

我將跟進更多文章，深入探討模型驗證和模型監視階段所使用的技術指標/技術。

Watch this space for more on Data Science, Machine Learning, and Statistics!

觀看此空間，以獲取有關數據科學，機器學習和統計的更多信息！

Happy Learning! Happy Growing!:)

學習愉快！ 快樂成長！：)

翻譯自: https://towardsdatascience.com/why-is-model-validation-so-darn-important-and-how-is-it-different-from-model-monitoring-61dc7565b0c

線程監視器模型

總結

以上是生活随笔為你收集整理的线程监视器模型_为什么模型验证如此重要，它与模型监视有何不同的全部內容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網站內容還不錯，歡迎將生活随笔推薦給好友。