實驗人員考評指標

In the first part of my series on experimental design Thinking About Experimental Design, we covered the foundations of an experiment: the goals, the conditions, and the metrics. In this post, we will move away from the initial experimental set up to begin understanding baseline metrics and the nuances of picking appropriate conversion metrics and rates.

在我的實驗設(shè)計系列的第一部分中，“ 思考實驗設(shè)計”中 ，我們介紹了實驗的基礎(chǔ)：目標，條件和度量。 在這篇文章中，我們將脫離最初的實驗設(shè)置，開始了解基線指標以及選擇合適的轉(zhuǎn)化指標和費率的細微差別。

介紹 (Introduction)

I introduced the goals of an experiment a business context, using the example of a lemonade stand: measuring the difference in outcomes (# of cups sold) under controlled conditions (color of cup) during comparable time-frames. In the example, I failed to mention a critical element of any experiment — a hypothesis, a preliminary presumption based on limited evidence. Strictly speaking, the goal of an experiment is to validate or refute ideas put forth by a hypothesis. A reasonable hypothesis for our lemonade stand example might be that “the color of the lemonade cup affects the number of cups sold.”

我以檸檬水攤位為例，介紹了在商業(yè)環(huán)境中進行實驗的目標：在可比較的時間范圍內(nèi)，在受控條件 (杯子的顏色)下測量結(jié)果 (所售杯子的數(shù)量)的差異 。 在示例中，我沒有提到任何實驗的關(guān)鍵要素- 假設(shè) ，即基于有限證據(jù)的初步假設(shè)。 嚴格來說，實驗的目的是驗證或駁斥假設(shè)提出的想法。 我們的檸檬水攤位示例的合理假設(shè)可能是“檸檬水杯的顏色會影響所售杯子的數(shù)量”。

從基線開始 (Starting from a Baseline)

If we’re just starting our lemonade stand (or other small business/business line), can you see a problem with immediately designing this particular experiment (or any experiment) to validate a hypothesis? Market research aside, we don’t have any reliable (backed by hard-data) predictions on what to expect for our number of cups sold!

如果我們只是開始我們的檸檬水攤位(或其他小型企業(yè)/業(yè)務(wù)線)，您是否會立即設(shè)計此特定實驗(或任何實驗)以驗證假設(shè)是否存在問題？ 除了市場研究，我們對售出的杯子數(shù)量還沒有可靠的預(yù)測(有硬數(shù)據(jù)支持)！

If we have poor sales the first two weeks with red cups, followed by boosted sales in the next two weeks with blue cups, I wouldn’t be comfortable concluding that blue cups are superior. The sudden change in sales may be due to the fact that people didn’t know about our stand in its opening weeks, and only recently discovered our stand. Remember, the goal of an experiment is to understand the effects of incremental changes. When we introduce a big change (ie. starting our stand or completely revamping our storefront), we introduce instability into our existing business, which will mask and/or distort the effect of our experimental conditions.

如果我們在頭兩周使用紅色杯子的銷售情況不佳，然后在接下來的兩周使用藍色杯子的銷售量有所增長，那么我不能肯定地說藍色杯子是上乘的。 銷售量的突然變化可能是由于人們在開放的幾周內(nèi)并不了解我們的展位，而只是在最近才發(fā)現(xiàn)了我們的展位。 請記住， 實驗的目的是了解增量變化的影響。 當我們進行重大更改時(即啟動我們的展位或完全改造店面)，我們會將不穩(wěn)定因素引入現(xiàn)有業(yè)務(wù)，這將掩蓋和/或扭曲我們的實驗條件的影響。

Before conducting an experiment it’s important to establish a stable baseline that will be used the judge the effects our incremental changes.

在進行實驗之前，重要的是要建立一個穩(wěn)定的基線，該基線將用于判斷我們的增量變化的影響。

To make our example more concrete, let’s say we’ve been running our lemonade stand for 1.5 years. We’re relatively well-known in the community, but we’ve run out of ideas on how we can continue growing our business. After prioritizing our business goals and brainstorming relevant conversion metrics and rates*, we’ve decided to analyze the conversion metric:

為了使我們的示例更具體，假設(shè)我們已經(jīng)運行檸檬水攤位1.5年了。 我們在社區(qū)中相對知名，但是關(guān)于如何繼續(xù)發(fā)展業(yè)務(wù)的想法已經(jīng)耗盡。 在確定了業(yè)務(wù)目標的優(yōu)先級并集體討論了相關(guān)的轉(zhuǎn)化指標和轉(zhuǎn)化率*之后，我們決定分析轉(zhuǎn)化指標：

# of cups sold / people who stop by our stand.

售出的杯子數(shù)/站在我們展位旁的人。

I have generated some dummy data representing our monthly sales for the past year. The relative stability of our business with respect to the conversion rate. Now, when we introduce our experimental condition (changing the color of the cup), we have a reliably predictable baseline conversion rate (3.62% average) with which we can compare our new outcome.

我已經(jīng)生成了一些虛擬數(shù)據(jù)，這些數(shù)據(jù)代表了過去一年的每月銷售額。 我們的業(yè)務(wù)相對于轉(zhuǎn)換率的相對穩(wěn)定性。 現(xiàn)在，當我們介紹實驗條件(改變杯子的顏色)時，我們有了可靠的可預(yù)測的基線轉(zhuǎn)化率(平均3.62％)，可以與我們比較新的結(jié)果。

Image by Author圖片作者

考慮轉(zhuǎn)化率的變化 (Thinking about changes in our conversion rate)

At this point, it is easy to forget that our target metric is a conversion rate and begin brainstorming incremental changes that increase # of cups sold. The use of a conversion rate instead of an absolute value requires us to expand our focus from a single metric, to the relationship between two related metrics — from a focus on scale to a focus on scale & efficiency.

在這一點上，很容易忘記我們的目標指標是轉(zhuǎn)化率，并開始集思廣益，以增加銷售杯數(shù)的增量變化。 使用轉(zhuǎn)換率而非絕對值要求我們將重點從單一指標擴展到兩個相關(guān)指標之間的關(guān)系-從關(guān)注規(guī)模到關(guān)注規(guī)模和效率。

To increase our conversion rate, we must develop a strategy to increase the number of cups sold at a greater rate than we increase the number of people who stop by our stand. Altering the color or design of the cup may be an interesting business initiative; this is assuming that we believe people stop by our stand mostly for the lemonade and our beautiful stand and not due to the color of the cup.

為了提高轉(zhuǎn)化率，我們必須制定一項戰(zhàn)略，以比增加增加在我們展臺前停留的人數(shù)更多的速度增加杯子的銷售數(shù)量。 改變杯子的顏色或設(shè)計可能是一個有趣的商業(yè)嘗試； 這是假設(shè)我們相信人們主要是為了檸檬水和漂亮的立場而停下來，而不是因為杯子的顏色。

From a business perspective, we often read about using data and experiments to deliver actionable insights. In addition to our baseline rate, it is important to set reasonable target rates under each of our experimental conditions; we do not want to be taking action on any small change in conversion rates. Setting a target conversion rate is as much an art as it is a science, and can be based on a combination of past data and intuitive business sense. In our lemonade example, we might say that we will switch the color of our cups moving forward if our conversion rate is 4.12% over the next couple of months, an increase of .5%.

從業(yè)務(wù)角度來看，我們經(jīng)常閱讀有關(guān)使用數(shù)據(jù)和實驗來提供可行見解的信息。 除了我們的基準速率外，在每個實驗條件下設(shè)定合理的目標速率也很重要； 我們不希望對轉(zhuǎn)化率的任何小變化采取行動。 設(shè)定目標轉(zhuǎn)化率既是一門藝術(shù)，也是一門科學(xué)，并且可以基于過去的數(shù)據(jù)和直觀的商業(yè)意識相結(jié)合。 在我們的檸檬水示例中，我們可以說，如果未來幾個月我們的轉(zhuǎn)化率為4.12％(增加0.5％)，我們將改變杯子的顏色。

結(jié)論： (Conclusion:)

To summarize what we have accomplished in our lemonade example:

總結(jié)一下我們在檸檬水示例中所取得的成就：

We defined our business goal: increase cups sold

我們確定了我們的業(yè)務(wù)目標： 增加杯子銷量

2. We defined our conversion metric and conversion rate: cups sold & cups sold / foot traffic

2.我們定義了轉(zhuǎn)化指標和轉(zhuǎn)化率： 售出杯數(shù)和售出杯數(shù)/人流量

3. We developed a controlled incremental change that will (hypothetically) affect our outcome

3.我們開發(fā)了受控的增量更改，該更改將(假設(shè)地)影響我們的結(jié)果

4. We established a stable baseline for comparison.

4.我們建立了穩(wěn)定的基線進行比較。

If our conversion metric achieves the target goal, we can conduct business under the new conditions moving forward right? Well, not quite.

如果我們的轉(zhuǎn)化指標實現(xiàn)了目標，那么我們可以在新的條件下開展業(yè)務(wù)嗎？ 好吧，不完全是。

So far, you may have realized that I have not introduced any statistics in our experiment! At this point, it may not be entirely clear why we need statistics to validate our hypothesis and complete our experiment. Nonetheless, when dealing with such uncertainty, we want a way of quantifying our decision-making process; we use statistics quantify the strength of our experimental evidence. In the next article of this series, I will begin introducing ideas of basic statistical concepts and test statistics as they apply to our experimental design.

到目前為止，您可能已經(jīng)意識到我沒有在實驗中引入任何統(tǒng)計信息！ 在這一點上，可能還不清楚，為什么我們需要統(tǒng)計數(shù)據(jù)來驗證我們的假設(shè)并完成我們的實驗。 但是，在處理此類不確定性時，我們需要一種量化決策過程的方法。 我們使用統(tǒng)計數(shù)據(jù)來量化我們的實驗證據(jù)的強度。 在本系列的下一篇文章中，我將開始介紹適用于我們的實驗設(shè)計的基本統(tǒng)計概念和測試統(tǒng)計概念。

[1]: Monika Wahi. (2020). The Data Science of Experimental Design. LinkedIn Learning. https://www.linkedin.com/learning/the-data-science-of-experimental-design

[1]：莫妮卡·瓦希(Monika Wahi)。 (2020)。 實驗設(shè)計的數(shù)據(jù)科學(xué)。 LinkedIn學(xué)習(xí)。 https://www.linkedin.com/learning/the-data-science-of-experimental-design

[2]: Icons & Images. Pexels: https://www.pexels.com/

[2]： 圖標和圖像。 Pexels： https ：//www.pexels.com/

翻譯自: https://towardsdatascience.com/understanding-experiment-metrics-ecb0d759f743

實驗人員考評指標

總結(jié)

以上是生活随笔為你收集整理的实验人员考评指标_了解实验指标的全部內(nèi)容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網(wǎng)站內(nèi)容還不錯，歡迎將生活随笔推薦給好友。