a/b测试_如何进行A / B测试？
a/b測(cè)試
The idea of A/B testing is to present different content to different variants (user groups), gather their reactions and user behaviour and use the results to build product or marketing strategies in the future.
A / B測(cè)試的想法是將不同的內(nèi)容呈現(xiàn)給不同的變體(用戶組)，收集他們的React和用戶行為，并使用結(jié)果在將來構(gòu)建產(chǎn)品或營銷策略。
A/B testing is a methodology of comparing multiple versions of a feature, a page, a button, headline, page structure, form, landing page, navigation and pricing etc. by showing the different versions to customers or prospective customers and assessing the quality of interaction by some metric (Click-through rate, purchase, following any call to action, etc.).
A / B測(cè)試是通過向客戶或潛在客戶顯示不同版本并評(píng)估質(zhì)量來比較功能，頁面，按鈕，標(biāo)題，頁面結(jié)構(gòu)，表單，著陸頁，導(dǎo)航和定價(jià)等多個(gè)版本的方法按某種指標(biāo)(點(diǎn)擊率，購買，任何號(hào)召性用語等)進(jìn)行互動(dòng)的次數(shù)。
This is becoming increasingly important in a data-driven world where business decisions need to be backed by facts and numbers.
在數(shù)據(jù)驅(qū)動(dòng)的世界中，這一點(diǎn)變得越來越重要，在這個(gè)世界中，業(yè)務(wù)決策需要事實(shí)和數(shù)字的支持。
如何進(jìn)行標(biāo)準(zhǔn)的A / B測(cè)試 (How to conduct a standard A/B test)
1.提出你的假設(shè) (1. Formulate your hypothesis)
Before conducting an A/B testing, you want to state your null hypothesis and alternative hypothesis:
在進(jìn)行A / B測(cè)試之前，您需要陳述零假設(shè)和替代假設(shè)：
The null hypothesis is one that states that there is no difference between the control and variant group.The alternative hypothesis is one that states that there is a difference between the control and variant group.
零假設(shè) 是一個(gè)狀態(tài)存在 的控制和變體group.The 備選假設(shè) 沒有區(qū)別 是一個(gè)狀態(tài)存在 的控制和變體組之間的差。
Imagine a software company that is looking for ways to increase the number of people who pay for their software. The way that the software is currently set up, users can download and use the software free of charge, for a 7-day trial. The company wants to change the layout of the homepage to emphasise with a red logo instead of blue logo that there is a 7-day trial available for the company’s software.
想象一下，一家軟件公司正在尋找增加軟件購買費(fèi)用的人數(shù)的方法。 用戶可以免費(fèi)下載和使用該軟件的當(dāng)前設(shè)置方式，試用期為7天。 該公司希望更改首頁的布局，以紅色徽標(biāo)代替藍(lán)色徽標(biāo)來強(qiáng)調(diào)該公司的軟件有7天的試用期。
Here is an example of hypothesis test: Default action: Approve blue logo.Alternative action: Approve red logo.Null hypothesis: Blue logo does not cause at least 10% more license purchase than red logo.Alternative hypothesis: Red logo does cause at least 10% more license purchase than blue logo.
以下是假設(shè)檢驗(yàn)的示例： 默認(rèn)操作：批準(zhǔn)藍(lán)色徽標(biāo)。 替代措施：批準(zhǔn)紅色徽標(biāo)。 無假設(shè)：藍(lán)色徽標(biāo)不會(huì)導(dǎo)致購買的許可證比紅色徽標(biāo)多至少10％。 替代假設(shè)：紅色徽標(biāo)確實(shí)導(dǎo)致購買的許可證比藍(lán)色徽標(biāo)多至少10％。
It’s important to note that all other variables need to be held constant when performing an A/B test.
重要的是要注意，在執(zhí)行A / B測(cè)試時(shí)，所有其他變量都必須保持恒定。
2.確定劃分和評(píng)估指標(biāo) (2. Deciding on Splitting and Evaluation Metrics)
We should consider two things: where and how we should split users into experiment groups when entering the website, and what metrics we will use to track the success or failure of the experimental manipulation. The choice of unit of diversion (the point at which we divide observations into groups) may affect what evaluation metrics we can use.
我們應(yīng)該考慮兩件事：進(jìn)入網(wǎng)站時(shí)應(yīng)在何處以及如何將用戶分為實(shí)驗(yàn)組，以及我們將使用什么指標(biāo)來跟蹤實(shí)驗(yàn)操作的成功或失敗。 轉(zhuǎn)移單位的選擇(將觀察分為幾組的點(diǎn))可能會(huì)影響我們可以使用的評(píng)估指標(biāo)。
The control, or ‘A’ group, will see the old homepage, while the experimental, or ‘B’ group, will see the new homepage that emphasises the 7-day trial.
對(duì)照組(即“ A”組)將看到舊的主頁，而實(shí)驗(yàn)組(即“ B”組)將看到強(qiáng)調(diào)7天試用期的新主頁。
Three different splitting metric techniques:
三種不同的拆分指標(biāo)技術(shù)：
a) Event-based diversionb) Cookie-based diversion c) Account-based diversion
a)基于事件的轉(zhuǎn)移b)基于Cookie的轉(zhuǎn)移c)基于帳戶的轉(zhuǎn)移
An event-based diversion (like a pageview) can provide many observations to draw conclusions from, but if the condition changes on each pageview, then a visitor might get a different experience on each homepage visit. Event-based diversion is much better when the changes aren’t as easily visible to users, to avoid disruption of experience.
基于事件的轉(zhuǎn)移 (如綜合瀏覽量)可以提供許多觀察結(jié)果，以得出結(jié)論，但是如果條件在每個(gè)綜合瀏覽量上都發(fā)生變化，那么訪問者可能會(huì)在每次首頁訪問中獲得不同的體驗(yàn)。 當(dāng)更改對(duì)用戶而言不那么容易看到時(shí)，基于事件的轉(zhuǎn)移要好得多，這樣可以避免體驗(yàn)中斷。
In addition, event-based diversion would let us know how many times the download page was accessed from each condition, but can’t go any further in tracking how many actual downloads were generated from each condition.
此外，基于事件的轉(zhuǎn)移將使我們知道從每個(gè)條件訪問了多少次下載頁面，但無法進(jìn)一步跟蹤從每個(gè)條件產(chǎn)生了多少實(shí)際下載。
Account-based can be stable, but is not suitable in this case. Since visitors only register after getting to the download page, this is too late to introduce the new homepage to people who should be assigned to the experimental condition.
基于帳戶的帳戶可以穩(wěn)定，但在這種情況下不適合。 由于訪問者僅在進(jìn)入下載頁面后進(jìn)行注冊(cè)，因此將新首頁介紹給應(yīng)該分配到實(shí)驗(yàn)條件的人們?yōu)闀r(shí)已晚。
So this leaves the consideration of cookie-based diversion, which feels like the right choice. Cookies also allow tracking of each visitor hitting each page. The downside of cookie based diversion, is that it get some inconsistency in counts if users enter the site via incognito window, different browsers, or cookies that expire or get deleted before they make a download. As a simplification, however, we’ll assume that this kind of assignment dilution will be small, and ignore its potential effects.
因此，這無需考慮基于cookie的轉(zhuǎn)移 ，這似乎是正確的選擇。 Cookies還可以跟蹤每個(gè)訪問者訪問每個(gè)頁面的情況。 基于cookie的轉(zhuǎn)移的缺點(diǎn)是，如果用戶通過隱身窗口，不同的瀏覽器或過期或在下載前被刪除的cookie進(jìn)入站點(diǎn)，則計(jì)數(shù)會(huì)出現(xiàn)一些不一致的情況。 但是，為簡化起見，我們將假定這種分配稀釋很小，并忽略其潛在影響。
In terms of evaluation metrics, we should prefer using the download rate (# downloads / # cookies) and purchase rate (# licenses / # cookies) relative to the number of cookies as evaluation metrics.
在 評(píng)估指標(biāo)方面 ，相對(duì)于Cookie數(shù)量，我們應(yīng)該更喜歡使用下載率 (＃次下載/＃cookie)和購買率 (＃個(gè)許可/＃cookies)作為評(píng)估指標(biāo)。
Product usage statistics like the average time the software was used in the trial period are potentially interesting features, but aren’t directly related to our experiment. Certainly, these statistics might help us dig deeper into the reasons for observed effects after an experiment is complete. But in terms of experiment success, product usage shouldn’t be considered as an evaluation metric.
產(chǎn)品使用情況統(tǒng)計(jì)信息(例如軟件在試用期內(nèi)的平均使用時(shí)間)可能是有趣的功能，但與我們的實(shí)驗(yàn)沒有直接關(guān)系。 當(dāng)然，這些統(tǒng)計(jì)信息可能有助于我們?cè)趯?shí)驗(yàn)完成后更深入地觀察觀察到的效果的原因。 但就實(shí)驗(yàn)成功而言，不應(yīng)將產(chǎn)品使用情況視為評(píng)估指標(biāo)。
3.創(chuàng)建您的對(duì)照組和測(cè)試組 (3. Create your control group and test group)
Once you determine your null and alternative hypothesis, the next step is to create your control and test (variant) group. There are two important concepts to consider in this step, sampling and sample size.
一旦確定了零假設(shè)和替代假設(shè)，下一步就是創(chuàng)建對(duì)照和測(cè)試(變量)組。 在此步驟中，有兩個(gè)重要概念需要考慮，即采樣和樣本量。
SamplingRandom sampling is one most common sampling techniques. Each sample in a population has an equal chance of being chosen. Random sampling is important in hypothesis testing because it eliminates sampling bias, and it’s important to eliminate bias because you want the results of your A/B test to be representative of the entire population rather than the sample itself.
采樣隨機(jī)采樣是一種最常見的采樣技術(shù)。 總體中的每個(gè)樣本都有相等的機(jī)會(huì)被選中。 隨機(jī)抽樣在假設(shè)檢驗(yàn)中很重要，因?yàn)樗顺闃悠?#xff0c;而消除偏差也很重要，因?yàn)槟Ｍ鸄 / B檢驗(yàn)的結(jié)果能夠代表整個(gè)總體而不是樣本本身。
A problem of A/B tests is that if you haven’t defined your target group properly or you’re in the early stages of your product, you may not know a lot about your customers. If you’re not sure who they are (try creating some user personas to get started!) then you might end up with misleading results. Important to understand which sampling method that suits your use case.
A / B測(cè)試的問題是，如果您沒有正確定義目標(biāo)組，或者您處于產(chǎn)品的早期階段，那么您可能對(duì)客戶了解的不多。 如果不確定他們是誰(嘗試創(chuàng)建一些用戶角色來開始！)，那么最終可能會(huì)產(chǎn)生誤導(dǎo)性的結(jié)果。 重要的是要了解哪種采樣方法適合您的用例。
Sample SizeIt’s essential that you determine the minimum sample size for your A/B test prior to conducting it so that you can eliminate under coverage bias, bias from sampling too few observations.
樣本大小 ，你先確定你的A / B測(cè)試的最小樣本量進(jìn)行，這樣你可以在覆蓋偏倚 ，從取樣太少觀察偏見消除它是必不可少的。
4. A / B測(cè)試的時(shí)間 (4. Length of the A/B test)
A calculator like this one can help you determine the length of time you need to get any real significance from your A/B tests.
像這樣的計(jì)算器可以幫助您確定從A / B測(cè)試中獲得任何實(shí)際意義所需的時(shí)間。
History data shows that there are about 3250 unique visitors per day. There are about 520 software downloads per day (a .16 rate) and about 65 licenses purchased each day (a .02 rate). In an ideal case, both the download rate and license purchase rate should increase with the new homepage; a statistically significant negative change should be a sign to not deploy the homepage change. However, if only one of our metrics shows a statistically significant positive change we should be happy enough to deploy the new homepage
歷史數(shù)據(jù)顯示，每天大約有3250位唯一身份訪問者。 每天大約有520個(gè)軟件下載( .16比率 )，每天購買約65個(gè)許可證( .02比率 )。 在理想情況下，下載率和許可證購買率均應(yīng)隨新首頁的增加而增加； 具有統(tǒng)計(jì)意義的負(fù)面變化應(yīng)該是不部署主頁更改的標(biāo)志。 但是，如果只有一項(xiàng)指標(biāo)顯示出統(tǒng)計(jì)上顯著的積極變化，那么我們應(yīng)該很樂意部署新的首頁
For an overall 5% Type I error rate with Bonferroni correction and 80% power, we should require 6 days to reliably detect a 50 download increase per day and 21 days to detect an increase of 10 license purchases per day. Performing both individual tests at a .05 error rate carries the risk of making too many Type I errors. As such, we’ll apply the Bonferroni correction to run each test at a .025 error rate so as to protect against making too many errors.
對(duì)于具有Bonferroni校正和80％功率的5％I型錯(cuò)誤率，我們應(yīng)該需要6天才能可靠地檢測(cè)到每天50個(gè)下載量的增加，而需要21天才能檢測(cè)到每天10個(gè)許可證購買量的增加。 以.05的錯(cuò)誤率執(zhí)行兩項(xiàng)測(cè)試都可能導(dǎo)致I型錯(cuò)誤過多。 因此，我們將應(yīng)用Bonferroni校正以.025的錯(cuò)誤率運(yùn)行每個(gè)測(cè)試，以防止發(fā)生太多錯(cuò)誤。
Use the link above for the test days calculations: Estimated existing conversion rate (%): 16% Minimum improvement in conversion rate you want to detect (%): 50/520*100 %Number of variations/combinations (including control): 2Average number of daily visitors: 3250Percent visitors included in test? 100%Total number of days to run the test: 6 days
使用上面的鏈接進(jìn)行測(cè)試日計(jì)算： 估計(jì)現(xiàn)有轉(zhuǎn)化率(％)： 16％ 您要檢測(cè)的轉(zhuǎn)化率的最小改進(jìn)(％)： 50/520 * 100％ 變體/組合數(shù)(包括對(duì)照)： 2 每日平均訪客人數(shù)： 3250 測(cè)試中是否包含訪客？ 100％ 運(yùn)行測(cè)試的總天數(shù)： 6 天
Estimated existing conversion rate (%): 2 % Minimum improvement in conversion rate you want to detect (%): 10/65*100 %Number of variations/combinations (including control): 2Average number of daily visitors: 3250Percent visitors included in test? 100%Total number of days to run the test: 21 days
估計(jì)的現(xiàn)有轉(zhuǎn)化率(％)： 2％ 您要檢測(cè)的轉(zhuǎn)化率的最低改進(jìn)(％)： 10/65 * 100％ 變體/組合數(shù)(包括對(duì)照)： 2 平均每日訪問者數(shù)量： 3250 包含的 訪問者 百分比在測(cè)試中？ 100％ 運(yùn)行測(cè)試的總天數(shù)： 21天
One thing that isn’t accounted for in the base experiment length calculations is that there is going to be a delay between when users download the software and when they actually purchase a license. That is, when we start the experiment, there could be about seven days before a user account associated with a cookie actually comes back to make their purchase. Any purchases observed within the first week might not be attributable to either experimental condition. As a way of accounting for this, we’ll run the experiment for about one week longer to allow those users who come in during the third week a chance to come back and be counted in the license purchases tally.
在基礎(chǔ)實(shí)驗(yàn)時(shí)長計(jì)算中未考慮的一件事是，用戶下載軟件的時(shí)間與實(shí)際購買許可證之間將有一個(gè)延遲。 也就是說，當(dāng)我們開始實(shí)驗(yàn)時(shí)，可能需要大約7天的時(shí)間，與Cookie相關(guān)聯(lián)的用戶帳戶才能真正恢復(fù)購買。 在第一周內(nèi)觀察到的任何購買都可能與實(shí)驗(yàn)條件無關(guān)。 為了說明這一點(diǎn)，我們將實(shí)驗(yàn)進(jìn)行大約一周的時(shí)間，以使在第三周內(nèi)進(jìn)入的用戶有機(jī)會(huì)回來并計(jì)入許可證購買計(jì)數(shù)。
As for biases, we don’t expect users to come back to the homepage regularly. Downloading and license purchasing are actions we expect to only occur once per user, so there’s no real ‘return rate’ to worry about. One possibility, however, is that if more people download the software under the new homepage, the expanded user base is qualitatively different from the people who came to the page under the original homepage. This might cause more homepage hits from people looking for the support pages on the site, causing the number of unique cookies under each condition to differ. If we do see something wrong or out of place in the invariant metric (number of cookies), then this might be an area to explore in further investigations.
至于偏見，我們不希望用戶定期返回首頁。 下載和購買許可證是我們希望每個(gè)用戶僅執(zhí)行一次的操作，因此無需擔(dān)心真正的“回報(bào)率”。 但是，一種可能性是，如果有更多的人在新首頁下下載該軟件，則擴(kuò)展的用戶基礎(chǔ)在質(zhì)量上將不同于訪問原始首頁下的頁面的人。 這可能會(huì)導(dǎo)致人們?cè)诰W(wǎng)站上尋找支持頁面的點(diǎn)擊量增加，從而導(dǎo)致每種情況下唯一Cookie的數(shù)量有所不同。 如果我們?cè)诠潭ㄖ笜?biāo)(Cookie的數(shù)量)中確實(shí)發(fā)現(xiàn)了錯(cuò)誤或不正確的地方，那么這可能是需要進(jìn)一步研究的領(lǐng)域。
5.進(jìn)行測(cè)試 (5. Conduct the test)
Once you conduct your experiment and collect your data, you want to determine if the difference between your control group and variant group is statistically significant. There are a few steps in determining this:
完成實(shí)驗(yàn)并收集數(shù)據(jù)后，您要確定對(duì)照組和變異組之間的差異是否在統(tǒng)計(jì)上顯著。 確定此步驟有幾個(gè)步驟：
First, you want to set your alpha, the probability of making a type 1 error. Typically the alpha is set at 5% or 0.05
首先，您要設(shè)置alpha ，即發(fā)生1型錯(cuò)誤的概率。 通常將alpha設(shè)置為5％或0.05
- Second, you want to determine the probability value (p-value) by first calculating the t-statistic using the formula above or using z-score. 其次，您想通過首先使用上述公式或使用z分?jǐn)?shù)計(jì)算t統(tǒng)計(jì)量來確定概率值(p值)。
- Lastly, compare the p-value to the alpha. If the p-value is greater than the alpha, do not reject the null! 最后，將p值與alpha進(jìn)行比較。 如果p值大于alpha，請(qǐng)不要拒絕null！
5.1使用實(shí)際統(tǒng)計(jì)數(shù)據(jù)比較結(jié)果 (5.1 Use actual statistics to compare the results)
Do not rely on simple 1 on 1 comparison metrics to dictate what works and does not work. “Version A yields a 20 percent conversion rate and Version B yields a 22 percent conversion rate, therefore we should switch to Version B!” Please do not do this. Use actual confidence intervals, z-scores, and statistically significant data.
不要依靠簡單的一對(duì)一比較指標(biāo)來確定哪些有效，哪些無效。 “ 版本A產(chǎn)生20％的轉(zhuǎn)換率， 版本B產(chǎn)生22％的轉(zhuǎn)換率，因此我們應(yīng)該切換到版本B！” 請(qǐng)不要這樣做。 使用實(shí)際的置信區(qū)間，z得分和具有統(tǒng)計(jì)意義的數(shù)據(jù)。
5.2產(chǎn)品增長 (5.2 Product Growth)
Changing colours and layout may have a marginal impact on your key performance metrics. However, these results seem to be very short-lived. Product growth does not result from changing a button from red to blue, it comes from building a product that people want to use.
更改顏色和布局可能會(huì)對(duì)關(guān)鍵績效指標(biāo)產(chǎn)生輕微影響。 但是，這些結(jié)果似乎是短暫的。 產(chǎn)品的增長并非來自將按鈕從紅色更改為藍(lán)色的結(jié)果，而是來自構(gòu)建人們想要使用的產(chǎn)品。
Instead of choosing feature that you think might work, you can use an A/B test to know what works.
您可以使用A / B測(cè)試來了解有效的方法，而不是選擇您認(rèn)為可能有效的功能。
5.3分析數(shù)據(jù) (5.3 Analyse Data)
For the first evaluation metric, download rate, there was an extremely convincing effect. An absolute increase from 0.1612 to 0.1805 results in a z-score of 7.87 (z-score = 0.1805–0.1612/0.0025) and p-value < .00001, well beyond any standard significance bound. However, the second evaluation metric, license purchasing rate, only shows a small increase from 0.0210 to 0.0213 (following the assumption that only the first 21 days of cookies account for all purchases). This results in a p-value of 0.398 (z = 0.26).
對(duì)于第一個(gè)評(píng)估指標(biāo)，下載率，具有令人信服的效果。 從0.1612到0.1805的絕對(duì)增加會(huì)導(dǎo)致z得分為7.87(z得分= 0.1805–0.1612 / 0.0025)，p值<.00001，遠(yuǎn)遠(yuǎn)超出了任何標(biāo)準(zhǔn)顯著性范圍。 但是，第二個(gè)評(píng)估指標(biāo)，即許可證購買率，僅顯示從0.0210到0.0213的小幅增長(假設(shè)所有購買的數(shù)據(jù)僅占cookie的前21天)。 這導(dǎo)致p值為0.398(z = 0.26)。
6.得出結(jié)論 (6. Draw Conclusions)
Despite the fact that statistical significance wasn’t obtained for the number of licenses purchased, the new homepage appeared to have a strong effect on the number of downloads made. Based on our goals, this seems enough to suggest replacing the old homepage with the new homepage. Establishing whether there was a significant increase in the number of license purchases, either through the rate or the increase in the number of homepage visits, will need to wait for further experiments or data collection.
盡管沒有獲得購買許可證數(shù)量的統(tǒng)計(jì)意義，但新主頁似乎對(duì)下載的數(shù)量產(chǎn)生了很大影響。 根據(jù)我們的目標(biāo)，這似乎足以建議用新主頁替換舊主頁。 要確定購買許可證的數(shù)量是否顯著增加(無論是通過訪問率還是通過首頁訪問的數(shù)量增加)，都需要等待進(jìn)一步的實(shí)驗(yàn)或數(shù)據(jù)收集。
One inference we might like to make is that the new homepage attracted new users who would not normally try out the program, but that these new users didn’t convert to purchases at the same rate as the existing user base. This is a nice story to tell, but we can’t actually say that with the data as given. In order to make this inference, we would need more detailed information about individual visitors that isn’t available. However, if the software did have the capability of reporting usage statistics, that might be a way of seeing if certain profiles are more likely to purchase a license. This might then open additional ideas for improving revenue.
我們可能要做出的一個(gè)推斷是，新首頁吸引了通常不會(huì)試用該程序的新用戶，但是這些新用戶沒有以與現(xiàn)有用戶群相同的速度轉(zhuǎn)換為購買商品。 這是一個(gè)很好的故事，但是我們不能用給定的數(shù)據(jù)這么說。 為了進(jìn)行推斷，我們將需要有關(guān)不可用的單個(gè)訪客的更多詳細(xì)信息。 但是，如果該軟件確實(shí)具有報(bào)告使用情況統(tǒng)計(jì)信息的功能，則可能是查看某些配置文件是否更有可能購買許可證的一種方式。 然后，這可能會(huì)打開其他想法來提高收入。
翻譯自: https://towardsdatascience.com/how-to-conduct-a-b-testing-3076074a8458
a/b測(cè)試
總結(jié)
以上是生活随笔為你收集整理的a/b测试_如何进行A / B测试？的全部內(nèi)容，希望文章能夠幫你解決所遇到的問題。
- 上一篇： 做梦梦到别人买房子是什么意思
- 下一篇： 面向数据科学家的实用统计学_数据科学家必