[python skill]利用python實現(xiàn)假設(shè)性檢驗方法

刀爾東 2018-08-03 09:19:13 1244 收藏 2

版權(quán)

hello，大噶好，最近新學(xué)習(xí)了利用python實現(xiàn)假設(shè)性檢驗的一些方法，下面結(jié)合方法的數(shù)學(xué)原理做簡單的總結(jié)~

假設(shè)檢驗是推論統(tǒng)計中用于檢驗統(tǒng)計假設(shè)的一種方法。而“統(tǒng)計假設(shè)”是可通過觀察一組隨機變量的模型進(jìn)行檢驗的科學(xué)假說。[1]一旦能估計未知參數(shù)，就會希望根據(jù)結(jié)果對未知的真正參數(shù)值做出適當(dāng)?shù)耐普摗?/p>

統(tǒng)計上對參數(shù)的假設(shè)，就是對一個或多個參數(shù)的論述。而其中欲檢驗其正確性的為零假設(shè)(null hypothesis)，零假設(shè)通常由研究者決定，反應(yīng)研究者對未知參數(shù)的看法。相對于零假設(shè)的其他有關(guān)參數(shù)之論述是備擇假設(shè)(alternative hypothesis)，它通常反應(yīng)了執(zhí)行檢定的研究者對參數(shù)可能數(shù)值的另一種(對立的)看法(換句話說，備擇假設(shè)通常才是研究者最想知道的)。

假設(shè)檢驗的種類包括：t檢驗，Z檢驗，卡方檢驗，F檢驗等等。

參考：https://zh.wikipedia.org/wiki/%E5%81%87%E8%A8%AD%E6%AA%A2%E5%AE%9A

應(yīng)該說假設(shè)性檢驗是一種處理數(shù)據(jù)的思路，依據(jù)不同的實驗數(shù)據(jù)和目的可以使用不同的處理方法。如T檢驗，Z檢驗，卡方檢驗，F檢驗等等~~

--------------------------------------------------------------------------------------------------------------------

Permutation test on frog data

學(xué)習(xí)假設(shè)性檢驗之前大家不妨先復(fù)習(xí)一下之前轉(zhuǎn)載的一篇關(guān)于置換檢驗 permutation test的一篇博文，它是假設(shè)性檢驗的一種方法，比較簡單基礎(chǔ)，所以先從這種方法說起。

置換檢驗適用于兩組完整的實驗數(shù)據(jù)(n1,n2)之間的分析比較：

主要流程是：

1、合并n1，n2，無放回地抽取生成與原數(shù)據(jù)長度相等的兩組數(shù)據(jù)(n1`,n2`)

def permutation_sample(data1, data2):

"""Generate a permutation sample from two data sets."""

# Concatenate the data sets: data

data = np.concatenate((data1, data2))

# Permute the concatenated array: permuted_data

permuted_data = np.random.permutation(data)

# Split the permuted array into two: perm_sample_1, perm_sample_2

perm_sample_1 = permuted_data[:len(data1)]

perm_sample_2 = permuted_data[len(data1):]

return perm_sample_1, perm_sample_2

2、重復(fù)1過程，進(jìn)行多次實驗(如10000次)，將每次實驗得出數(shù)據(jù)(n1`,n2`)求平均，再將平均值做差：

def draw_perm_reps(data_1, data_2, func, size=1):

"""Generate multiple permutation replicates."""

# Initialize array of replicates: perm_replicates

perm_replicates = np.empty(size)

for i in range(size):

# Generate permutation sample

perm_sample_1, perm_sample_2 = permutation_sample(data_1,data_2)

# Compute the test statistic

perm_replicates[i] = func(perm_sample_1,perm_sample_2)

return perm_replicates

def diff_of_means(data_1, data_2):

"""Difference in means of two arrays."""

# The difference of means of data_1, data_2: diff

diff = np.mean(data_1)-np.mean(data_2)

return diff

3、這樣我們得到多個(10000)置換排列求得的結(jié)果，這些結(jié)果能代表模擬抽樣總體情況。

舉個栗子：

Kleinteich and Gorb (Sci. Rep.,?4, 5225, 2014) performed an interesting experiment with South American horned frogs. They held a plate connected to a force transducer, along with a bait fly, in front of them. They then measured the impact force and adhesive force of the frog's tongue when it struck the target.

K和G博士以前做過一系列實驗，研究青蛙舌頭的黏力與哪些因素有關(guān)。

他們經(jīng)過測試發(fā)現(xiàn)：FROG_A(老青蛙)的平均黏力0.71 Newtons (N)和FROG_B(小青蛙)的0.42 Newtons (N)。這0.29牛頓的差距僅僅因為測試樣本過少而偶然發(fā)生的嗎？還是由于年齡的差距確實影響了青蛙的舌頭黏力呢？

于是，兩位科學(xué)家使用置換檢驗的方法對數(shù)據(jù)進(jìn)行了分析：

# Compute difference of mean impact force from experiment: empirical_diff_means

empirical_diff_means = diff_of_means(force_a,force_b)#求得原始數(shù)據(jù)的平均值的差值

# Draw 10,000 permutation replicates: perm_replicates

perm_replicates = draw_perm_reps(force_a, force_b,

diff_of_means, size=10000)#重復(fù)一萬次實驗之后統(tǒng)計差值分布情況

# Compute p-value: p

p = np.sum(perm_replicates >= empirical_diff_means) / len(perm_replicates)#計算統(tǒng)計差值中比原始數(shù)據(jù)差值還大的可能

# Print the result

print('p-value =', p)

output：

p-value = 0.0063

可以看到，在這個假設(shè)中，我們認(rèn)為：FROG_A和FROG_B的分布是相同的(年齡并不影響舌頭的黏力)(You will compute the probability of getting at least a 0.29 N difference in mean strike force under the hypothesis that the distributions of strike forces for the two frogs are identical. )。經(jīng)過反復(fù)實驗之后，我們得到依據(jù)原始數(shù)據(jù)得出的(估計的，可能的)客觀世界均值的差值的分布情況，并求出了原始數(shù)據(jù)以及比原始數(shù)據(jù)更大(更加離譜的差值)的概率，他是0.6%，說明出現(xiàn)這種比原始數(shù)據(jù)還離譜的差值概率是很小的。所以我們只能否定原來的假設(shè)，認(rèn)為年齡是影響舌頭黏力的因素。

這就是簡單置換檢驗。

-----------------------------------------------------------------------------------------------------------------------------------

Bootstrap hypothesis tests

-----------------------------------------------------------------------------------------------------------------------------------

A one-sample bootstrap hypothesis test

下面科學(xué)家繼續(xù)對青蛙們進(jìn)行研究：

在后面的研究中，科學(xué)家們發(fā)現(xiàn)了另一組青年青蛙FROG_C(FROG_B也是年輕青蛙哦)，但不幸的是，FROG_C原始數(shù)據(jù)由于某些原因遺失，只記得它們的黏力均值為0.55N，而FROG_B的黏力均值是0.4191。因為沒有FROG_C原始數(shù)據(jù)，所以我們無法進(jìn)行置換檢驗，無法確定FROG_B和FROG_C是否服從同一種分布(是否是同一種青蛙)。它們是同一種青蛙嗎？為了進(jìn)行分析，兩個科學(xué)家絞盡腦汁，提出了一個大膽的想法：

既然不能確定分布情況，那我們假設(shè)FROG_B和FROG_C的黏力均值是一樣的好了(The mean strike force of Frog B is equal to that of Frog C.)：

Another juvenile frog was studied, Frog C, and you want to see if Frog B and Frog C have similar impact forces. Unfortunately, you do not have Frog C's impact forces available, but you know they have a mean of 0.55 N. Because you don't have the original data, you cannot do a permutation test, and you cannot assess the hypothesis that the forces from Frog B and Frog C come from the same distribution. You will therefore test another, less restrictive hypothesis: The mean strike force of Frog B is equal to that of Frog C.

To set up the bootstrap hypothesis test, you will take the mean as our test statistic.?Remember, your goal is to calculate the probability of getting a mean impact force less than or equal to what was observed for Frog B?if the hypothesis that the true mean of Frog B's impact forces is equal to that of Frog C is true. You first translate all of the data of Frog B such that the mean is 0.55 N. This involves adding the mean force of Frog C and subtracting the mean force of Frog B from each measurement of Frog B. This leaves other properties of Frog B's distribution, such as the variance, unchanged.

# Make an array of translated impact forces: translated_force_b

translated_force_b = force_b-np.mean(force_b)+0.55#改變FROG_B的黏力(為什么要改FROG_B的數(shù)據(jù)呢？？？認(rèn)為FROG_B的原始數(shù)據(jù)采集錯誤嗎？)

# Take bootstrap replicates of Frog B's translated impact forces: bs_replicates

bs_replicates = draw_bs_reps(translated_force_b, np.mean, 10000)#bootstrap reps

# Compute fraction of replicates that are less than the observed Frog B force: p

p = np.sum(bs_replicates <= np.mean(force_b)) / 10000#求p

# Print the p-value

print('p = ', p)

output:

p = ?0.0046

用人類普通話重復(fù)一下代碼語言就是：在我們做出的假設(shè)的前提下，修改了一組FROG_B的數(shù)據(jù)(因為我們沒有均值為0.55N的實驗數(shù)據(jù)，所以我們杜撰了這個？？？我不確定這里我的理解是否正確)，使之均值為0.55，記錄于translated_force_b中，我們經(jīng)過10000次bootstrap replicate實驗，我們得出了滿足假設(shè)條件的客觀世界的均值分布，最后求出比真實FORG_B數(shù)據(jù)更加極端的所有情況的概率是0.46%，可能性很小，所以否定了原來的假設(shè)，即FROG_B的黏力均值幾乎不可能和FROG_C相等。

可以看到這個實驗是針對一組有數(shù)據(jù)的樣本，和一組沒有數(shù)據(jù)的樣本(只是知道其中的某個統(tǒng)計量)，以這個沒有樣本的某個統(tǒng)計量為研究目的進(jìn)行的分析。

-----------------------------------------------------------------------------------------------------------------------------------

A bootstrap test for identical distributions

這個實驗是基于兩種樣本完整的數(shù)據(jù)，檢測他們是否具有相同的分布情況(Frog A and Frog B have identical distributions of impact forces )：

# Compute difference of mean impact force from experiment: empirical_diff_means

empirical_diff_means = diff_of_means(force_a,force_b)

# Concatenate forces: forces_concat

forces_concat = np.concatenate((force_a,force_b))

# Initialize bootstrap replicates: bs_replicates

bs_replicates = np.empty(10000)

for i in range(10000):

# Generate bootstrap sample

bs_sample = np.random.choice(forces_concat, size=len(forces_concat))

# Compute replicate

bs_replicates[i] = diff_of_means(bs_sample[:len(force_a)],

bs_sample[len(force_a):])

# Compute and print p-value: p

p = np.sum(bs_replicates>=empirical_diff_means) / len(bs_replicates)

print('p-value =', p)

output：

p-value = 0.0055

代碼過程解析：

1、首先我們認(rèn)為FROG_A和FROG_B的分布是相同的，那么我們就可以將其合并(forces_concat = np.concatenate((force_a,force_b)))；

2、利用bootstrap replicate，我們可以有放回地抽取force_a，force_b長度的兩組數(shù)據(jù)，求平均值(得到一種可能的客觀世界均值)，并對平均值做差。

3、重復(fù)2步驟10000次，我們就得到了可能的客觀世界中兩個里在假設(shè)情況下均值之差的分布情況。

4、檢驗比原始數(shù)據(jù)(empirical_diff_means = diff_of_means(force_a,force_b))更極端的情況發(fā)生的概率是多少，求出p值。

得到了概率為0.55%，很小，所以我們否定了原來的假設(shè)，即：FROG_A和FROG_B不應(yīng)該有相同的分布情況。

可以看到，與重置檢驗的方法類似，兩個檢驗方法的出發(fā)點都是：在兩個給出的樣本中，如果假設(shè)他們的分布相同，那么均值之差為0.29的情況下在是否是一個大概率事件。但兩種方法孰優(yōu)孰劣呢？datacamp中老師們給出的答案是：

Testing the hypothesis that two samples have the same distribution may be done with a bootstrap test, but a permutation test is preferred because it is more accurate (exact, in fact).

可見，重置檢驗的方法是更加值得信任的。

但重置檢驗方法也有它的局限性：

But therein lies the limit of a permutation test; it is not very versatile. We now want to test the hypothesis that Frog A and Frog B have the same mean impact force, but not necessarily the same distribution. This, too, is impossible with a permutation test.

當(dāng)我們只想比較FROG_A和FROG_B是否具有相同的均值，而不必知道他們是否有相同的分布時，重置檢驗就沒有辦法了。

-----------------------------------------------------------------------------------------------------------------------------------

A two-sample bootstrap hypothesis test for difference of means.

# Compute mean of all forces: mean_force

mean_force = np.mean(forces_concat)

# Generate shifted arrays

force_a_shifted = force_a - np.mean(force_a) + mean_force

force_b_shifted = force_b - np.mean(force_b) + mean_force

# Compute 10,000 bootstrap replicates from shifted arrays

bs_replicates_a = draw_bs_reps(force_a_shifted, np.mean, 10000)

bs_replicates_b = draw_bs_reps(force_b_shifted, np.mean, 10000)

# Get replicates of difference of means: bs_replicates

bs_replicates = bs_replicates_a - bs_replicates_b

# Compute and print p-value: p

p = np.sum(bs_replicates>=empirical_diff_means) / len(bs_replicates)

print('p-value =', p)

output：

p-value = 0.0043

可以看到，bootstrap analysis確實具有更加靈活多樣的檢驗?zāi)芰?#xff0c;不僅可以檢驗兩組數(shù)據(jù)是否具有相同的分布，而且可以檢驗數(shù)據(jù)是否具有相同的均值。

————————————————

版權(quán)聲明：本文為CSDN博主「刀爾東」的原創(chuàng)文章，遵循CC 4.0 BY-SA版權(quán)協(xié)議，轉(zhuǎn)載請附上原文出處鏈接及本聲明。

原文鏈接：https://blog.csdn.net/weixin_38760323/java/article/details/81369432

總結(jié)

以上是生活随笔為你收集整理的python怎么实现检验_[python skill]利用python实现假设性检验方法的全部內(nèi)容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網(wǎng)站內(nèi)容還不錯，歡迎將生活随笔推薦給好友。