變異函數 python

We need to kill the mutants — no, I’m not a villain from the X-Men comics. I’m a software engineer who wants to improve unit tests.

我們需要殺死這些突變體-不，我不是X戰警漫畫中的反派。 我是一位軟件工程師，希望改善單元測試。

In this article you will learn what mutation testing is and how it can help you to write better tests. The examples are for Python, but the concepts hold in general and in the end I have a list of tools in other languages.

在本文中，您將學習什么是突變測試以及它如何幫助您編寫更好的測試。 這些示例是針對Python的，但是這些概念是通用的，最后我有了其他語言的工具列表。

為什么我們需要進行突變測試？ (Why do we need mutation testing?)

Unit tests have the issue that it’s unclear when your tests are good enough. Do you cover the important edge cases? How do you test the quality of your unit tests?

單元測試存在一個問題，即您的測試何時足夠好尚不清楚。 您涵蓋重要的案例嗎？ 您如何測試單元測試的質量？

Typical mistakes are slight confusions. Accessing list[i] instead of list[i-1] , letting the loop run for i < n instead of i <= n , initializing a variable with None instead of the empty string. There are a lot of those slight changes which are usually just called “typos” or “off-by-one” mistakes. When I make them, I often didn’t think about the part thoroughly enough.

典型的錯誤是輕微的混亂。 訪問list[i]而不是list[i-1] ，讓循環在i < n而不是i <= n ，并使用None而不是空字符串初始化變量。 許多細微的變化通常被稱為“ 錯別字 ”或“ 一對一 ”錯誤。 當我制作它們時，我常常沒有充分地考慮零件。

Mutation testing tests your unit tests. The key idea is to apply those minor changes and run the unit tests that could fail. If a unit test fails, the mutant was killed. Which is what we want. It shows that this kind of off-by-one mistake cannot happen with our test suite. Of course, we assume that the unit tests themselves are correct or at worst incomplete. Hence you can see a mutation test as an alternative to test coverage. In contrast to test coverage, the mutation testing toolkit can directly show you places and types of mistakes you would not cover right now.

變異測試會測試您的單元測試 。 關鍵思想是應用這些較小的更改并運行可能失敗的單元測試。 如果單元測試失敗，則突變體被殺死。 這就是我們想要的。 它表明，我們的測試套件不可能發生這種一次性的錯誤。 當然，我們假設單元測試本身是正確的，或者最糟糕的是不完整。 因此，您可以看到變異測試可以替代測試覆蓋率。 與測試覆蓋率相反，變異測試工具包可以直接向您顯示您現在不會覆蓋的錯誤的位置和類型。

有哪些突變測試工具？ (Which mutation testing tools are there?)

There are a couple of tools like cosmic-ray, but Anders Hovm?ller did a pretty amazing job by creating mutmut. As of August 2020, mutmut is the best library for Python to do mutation testing.

有一對夫婦像宇宙射線的工具，但安德斯Hovm?ller通過創建mutmut做了非常了不起的工作。 截至2020年8月，mutmut是Python進行突變測試的最佳庫。

To run the examples in this article, you have to install mutmut:

要運行本文中的示例，您必須安裝mutmut ：

pip install mutmut

In other languages, you might want to try these:

使用其他語言，您可能需要嘗試以下方法：

• C / C++: mull

  C / C ++： 仔細考慮

• Java: PIT (GitHub)

  Java： PIT ( GitHub )

• JavaScript: Stryker

  JavaScript： Stryker

• PHP: Infection (formerly called humbug)

  PHP： 感染 (以前稱為humbug)

• Ruby: mutant

  Ruby： 突變體

• Rust: mutagen

  Rust： 誘變劑

• Swift: muter

  斯威夫特： 靜音

為什么分支和線路覆蓋范圍不夠？ (Why isn’t branch and line coverage enough?)

It is pretty easy to get to a high line coverage by creating bad tests. For example, take this code:

通過創建不良測試，很容易獲得較高的覆蓋率。 例如，使用以下代碼：

def fibonacci(n: int) -> int:
"""Get the n-th Fibonacci number, starting with 0 and 1."""
a, b = 0, 1
for _ in range(n):
a, b = b, a + b
return b # BUG! should be a!def test_fibonacci():
fibonacci(10)

This smoke test already adds some value as it makes sure that things are not crashing for a single input. However, it would not find any logic bug. There is an assert statement missing. This pattern can quickly drive up the line coverage up to 100%, but you are then still lacking good tests.

此煙霧測試已經增加了一些價值，因為它可以確保單個輸入不會崩潰。 但是，它不會發現任何邏輯錯誤。 缺少assert語句。 這種模式可以快速將線路覆蓋率提高到100％，但是您仍然缺乏良好的測試。

A mutation test cannot be fooled as easily. It would mutate the code and, for example, initialize b with 0 instead of 1:

突變測試不容易被愚弄。 它將使代碼突變，例如，將b初始化為0而不是1：

- a, b = 0, 1
+ a, b = 0, 0

The test would still succeed and thus the mutant would survive. Which means the mutation testing framework would complain that this line was not properly tested. In other words:

測試仍將成功，因此突變體將存活。 這意味著變異測試框架會抱怨此行未正確測試。 換一種說法：

Mutation testing provides another way to get a more rigid line coverage. It can still not guarantee that a tested line is correct, but it can show you potential bugs that your current test suite would not detect.

變異測試提供了獲得更嚴格的線覆蓋率的另一種方法。 它仍然不能保證測試的行是正確的，但是可以顯示當前測試套件無法檢測到的潛在錯誤。

創建突變體！ (Create the mutants!)

As always, I use my small mpu library as an example. At the moment, it has a 99% branch and 99% line coverage.

與往常一樣，我以我的小型mpu庫為例。 目前，它具有99％的分支和99％的線路覆蓋率。

$ mutmut run- Mutation testing starting -These are the steps:
1. A full test suite run will be made to make sure we
can run the tests successfully and we know how long
it takes (to detect infinite loops for example)
2. Mutants will be generated and checkedResults are stored in .mutmut-cache.
Print found mutants with `mutmut results`.Legend for output:
🎉 Killed mutants. The goal is for everything to end up in this bucket.
? Timeout. Test suite took 10 times as long as the baseline so were killed.
🤔 Suspicious. Tests took a long time, but not long enough to be fatal.
🙁 Survived. This means your tests needs to be expanded.
🔇 Skipped. Skipped.1. Running tests without mutations
? Running...Done2. Checking mutants
? 1818/1818 🎉 1303 ? 1 🤔 6 🙁 508 🔇 0

This takes over 1.5 hours for mpu. mpu is a small project, with only about 2000 lines of code:

對于MPU，這需要1.5個小時以上。 mpu是一個小項目，只有大約2000行代碼：

Language files blank comment code
---------------------------------------------------------------
Python 22 681 1399 2046

One pytest run of the mpu example project takes roughly 9 seconds and the slowest 3 tests are:

mppu示例項目的pytest運行大約需要9秒，最慢的3個測試是：

1.03s call tests/test_main.py::test_parallel_for
0.80s call tests/test_string.py::test_is_email
0.41s call tests/test_io.py::test_download_without_path

In the end, you will see how many mutants were successfully killed (🎉), how many received a timeout (?) and which ones survived (😕). Especially the timeout ones are annoying as they make the mutmut runs slower, but the code and the tests might still be fine.

最后，您將看到成功殺死了多少個突變體(🎉)，有多少突變體超時(?)，還有哪些幸存下來(😕)。 尤其是超時的代碼令人討厭，因為它們會使mutmut的運行速度變慢，但是代碼和測試可能仍然不錯。

應用哪些突變？ (Which mutations are applied?)

mutmut 2.0 creates the following mutants (source):

mutmut 2.0創建以下變量( 源 )：

• Operator mutations: About 30 different patterns like replacing + by - , * by ** and similar, but also > by >= .

  運算子突變 ：大約30種不同的模式，例如用+替換- ， *替換**和類似**以及>替換>= 。

• Keyword mutations: Replacing True by False , in by not in and similar.

  關鍵字突變 ：用False代替True ，用not in代替in not in并且類似。

• Number mutations: You can write things like 0b100 which is the same as 4, 0o100, which is 64, 0x100 which is 256, .12 which is 0.12 and similar. The number mutations try to capture mistakes in this area. mutmut simply adds 1 to the number.

  數突變 ：你可以寫的東西像0b100是一樣的4 ， 0o100 ，這是64個， 0x100這是256， .12是0.12和類似。 數字突變試圖捕獲該區域中的錯誤。 mutmut只是將數字加1。

• Name mutations: The name mutations capture copy vs deepcopy and "" vs None .

  名稱突變 ：名稱突變捕獲copy vs deepcopy copy和"" vs None 。

• Argument mutations: Replaces keyword arguments one by one from dict(a=b) to dict(aXXX=b).

  參數突變 ：將關鍵字參數從dict(a=b)到dict(aXXX=b) 。

• or_test and and_test: and ? or

  or_test和and_test ： and ? or

• String mutation: Adding XX to the string.

  字符串突變 ：在字符串中添加XX 。

Those can be grouped into three very different kinds of mutations: value mutations (string mutation, number mutation), decision mutations (switch if-else blocks, e.g. the or_test / and_test and the keyword mutations) and statement mutations (removing or changing a line of code).

可以將它們分為三種非常不同的突變： 值突變 (字符串突變，數字突變)， 決策突變 (切換if-else塊，例如or_test / and_test和關鍵字突變)和語句突變(刪除或更改行)代碼)。

The value mutations are most often false-positive for me. I’m not certain if I could write my code or my tests in another way to fix this. I’ve briefly discussed it with the library author, but apparently he does not have the same issue. If you’re interested in that discussion, see issue #175.

價值突變對我來說通常是假陽性。 我不確定是否可以用其他方式編寫代碼或測試來解決此問題。 我已經與庫作者進行了簡短的討論，但是顯然他沒有相同的問題。 如果您對此討論感興趣，請參閱問題＃175 。

如何獲取帶有mutmutHTML報告？ (How can I get a HTML report with mutmut?)

$ mutmut html

gives you

給你

Index page of the mutmut HTML report. Image by Martin Thoma.mutmut HTML報告的索引頁。 圖片由Martin Thoma提供。 The complete pd.py report. Image by Martin Thoma.完整的pd.py報告。 圖片由Martin Thoma提供。

As you can see, the index claims that 108 mutants survived and the HTML report only shows one. That one is also a false-positive as a change in the logging message does not cause any issue.

如您所見，該索引聲稱有108個突變體幸存下來，而HTML報告僅顯示了一個。 這也是一個假陽性，因為日志消息中的更改不會引起任何問題。

Alternatively, you can use the junit XML to generate a report:

另外，您可以使用junit XML生成報告：

$ pip install junit2html
$ mutmut junitxml > mutmut-results.xml
$ junit2html mutmut-results.xml mutmut-report.html

The report shows this index page:

該報告顯示此索引頁面：

Test report generated from JUnit XML. Image by Martin Thoma從JUnit XML生成的測試報告。 圖片由Martin Thoma提供

Clicking on one mutant, you gets this:

單擊一個突變體，您將得到：

Mutant #3 was killed, but mutant #4 survived. I did not use the global variable “countries” anywhere in the tests. Image by Martin Thoma.＃3突變體被殺死，但＃4突變體幸存下來。 我在測試中的任何地方都沒有使用全局變量“國家”。 圖片由Martin Thoma提供。

The issue with this generated HTML report is that it shows many results for a single line of code and no grouping. If the failures were grouped by file and if one could see the code in which lines with surviving mutants would be highlighted, it would be way more useful.

生成HTML報告的問題在于，它在一行代碼中顯示了許多結果，但沒有分組。 如果將失敗按文件進行分組，并且可以看到突出顯示幸存的突變體行的代碼，則將更加有用。

機器學習系統的變異測試 (Mutation Testing for Machine Learning Systems)

I’ve searched for cool applications of machine learning to generate mutants in code, but I’ve only found “Machine Learning Approach in Mutation Testing” from 2012 (12 citations).

我一直在尋找機器學習的涼爽應用，以在代碼中生成突變體，但從2012年起我才發現“突變測試中的機器學習方法”(被引用12次)。

I was hoping to find data-based code mutant generation techniques. For example, one could search for git commits which are bug fixes by examining the commit message. If the fix is rather short, this is a kind of mutation one could test for. Instead of generating all possible mutants, one could sample from the mutants in a way to first take the most promising ones; the ones that are most likely not perceived as a false-positive.

我希望找到基于數據的代碼突變體生成技術。 例如，可以通過檢查提交消息來搜索git提交，這些提交是錯誤修復程序。 如果修復程序很短，則這是一種可以測試的突變。 與其生成所有可能的突變體，不如從突變體中取樣，首先選擇最有希望的突變體。 最有可能不會被認為是假陽性的人。

Other work was more focused on making machine learning systems more robust (DeepMutation, DeepGauge, an Evaluation). I don’t know this stream of work well enough to write about it. But it sounds similar to techniques I know:

其他工作更多地集中在使機器學習系統更強大( DeepMutation ， DeepGauge ， 評估 )。 我不知道這方面的工作足夠好來寫。 但這聽起來類似于我所知道的技術：

• To overcome scarcity in training data, various data augmentation techniques such as rotations, flips, or color adjustments are applied. You can actually see those as mutations.

  為了克服訓練數據中的稀缺性，應用了各種數據 增強技術，例如旋轉，翻轉或顏色調整。 您實際上可以將其視為變異。

• Also, in the GAN setting where you have a generator and a discriminator, you could argue that the generator produces mutants and the discriminator should tell them apart.

  同樣，在GAN設置中有生成器和鑒別器的情況下，您可能會爭辯說生成器會產生突變體，并且鑒別器應將它們區分開。

• In order to force the network to learn more robust features, a technique called dropout (Tensorflow, Lasagne)is commonly used. You could say that a part of the input or the internal representation is randomly mutated by setting it to zero

  為了迫使網絡學習更強大的功能 ，通常使用一種稱為“ dropout” ( Tensorflow ， Lasagne )的技術。 您可以說，通過將輸入或內部表示的一部分設置為零可以對其進行隨機變異

想更多地了解單元測試嗎？ (Want to know more about unit testing?)

In this series, we already had:

在本系列中，我們已經有：

• Part 1: The basics of Unit Testing in Python

  第1部分： Python單元測試的基礎

• Part 2: Patching, Mocks and Dependency Injection

  第2部分： 修補，模擬和依賴注入

• Part 3: How to test Flask applications with Databases, Templates and Protected Pages

  第3部分： 如何使用數據庫，模板和受保護的頁面測試Flask應用程序

• Part 4: tox and nox

  第4部分： 有毒和無毒

• Part 5: Structuring Unit Tests

  第5部分： 結構單元測試

• Part 6: CI-Pipelines

  第6部分： CI管道

• Part 7: Property-based Testing

  第7部分： 基于屬性的測試

• Part 8: Mutation Testing
  第8部分：變異測試

In future articles, I will present:

在以后的文章中，我將介紹：

• Static Code Analysis: Linters, Type Checking, and Code Complexity
  靜態代碼分析：Linter，類型檢查和代碼復雜度

Let me know if you’re interested in other topics around testing with Python.

讓我知道您是否對使用Python測試有關的其他主題感興趣。

翻譯自: https://medium.com/analytics-vidhya/unit-testing-in-python-mutation-testing-7a70143180d8

變異函數 python

總結

以上是生活随笔為你收集整理的变异函数 python_使用Python进行变异测试的全部內容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網站內容還不錯，歡迎將生活随笔推薦給好友。