What is mutation testing?
Testing has a great impact on the quality of software, especially once the product matures and the user base grows. Mutation testing helps teams assess and improve the effectiveness of the testing suite. Read on to learn what mutation testing is, how it works, and when to use it.
Table of contents
What is mutation testing?
Mutation testing, also known as code mutation testing or mutation analysis, is a form of testing in which we change specific components of an application’s source code and check whether our tests will fail as a result.
We expect them to fail, as it means that the test suite catches the bugs when you expect it to. In other words, if you change your source code and your tests don’t fail, you didn’t write good tests.
Mutation testing examples
Let’s take a look at three types of code mutation testing and some examples:
- Value mutations
- Statement mutations
- Decision mutations
Value mutation testing
We change the value of constants or parameters, for example, x=5, y=10, with x=5, y=11. Note that we run one mutation test at a time. So we insert one mutation – in our example, we’ve changed the y value from 10 to 11 – and test it. Only after that test is finished, can we run another one with a new mutation, this time changing the x value. This rule applies to all examples of mutation testing.
Statement mutation testing
We rearrange, duplicate, or delete a statement, for example:
if(a > b)
print “Congrats! You ran your personal best!”
else
print “Good job!”
with:
if(a > b)
print “Congrats! You ran your personal best!”
else
print “Try harder!”
Decision mutation testing
We change logical or arithmetic operators, which means that we replace them with alternatives, for example: and with or, >= with <=, >= with ==, etc.
Why is code mutation testing important?
A mutation test is a test for your test. Well, that may seem like an additional layer for your product development that only adds to the complexity of your delivery process, especially if you already practice test-driven development.
While test coverage is a good indicator of the state of your testing efforts, which shows you how much of your code base is covered by tests, it doesn’t assess whether your tests will catch all errors. Poorly written tests may provide high test coverage while being insufficient in terms of ensuring security. Mutation testing helps developers to:
- Assess the efficacy of your test suite.
- Identify how you can improve your tests.
- Pinpoint the time and cause of regression of your test suite.
When to kill a mutant?
The software industry loves fancy names. So when you hear people talking about killing mutants, it doesn’t necessarily mean they’re daydreamers or storytellers.
When we change a part of your program through mutation analysis, we create an alternative version of your application, called a mutant. Then the tests check it, and if your test suite rejects the mutant (which we hope it does) we say it killed the mutant.
Mutation score — how many mutants did you kill?
By calculating the mutation score, you can measure the quality of your testing environment. The mutation score in software testing is simply the number of mutants killed over the total number of mutants. It can be generated for your entire test suite, as well as individual files in your code base.
Mutation score = # of mutants killed / total # of mutants x 100%
The ideal mutation score is 100%. Achieving that for the whole code base is nearly impossible, therefore it’s recommended to keep it as high as you can. A low mutation score indicates that you should add test cases or write different assertions, in order to get rid of that issue within your test suite.
When to use mutation testing software?
The good news is you don’t need to do all mutation testing manually. A mutation testing framework automates code modifications and accelerates testing. You can choose from various tools, depending on the technology you’re using in your project.
Some of the most popular mutation testing software and libraries are: Pester for Python, Infection for PHP, Mutode for Javascript and NodeJS, and PIT for Java.
There are differences in the QA approach for product life cycle phases. Mutation analysis is a powerful technique to implement from the moment your tests start running till the end of your product’s life cycle. It will help you determine whether the testing suite is effective and can accurately detect errors or faults. In the end, mutation testing is yet another tactic that can help you prevent technical debt and build a future-proof digital product.
Share this article:
