Advance notice: when you talk about the quality of unit tests, there are several factors that come into play, such as the design, readability, maintainability and whether they can run in isolation. In this blog, I focus on the aspect of unit testing effectiveness.

Example

To illustrate mutation testing succinctly, take the following example of a Calculator class where we look at the Add method with its corresponding test method.

public static int Add(int first, int second)
{
    return first + second;
}

[DataRow(1, 0, 1)]
[DataTestMethod]
public void Add_First_And_Second_Should_Result_In_Expected(int first, int second, int expected)
{
    var result = Calculator.Add(first, second);
    result.Should().Be(expected);
}

Because the Add method only has one line, which is exercised by the unit test, the method has a code coverage of 100%.

If the + operator were replaced by the – operator in the Add method, the expected outcome would also be the 1, just like in the current test case. So with only the current test case, the test would not fail on the replaced operator, which of course it should when unit testing. To improve the quality of this test, for example a [DataRow(10, 2, 12)] test case could be added so that the wrong operator will be noticed because the unit test will start to fail.

Mutants

Similar to the example above, Stryker mutates your code base with “mutants” and performs unit tests:

  • Does at least 1 unit test fail due to a mutation? Mutant killed –> good test cases
  • Do the tests still pass? Mutant survived –> test cases are missing

This results in a total score of 0 – 100%.

Replacing the + operator with its opposite was just one example, but Stryker can perform many kinds of mutations. In addition to those on the arithmetic operators, there are also those that work on equality, logical, unary, update, bitwise and null-coalescing operators, on booleans, assignments, initializers, statements (which are simply removed from your code), LINQ statements, changes to string literals and on regular expressions. See the full list of mutators.

By the way, with configuration you can arrange which types of mutations should be excluded, or more globally through a mutation level.

How-to

It is very easy to start an analysis (with default configuration) for a given test project, go to the command prompt on the path of the test project and run:

dotnet new tool-manifest
dotnet tool install dotnet-stryker
dotnet stryker

The first command creates a tool manifest file if not already present, the second installs the Stryker.NET tool in it, and the third starts the analysis. By default, an HTML report is generated with the results (by adding -o to the last command, the generated report will open automatically):

There are many configuration options, including the already mentioned mutation level, exclusions and various optimizations. These can go in a JSON or YAML file, which can be passed to the command on run with a --config-file argument. This way, the file can be used to apply the same configuration to different test projects as well.

Reporters

Stryker has several reporters, the HTML reporter is shown above. This offers the best overview of the mutations carried out, which ones were killed or survived, and which tests are responsible for this. Other reporters are ‘markdown’, for a concise tabular overview of the analysis (useful for example in job summaries in pipelines), ‘cleartext’/‘cleartexttree’ for results directly in the terminal and ‘dots’, useful in build pipelines to see basic progress. Also it is possible to upload the results to Stryker Dashboard.

Build pipelines

To set up the analyzes in Azure DevOps, a PowerShell task is required that installs the Stryker tool, another task which starts the analysis (one for each test project) and finally an extension from the Marketplace which publishes the mutation report to an additional tab next to the build summary.

Similar in GitHub Actions, where the same commands must be issued within the steps, after which the results in markdown can be added to the job summary, or as an attachment to the job with the results in HTML by means of an upload-artifact action.

Threshold configuration makes it possible to require a minimum score for new code for a PR. If this is not achieved, then the build breaks:

Optimalisations

An analysis can take several minutes, depending on the size of your projects-under-test and test projects. There are a few options to reduce analysis duration:

  • Using the ‘since’ option, to only consider changes in your code after a certain commit or tag. Use the ’target’ option with respect to another branch such as ‘main’. You may combine this with the ‘baseline’ option for a full report.
  • Lowering the configured mutation level, of course at the expense of the number of mutants made and thus the thoroughness of the analysis.
  • Configuring exclusions of methods, classes and certain types of mutators.
  • By default, Stryker uses half the number of logical cores in the machine, you can increase this through configuration.

Conclusions

While mutation testing as a technique is quite old, it is still relatively unknown among .NET developers. Unjustified, because it offers a good insight into the effectiveness of the unit tests present. What you want is, among other things, that you want to make sure that your code functions correctly and meets the requirements. This requires complete unit tests that cover the necessary scenarios, verify calls and check results. Also, it is indispensable when refactoring existing code, as happens when improving code quality and lowering technical debt. In addition, it helps to detect any errors at an early stage.

Stryker offers a low-threshold way to handle this in an automated way for this technique and, moreover, it can be neatly integrated in your CI/CD pipelines. Even if you don’t integrate it into it and only run it on your developer machine; it trains you anyway in writing effective and complete unit tests. After all, this also contributes to being a good developer, which of course we all would like to be 😉