Using Major to perform mutation analysis of Java programs

25 October 2014

Updated: 12 October 2016

creative commons licensed ( BY-NC-SA ) flickr photo shared by Rami™

If you have a JUnit test suite, it is important that you ask the question "how good are my test cases?" If you have a low quality test suite, then you may not be doing a good job at establishing a confidence in the correctness of the program under test—and there also could be defects lurking in the application!

One strategy for assessing the quality of your test suite is to run coverage analysis with a tool like JaCoCo. Since a test case cannot find a defect unless it first executes it, the idea is that you can judge a test suite to be good if it adequately covers the code in the program under test.

However, coverage analysis does not help a tester to know with certainty that the tests are design in such a way that they are likely to find defects. As an alternative to, or in addition to coverage analysis, mutation testing inserts simple faults into the program under test to determine whether or not the tests can find them. Intuitively, a test case that can find many of these faults is judged to be a good one.

While mutation analysis is simple and useful, historically, it has been difficult to apply to large-scale programs because it was computationally expensive. This all changed with the design, implementation, and release of Major, a compiler-integrated mutation testing tool for the Java programming language. You can learn more about Major by reading (Just, Kapfhammer, & Schweiggert, 2011) and (Just, Kapfhammer, & Schweiggert, 2011); more details about how Major makes mutation testing scalable through use of non-redundant mutation operators and test suite prioritization is available in (Just, Kapfhammer, & Schweiggert, 2012).

Starting to use the tool is as easy as changing the Java compiler that you use at build time, as demonstrated by the following line from Major's online documentation.

$ javac -XMutator:ALL
# Generated Mutants: 190 (96 ms)

Rene Just, the creator of Major, has taken great care to ensure that it is fast, allowing mutation testing to be applied to large real-world programs. Major v1.1.8, for the Java 7 programming language, is now available for download—check Major's site so that you can download the tool and get started with mutation analysis for your JUnit test suites! If you are interested in Major, then you may also want to investigate the use of PIT, another mutation testing tool for programs written in the Java language.

Interested in learning more about this topic? Since this blog post was first written, my colleagues and students and I have published additional papers about mutation testing, with the most noteworthy one being (McCurdy, McMinn, & Kapfhammer, 2016).

Enjoy this post? If so, please read, SEED Interview with Timothy Tsai, my most recent article.



Like my work? Support it!