Regression testing of software is costly — but you can do something about it!

03 July 2017

flickr photo by stuartpilbrow shared under a Creative Commons (BY-SA) license

To establish a confidence in the correctness of their software, developers will often write test suites that they will re-run as the program is modified, attempting to ensure that they have not introduced regressions, or faults that arise as an unintended side effect of changes. This process, known as regression testing, is important and yet expensive.

Regression testing is costly because, as software grows in size and complexity, a suite accumulate more tests and therefore take longer to run. For instance, without any optimizations, testing the core libraries in the Zendesk application takes four hours. Moreover, a 2011 conference paper reports that the regression testing of a Microsoft product required several days. Of course, even if a regression test suite only takes minutes to run, developers may find its execution distracting.

Prior work has developed many techniques to address the substantial computational cost associated with regression testing, with test suite reduction and prioritization emerging as two of the most promising. Test suite reduction aims to find a smaller test suite that covers the same requirements as does the full test suite (Lin, Tang, & Kapfhammer, 2014). Alternatively, test suite prioritization aims to find a test ordering such that faults in a program can be detected as early as is possible, thus enabling the processes of debugging and program repair to quickly start (Lin, Chen, Tsai, & Kapfhammer, 2013).

Another approach to regression testing involves selecting only those test cases that exercise the program functionality that was most recently changed. A recent paper reports that test selection saves Microsoft millions of dollars every year — all without compromising the quality of the program under test. So, interested in learning more about regression testing? If you are, then please read (Kapfhammer, 2010).

Do you perform regression testing of software in industry? Or, do you conduct regression testing on an open-source tool? Are you interested in discovering ways to transition cutting-edge research into your practice of regressing testing? If you answered "yes" to any of these questions, I hope that you will soon contact me with your own ideas and experiences.

Finally, want to cite the 2011 conference paper that I mentioned? You can use this BibTeX reference:

  author      = {Carlson, Ryan and Do, Hyunsook and Denton, Anne},
  title       = {A Clustering Approach to Improving Test Case Prioritization: An Industrial Case Study},
  booktitle   = {Proceedings of the 27th International Conference on Software Maintenance},
  year        = {2011}

Enjoy this post? If so, please read, SEED Interview with Timothy Tsai, my most recent article.



Like my work? Support it!