Take the 2-minute tour ×

I recently had a test assignment where I was testing a pretty well solidified API. The team started phase 1 by creating basic functional, boundary and error tests for each API call. It turned out to be three or four tests per API call. The error checking was really basic. Things like row counts for inserts and so on. We found a majority of the bugs in this phase during test case creation.

Once that cycle was done and all the bugs found were fixed, phase 2 used a data driven approach. We exercised tens of thousands of variations using a fairly robust set of test data with "poison pills" in the data wherever we could. To my surprise, the number and severity of bugs found in phase 2 were quite low compared to the number of person hours invested. I had expected to find some outlying killer bugs or at least a fair number of medium priority bugs.

Based on the number and severity of bugs found, the team's time could have been better spent with ad-hoc testing or moving on to other product areas.

Is my result a fluke, or when you are automating at the API layer do you find you hit the point of diminishing returns relatively quickly?

share|improve this question
Good question :) I want to point out that you haven't reaped the full benefit of your automated tests yet. They will continue to provide value for future iterations as well, as regression tests. –  Ethel Evans May 23 '11 at 22:37
Quite true. It would become evident in the future releases if the data driven set of test cases provide enough test coverage. –  Aruna May 24 '11 at 4:20
We were already at 100% code coverage when we starting the iteration tests. The product api was completely revamped in V2, so the automation was mostly scrapped. It seems like over half of all automation has a very low lifespan for various reason. –  Dustin Andrews May 24 '11 at 18:42
add comment

2 Answers

up vote 6 down vote accepted


Great question. The diminishing marginal returns you saw are frequently encountered. The following chart, while not the easiest to decipher, explains a lot of why diminishing marginal returns occur:

Decreasing marginal returns in software testing

The chart, taken from an article called "Combinatorial Software Testing" published in IEEE Computer that I co-wrote with 3 PhD's, explains that the significant majority of defects can be triggered by testing each value you can think of at least once. Several thorough studies were done to answer the straightforward and important question, "how many test inputs would be required to trigger defects in production today?" Four very different Systems Under Test were examined (including defects in medical devices - in red - and the Mozilla browser - in green). Each defect in each SUT was categorized... to trigger this defect, we only need one test input; in order to trigger this one, this specific PAIR of test inputs need to appear in the same test case, etc. Approximately 85% of the defects in production could be triggered by just one or two test inputs. Exactly one defect in the four studies required six specific test inputs to be triggered. None required 7 or more.

What are the implications of these findings?

They are huge. According to these findings:

The biggest bang for your buck will come from testing every test input in at least one test case.

  • That sounds pretty similar to what you were doing in your initial set of tests.

The next biggest bang for your buck will come from testing every PAIR of test inputs together in at least one test case.

  • Your initial set of tests also included a great many pairs of values; they would have to unless your tests each consisted of just a single test input.

The next biggest bang for your buck will come from testing every single possible combination of three test inputs together in at least one test case.

If you continue from there to cover all combinations of 4 test inputs together, and/or all combinations of 5 test inputs together, and/or all combinations of 6 test inputs together, in many cases, you may execute hundreds of thousands of combinations of test inputs without seeing a single defect.

  • As you moved from your early tests (consisting of a relatively small number of tests) to your later tests (consisting of tens of thousands of tests), what additional new coverage were you achieving?

  • From my analysis of dozens of similar projects with findings similar to yours, I strongly suspect that the tens of thousands of tests you ran in the "second round" of your testing, you were adding very few combinations involving "as yet untested PAIRS of test inputs" (which would have given you a reasonable chance of finding an as yet undetected bug). Rather, the net new coverage added in each test would often be several "as yet untested specific combinations involving, say, 4 or more test inputs tested together for the first time in this test" (which, statistically, we can see are extraordinarily unlikely to trigger defects).

The final implication from these findings is that if you want to maximize the efficiency of finding defects in a situation like yours:

First, execute all the test inputs at least once.

Next, execute a set of tests that will cover all pairs of test input values (e.g., execute pairwise tests - AKA allpairs tests / 2-way tests), then consider executing the smallest possible set of tests that will ensure coverage of all possible combinations involving 3-way combinations. If you find very few defects when executing your 3-way tests, you may seriously consider stopping your testing without executing higher strength test sets.

share|improve this answer
Note there is open source software available for generating the kinds of test vectors Justin described. –  user246 May 24 '11 at 0:31
This pretty accurately describes my approach. I actually wrote my own software for creating the 2-pair and 3-pair combination, which is what we automated. In essence I wrote a program that wrote my test cases. Your graph really confirms the results of my experiment. I would conclude that in a V1 environment getting past pairwise testing doesn't have strong ROI for test hours invested. In a more stable product where you rightly expect more versions to come, it's more likely to be worth the investment. –  Dustin Andrews May 24 '11 at 18:45
Dustin, You wrote your own software to create 2-way and 3-way coverage?! Cool! You're a man after my own heart. It's a shame so few testers know about these powerful test design methods. Please check out our Hexawise test case design tool and let me know your feedback. My experience confirms your assertion that moving from 2-way (pairwise) to 3-way tests has a much lower ROI than the 2-way tests because you'll often quadruple the # of tests executed to get to 3-way coverage with a much lower number of new defects found per test as compared to the bugs per test ratio from the 2-way tests. –  Justin May 24 '11 at 19:40
add comment

Firstly, re :: "I had expected to find some outlying killer bugs or at least a fair number of medium priority bugs."

What was the reason for these expectations ? Was it the coverage of the tests ? Was it a perceived flakiness in the code ? Were you expecting killer bugs to be still there inspite of them being found and fixed in the previous cycle ? Was there an overlap in paths which were exercise in the previous test run or the current one ?

Secondly,to answer your direct question :: Whether you have hit the point of dimishing return "quickly" or not would depend upon whether your expectations from the second test run ( questions above) being aligned realisticly? Were you expecting too much from the second test run ? And now, where you stand today , still I think it is worth the investment because these would act as crucial regression tests now ! , provided they are robust ( show passes and failure s consistently ) , are actually capable in catching regression ( can be simulated through error injection in the code ?)

share|improve this answer
add comment

Your Answer


