What is the leading source?

Ever wondered which data you should rely on? Are we monitoring the test scripts or are we monitoring the issue database? Based on which information are you able to give an advise about quality?

It seems obvious to claim that both sources are important and necessary. Both sources appear on those metrics dashboard often are used. Which means that information shown there is used to provide detailed advice whether to go into production or not.

Only what are you doing when they are not in sync? This should not happen? Of course it does!

Situation A:
All test cases are executed and all issues are closed. Based on which source is advice given? Does it matter? Yes, who gives the guarantee that you found all issues? Which source gives an indication about quality? It seems that the list of pending issues provide sufficient arguments towards management to go live. In this case, the issue database gets the benefit supported by the number and type of tests executed. the issue database is the leading source.

Situation B:
All test cases are executed and some failed, for these issues are registered and still open. It seems that test scripts and issue database are in sync providing the same information.
Are you using information the issue database or the test cases? In this situation it seems that the issue database tells nothing about what you tested. The test scripts should. if al important/critical tests are performed then you might be able to give advice based on this information and identify possible risks when going live with known issues. The test scripts are the leading source.

Situation C:
All test cases are performed and passed; only there are some issues still open in the issue database. Those were found during testing only not related to any test case.
There might be situations when test cases are based on designs and they are not telling everything. During testing new issues are found which are not related to any test case. Another situation can be that issues are found which are also in production. In these cases the test scripts cannot be used as resource to provide advice as according to them, there are no issues . The issue database will be used and based on the impact of open issues an advice will be given. The issue database is the leading source.

Situation D:
All issues are closed and not all test cases are executed. Under time pressure you are not able to execute all test cases. Though, all mandatory cases are executed. Bases on information about the test cases an advice can be given. The test scripts are now the leading source.

Situation E:
All issues are closed only the test scripts are showing that there are issues pending. According to the issue database there is no risk anymore. According to the test scripts there is some risk although all cases are executed; there are issues with pending status in the scripts. An approach would be performing a recheck on the issue database. The question here is, does the issue database provide enough information to change the status in the test script? Or, are you executing that test case again? What is now the leading source? If the issue database contains test results which proofs the issue is solved and the time stamp corresponds with the time stamp in the test script, the issue database is the leading source. if insufficient information is available, the test script becomes the leading source.

Situation F:
Not all issues are closed and not all test cases are executed. This is a tricky one. It might be easy to give a negative advice as not everything is done; test cases and issues are open. What if those are not that important? What is the leading source then? I suggest that both sources can provide some information only not enough to become the leading source. Another source should be contacted. This can be a requirement list, a risk list, the manager, the business. In this case a combination of sources should be used.

Situation G:
No test cases are executed, and some issues are still open. This seems to be odd to go live with this information. What about an update of patches. Sometimes tests are performed without writing them down. In this case the only source you can rely on initially is the issue database in combination with knowledge of people. Let's say that the issue database is the leading source.

Of course there are other situations also. The intention of this post is visualize that based on the situation the source that provides information to make decisions differs. You should be aware that you can go into production based on number of test cases or number of issues with respect to their status.