Evaluating exit criteria is the activity where test execution is assessed against the defined objectives. This should be done for each test level, as for each we need to know whether we have done enough testing. Based on our risk assessment, we’ll have set criteria against which we’ll measure ‘enough’. These criteria vary for each project and are known as exit criteria. They tell us whether we can declare a given testing activity or level complete. We may have a mix of coverage or completion criteria (which tell us about test cases that must be included, e.g. ‘the driving test must include an emergency stop’ or ‘the software test must include a response measurement’), acceptance criteria (which tell us how we know whether the software has passed or failed overall, e.g. ‘only pass the driver if they have completed the emergency stop correctly’ or ‘only pass the software for release if it meets the priority 1 requirements list’) and process exit criteria (which tell us whether we have completed all the tasks we need to do, e.g. ‘the examiner/tester has not finished until they have written and filed the end of test report’). Exit criteria should be set and evaluated for each test level. Evaluating exit criteria has the following major tasks:
• Check test logs against the exit criteria specified in test planning: We look to see what evidence we have for which tests have been executed and checked, and what defects have been raised, fixed, confirmation tested, or are outstanding.
• Assess if more tests are needed or if the exit criteria specified should be changed: We may need to run more tests if we have not run all the tests we designed, or if we realize we have not reached the coverage we expected, or if the risks have increased for the project. We may need to change the exit criteria to lower them, if the business and project risks rise in importance and the product or technical risks drop in importance. Note that this is not easy to do and must be agreed with stakeholders. The test management tools and test coverage tools that we’ll discuss in Chapter 6 help us with this assessment.
• Write a test summary report for stakeholders: It is not enough that the testers know the outcome of the test. All the stakeholders need to know what testing has been done and the outcome of the testing, in order to make informed decisions about the software.