It's easy to monitor on defects, but when you start reaching your 1000th defect and you keep track of all kinds of data, you might start loosing control on what the actual quality of your application might look like. You start to lose overview on where the issues lie and what actions need to be taken to control product quality within delivery time. You drown in the graphs you need to present and in stead of reading your report yourself and making the correct conclusions, you'll use your energy creating the shiny report that nobody reads.
Many test coordinators and test managers let themselves drive by all the information available and recorded in the defect registration tool, presenting all kinds of exotic graphs but at the same time lose focus on the actual reason of the defect report - Giving insight in product quality.
So how do we provide insight in product quality, making use of a defect report?
1. First of all, before starting to report, you need to get data in the defect registration system.
You only want to capture data that is useful to show product quality. All other data is waste.
- Divide your software into logical blocks to report defects on (Using functional and technical blocks) to report defects on eg: Intake; Contract; Interfaces System A; Interfaces System B,... Make sure you don't have more than 9 of those. If you have more, regroup.
- Define defect priorities and/or severities : 'Urgent - High - Medium - Low' will do for starters
- Defect statuses to report on do not equal your actual defect statuses - simplify and group them for a report. Use:
- Open (For the developer to pick up or to resolve)
- Closed - Rejected (Closed - Agreed not to fix)
- Resolved (Needs a retest)
- Closed - Fixed ( Closed and fixed)
- Escalated (No agreement found - needs to be taken up in a defect board)
- You want to know what the actual issue was of defect when it is being resolved. This can be data, test, design, front end code, back end code, business rule, migration... again make sure you don't have more than 9
2. Present your data in a way that anyone can understand what you want to show.
- The titles of the graphs you present, should be meaningful and unambiguous
- The report should be printable on maximum 2 pages and still be readable
- Use the correct chart types to display what you want to show. Trend views are no luxury.
- The colors you present need to be in line with their meanings
- Always provide an interpretation. A report that only shows metrics is can be easily misunderstood.
An example of how a defect report could look like, is shown below:
