Founder & CEO, QA InfoTech
Mukesh Sharma is the founder and CEO of QA InfoTech. QA InfoTech (an ISO 9001:2008 and CMMI Level... more>>
Test metrics have long been a debated topic around the usefulness factor and associated overhead in assimilating and analyzing them. The question continues to linger especially in the Agile cycle where project timelines are very short. One has to plan very effectively:
• On what metrics make the most sense for the given project at hand
• On how best to collect data such that they are part of the test execution engine (automated wherever possible)
• To explain the relevance of such metrics to the larger product team so stakeholders understand the value and support in drawing meaningful and actionable results from such metrics
The bottom line is that test metrics continue to be a very valuable yardstick in objectively measuring the effectiveness of the test effort and the product's quality; what metrics are used and how the whole program is implemented, including course correction in the metrics used as well as the product engineering effort itself, is what differentiates a successful quality program from a not so successful one.
Let's group the metrics into two heads:
a) Test effort/management metrics and
b) Product Quality metrics. Most metrics will be an overlap which are going to be useful in a traditional project too, but keep in mind that what I call out here are particularly relevant to Agile projects given the intrinsic characteristics of such projects.
Test Effort/Management Metrics:
• Code Coverage - This is more important than how many tests are actually executed. A tester may not have time in an Agile project to execute all tests through formally written test cases. Some may be done ad hoc, some may be done while investigating an issue with a developer, some may be done in a meeting with clients / users etc. What matters at the end is the coverage that has been achieved through all possible execution modes. If Test Driven Development (TDD) is followed, where tests are automated even before the product code has been written, metrics to map such tests with requirements to ensure coverage are important
• Defect Removal and Containment Efficiency (DRE and DCE) - given how short the release timelines are, these metrics help provide an objective picture of the agility of the product team to find defects as soon as they are introduced, contain and remove them. On a similar note, to gauge the test team's effort, turnaround time to regress a bug is a also a good metric to see if there is any scope for improvement
• How defects are found - when defects are filed, testers and the rest of the product team should be encouraged to use the "How Found" field, as that will help understand the major sources of finding bugs. If bugs from test cases are quite low, further analysis might be useful to see if any change in the overall test strategy and execution process is needed
• Test Automation Effectiveness - time associated with creating and maintaining test automation is important to consider especially since the product team relies quite heavily on the automated suite to promote faster test execution. Metrics around measuring the time required to create and maintain automation suites as well as automation reliability (consistency in test automation results) are especially important for Agile projects
• Test automation vs. Manual Testing - metrics to measure defects found through test automation vs. defects found through manual testing can be very useful in making the call on where to invest more or how to account for deltas in case, test automation is not very useful in surfacing defects
Product Quality Metrics:
• Defects continue to be the prime indicator of how product quality is faring. Analyzing defects from various angles, specific to agile projects include defects:
o By module to determine health of individual modules
o Based on priority/severity
o Found post release/ reported by users
o Regression metrics to gauge if specific areas are particularly prone to problems
o By type to analyze the cause - viz. design issues, architecture issues, coding issues. Such analyses will help strengthen problem areas, and the more robust the process can be made, the better for the Agile delivery model given the constraints the team operates in
• Competitive metrics - especially around product's performance in line with what the competition has to offer in areas such as feature sets, performance (page load times, response times) can be very useful in gauging your product's quality
• User satisfaction metrics - especially when you have a beta set of users to evaluate your product, engaging with them to get feedback on your product on a variety of areas, will be very meaningful in prioritizing user stories in Agile projects.
When you decide the set of software testing metrics you want to use for an Agile project, I suggest you approach it from a couple of angles:
1. Understand your project specific constraints
2. Understand the core drivers of an Agile delivery model
Using these as a baseline and the above identified specific metrics as a starting point, chalk a plan for an implementation that does not involve a lot of overhead in collecting and representing the data. Look for tools you can use to expedite this process and explain the value of the model to your test team so they help you successfully implement the system as opposed to just unwillingly submitting data. Be flexible to tune the system along the way, accepting the fact that changing dynamics in an Agile project call for tweaks and adjustments in all areas, and that the Test Metric System is no exception to such changes. After all such changes help you incorporate lessons you've learnt along the way, making your metrics system more meaningful and aligned to your product's life cycle.
Experts on QA
Latest postings by this author
Top Expert Articles