Michael Feathers’ recent blog post Prosthetic Goals and Metrics That Matter that references Brian Marrick’s Code Coverage Misuse paper, seemed to cast a negative light on code coverage tools. I was glad to see that in a subsequent comment, he clarified his statement by saying that it was setting coverage numbers as an organizational goal rather than simply the use of coverage tools that he objects to.
Feathers states: “You can’t measure quality with [code] coverage.” I don’t necessarily agree with this. Just setting a threshold of say, 90% as an incentive marker is, I agree, not a good idea, because it encourages developers to look for the ‘easy’ tests to push their numbers up. However, with a good program of test reviews in place, setting goals can be motivational for developers and enhance the quality of the application, especially if thresholds are set low to start with, and gradually changed over the life of the project.
Note I said “changed” rather than just “increased.” If a major design change is applied, a lot of code and tests may be removed, which will throw off coverage numbers.
Is comparing these numbers a bad thing? I don’t believe so. It means developers who perhaps were not testing before are now gaining experience in writing tests, and a safety net is being put in place for more of the code (I agree that testing simple Getters and Setters is a waste of time though). If your team includes experienced developers, and peer/team lead reviews are occurring (which should be the case), any persistent gaming of these numbers by individuals will be recognized over time.
Using a code coverage tool will not only increase the quality of the application by showing what has been tested, but also what has not been tested. On many consulting engagements, I have used a code coverage tool to point to areas in the code that have, as yet, not been tested. I regularly find that within these applications, similar patterns present themselves (a frequent occurrence is a lack of testing of exception handling) and with the tool I use, the owner of the code is identified as well. I can quickly point out obvious areas in the code that would be good candidates (and high priority) for testing, and discuss suitable training areas for individuals who are consistently missing these areas that should be under test.
I believe we are at a time in the software industry where quality issues are being discussed more openly, and motivation to look into these quality areas needs to be encouraged. I believe that it is the responsibility of management and experienced technicians who have seen the lack of quality on prior projects, not to mention the results, to motivate and yes, maybe give incentives to teams that are pro-active in implementing quality procedures. However, a good review process to stop any gaming of these numbers is crucial for success.
I would argue that any tool, used properly, that can be incorporated into an automated build process (and used by individuals) to enhance the quality of a developer’s work and therefore any application they are contributing to, is a Good Thing
Recent Comments