QualityDev.net

Currently the IT media is fawning over Wednesday’s public beta release of Microsoft’s Windows Server Longhorn Beta 3. With most customers planning to wait at least a year before upgrading to Longhorn, a lot of focus will be on the organizations willing to jump into the game of “Russian Roulette” when opting to upgrade. For those that do, there will be some additional overheads: hardware (I’m assuming they are going to run in parallel with current systems for a while), software (obviously) and training. But the main focus will be the comments on installing, reinstalling, configuring and managing this system. Oh, did I mention reinstalling…?

Obviously many, many companies are reliant on Microsoft’s products (us included) to run their business, and it has become the norm to expect quick bug patches and late releases of Service Packs as experience has demonstrated. There seems to be an almost audible ‘groan’ in the industry when something goes bad with a Microsoft product, whether it is a slip in release date or security issue, followed by an ‘understanding’ sense of “Well, it’s Microsoft, they’ll get it fixed soon”.

Joe Wilcox’s article in eWeek, Microsoft’s Big Bang is When? is an unsurprising stream of unreconciled comments from various product managers, vice presidents and the marketing department. Another common part of Microsoft’s reputation in the past.

In my opinion, these issues are largely a reflection of the quality of the various Microsoft products’ code bases. The rewrite of the Vista code back in 2004 must have sent shivers down the spine of management at the time. However, this decision was made on the basis that the quality of that code would have not been anywhere near what has been released this week, so the fact that Microsoft actually made that tough decision must be applauded. It is rare to see such things reported in the media, and moves the industry toward raising the ‘quality’ issue and demonstrating that quality does matter to us. It reminds me of the ‘prototyping’ practice – how many teams have just continued to build an application on top of prototype code rather than scrapping the prototype and rewriting a better quality application?

The recent flurry of security holes found and patches delivered to key Microsoft products also serves to taint their reputation but being the organizational giant they are, with millions of happy users around the world, the image they project with a great marketing campaign and the reputation with home users, significantly outweighs their quality problems.

My conclusion? You could not honestly say that code quality plays a big role in Microsoft’s success. You’d have to say that they are better at marketing than at creating high quality applications. But, it’s truly encouraging to see them making bold decisions to improve the quality of their output, which sets a great example for the rest of us, who may not have the, uh, inertia of a company like Microsoft.

My initial reaction to Andrew Binstock’s post on Effectiveness of Pair-wise Tests, was: “Excellent! A way of getting a better return on developers’ testing time. Surely that’s what every manager wants!”

Resource allocation for testing legacy code (I steal Michael Feathers’ definition here – legacy code is any code that hasn’t been unit tested) is one of the biggest factors when considering code quality improvement initiatives. It’s the same old story: the development team understands that this should be done and [most] understand the benefits it will bring, however, the business demands that applications get to the customer on time regardless (and I do understand the business reasons for this).

In one example, Binstock gives results from BJ Rollinson’s testing of attrib.exe (which takes a path + 6 optional args) using Pairwise testing as:

Minimal: 9 tests, 74% code coverage, 358 code blocks covered.
Pairwise: 13 tests, 77% code coverage, 370 code blocks covered
Maximal: 972 tests, 77% code coverage, 370 code blocks covered

77% code coverage represents a very good return for the work involved, but “hold on”, I thought, “this means that almost a quarter of this code is still untested”. I haven’t seen the details, but I would be willing to bet that part of the untested code is exception handling code (please correct me if I’m mistaken, Mr. Rollinson).

There is no doubt that new methods to get best practices such as unit testing into the mainstream development lifecycle (as efficiently as possible) represent much needed and welcome progress. I also believe that setting a procedure for unit testing legacy code is even more vital (especially in the short term). Not only can lower hanging fruit be plucked to make commonly used parts of the application more robust, but also one of the most common security violations (the dreaded output to the screen of a stack trace when an error occurs) can easily be eliminated.

In almost every application I have seen unit tested, this lack of attention to exception handlers is a common oversight. Every developer knows that users have an uncanny knack of using our applications in ways that we never intended. When that happens, we need to consider what they see. If you are working for a financial company on some sort of web banking/payments application, what information could your users get access to?