Help! The Testers Want to Break the Bank

by Alan S. Koch, PMP

Establishing Quality Objectives
Before you can make a good decision about quality-related issues, you must determine the quality objectives that you need to achieve on this project. Testers often want to make the product's quality as great as possible, without consideration for other project objectives. This can lead them to lobby for greater emphasis on quality-related activities than is actually called for.

Establishing quality objectives is best done by looking back at prior projects. Did the last few projects produce systems that were of acceptable quality? Or did they fall short to some degree? These prior projects provide us with a basis for establish quality objectives for this one. If prior quality was judged to be sufficient, then we should probably plan to achieve the same quality levels this time.

On the other hand, if prior quality was judged to be insufficient, then we should plan to improve on it. How much improvement we want to make this time is a matter of judgment. While we might like to jump directly from "poor" to "perfect," we should be realistic in setting quality objectives, and plan for some incremental improvement (e.g. reducing delivered defect levels by 10% or 20%).

Planning to Meet Our Quality Objectives
Once we have established our quality objectives, we can plan for the activities that are needed to achieve them. If prior product quality was judged to be sufficient, then we should expect that placing the same emphasis on quality this time will also result in adequate quality.

For example, if we spent 5% of the schedule on Peer Reviews and 30% of the schedule on testing, then in order to achieve the same level of quality, we should plan to spend those same percentages this time. If we spend less time on those quality-related activities this time, we should not be surprised if we produce poorer quality!

If we need to improve on our past quality performance, then we must plan to invest more time in the activities that will result in improved quality than we did in those prior projects. Does this mean that to achieve a 20% improvement in quality, we must spend 20% more time in testing? Unfortunately, it isn't that straightforward. There are actually a wide variety of investments we can make that will pay quality dividends.

Increase testing
If on those prior projects, testing was abbreviated and the product released before it had demonstrated readiness, then increasing the amount of time for testing may be called for. Although testing finds many defects that we fix before release, that is not its primary purpose. The main purpose of testing is to demonstrate that the system is stable, useable, and ready for release. So, we must allocate enough testing time for the product to reach that point.

How much more time should we allocate? This can only be determined by talking with the testers. What percentage of the planned tests ran successfully before release on those prior projects? If the number was in the 70%-80% range, then you were probably approaching stability when testing was stopped. In that case, a marginal increase in test time might be all that is required. But if only 50% or less of the tests had run successfully, then much more time is likely needed to reach the stability point.

But more testing is not our only option. There are other options that can improve the quality of the system before testing begins, thus allowing the same amount of testing (or maybe even less testing) to bring us to that stability point.

Peer Reviews
If you do not employ peer reviews, then this is a significant opportunity for you to achieve aggressive quality objectives without increases in testing. Peer reviews can save significant project time because they can result in more defects found and fixed per hour spent than can ever be achieved during the testing phase.

When a group first begins to do peer reviews, it is not unusual for them to find 40%-50% of the defects in the product during the reviews. Imaging that: half of the defects that might have been found during testing being fixed before testing even starts! And as your team becomes proficient in doing reviews, they are likely to find 60%-70% of the existing defects.

There is a tremendous amount of information about the effectiveness of reviews that has been published in magazines, books, research papers, and on the Internet. If you question their value, I recommend that you read about the results other companies have experienced.

Defect Prevention
Many defects can be prevented from happening by strengthening the engineering practices you use. For example, if your system or component designs are generally documented on the backs of bar napkins, then more attention to design could pay significant dividends.

Many projects don't focus enough on architectural analysis before the system is built. People make educated guesses about how it should be structured, and rely on assumptions about how the code they are writing will fit together with other parts of the system. When those assumptions or guesses turn out to be erroneous, the necessary rework often results in convoluted implementations that are prone to failure.

Although up-front design work is how most of us would approach this problem, the Agile methods (e.g. XP) prescribe a different approach: Refactoring. Refactoring is essentially reworking the design after the fact. That is, when it becomes evident that the original structure is not working well, you stop and redesign the system or component, and then rework or replace the code to match that new design. Although Refactoring may sound wasteful, it is often a better spend of time than pressing forward with an ill-conceived design and fixing all of the problems that it causes.

Another way to prevent defects is by adopting coding standards that lead programmers to use sound practices and to avoid those that cause problems. Coding standards also make it easier for programmers to perform peer reviews of code, because they facilitate their ability to read and quickly understand each other's programs.

Improve Defect Removal Effectiveness
Finally, we can achieve better quality performance by improving the effectiveness of the testing or reviews we already do. Every one of these activities has a certain yield (percent of existing defects that it finds), and a certain efficiency (number of defects fixed per hour spent).

For example, if your testing is finding fewer than 40% of the existing defects, then you should be able to make it more effective (though exceeding 60% yield is uncommon for testing). Peer reviews can reach yields of 75%-80% after the team members have become proficient with them. If your review yields are below 60%, then you should look at improving them. If your reviews are resulting in fewer defects found and fixed than your testing, then there is clearly room to improve their efficiency!

In order to determine what you need to do to improve testing or reviews, you will have to look at how your practices differ from the industry best practices that have been abundantly published in the literature. Perhaps your team members could benefit from training to become more effective. Or maybe it is a matter of adjusting the procedures you use.

Achieving Your Quality Objectives
You need not break the bank (or blow out the schedule) in order to achieve your quality objectives. By making appropriate investments, you can often realize the quality levels you need while staying within your project constraints. And in the cases where the project constraints are unreasonable, the research you have done and the investment you have made will provide you with the information you need to negotiate with project stakeholders and bring their expectations in line with what can be achieved.

In the end, you will have a project that achieves its quality goals while staying within budget and schedule.

©Copyright 2000-2017 Emprend, Inc. All Rights Reserved.
About us   Site Map   View current sponsorship opportunities (PDF)
Contact us for more information or e-mail
Terms of Service and Privacy Policy