To Release or Not To Release

The cost of poor quality software is staggering. A recent report prepared for the National Institute of Science and Technology (NIST) found that:

“Estimates of the economic costs of faulty software in the U.S. range in the tens of billions of dollars per year and have been estimated to represent approximately just under 1 percent of the nation’s gross domestic product (GDP).” [1]

Some of the reasons we produce so much “faulty” software are:

• We lack a business-appropriate definition of quality for many software products
• We fail to identify and use quality indicators during the development process
• We tend to rely solely on test results when making the release decision
• Customers are not sufficiently outraged to force companies to change their ways

How do you make the decision to release software?

Within most organizations, there is an implicit relationship between testing and quality. What we know to be true is that test results represent one indicator of quality. As pointed out by Steve McConnell:

“Testing by itself does not improve software quality. Test results are only an indicator of quality. Trying to improve quality by increasing testing is like trying to lose weight by weighing yourself more often.” [4]

There are other indicators of quality that can be used in addition to test results. These other indicators include such measures as:

• Requirements Stability
• Design Stability
• Code Stability
• Defect Stability

Get more info on other indicators of quality…

These other indicators can provide valuable additional information that Management can use to help make this important decision. If I were responsible for making the release decision, I would want as much factual information that could reasonably be collected.

Frequently though, test results are the only indicator of quality that software companies look at to make the release decision. But, as I pointed out in my December newsletter:

• If you don’t have a definition of quality, you can’t measure it.

So the situation your company may be in is:

• Test results represent the only indicator of quality used to make the release decision
  
  
• You don’t have a definition of quality and therefore you can’t measure it
  
  
• You don’t have an understanding of the quality expectations of your customers

The problem you face then is:

• How can you determine how much and what kinds of testing are sufficient?
  
  
• How can you make a business-appropriate release decision?

Is your Software “Good Enough”?

For software that is not safety-critical, mission-critical or life supporting, there is a school of thought that says this type of software doesn’t have to be perfect, it has to be “good enough”. Stated another way:

• “Our task is not to blindly eliminate all problems, but to understand the problems and benefits of a situation well enough to eliminate (or prevent) the right problems and also deliver the right benefits.” [2]

One of the definitions of software quality I included in my December newsletter is based on the “Good Enough” principle as defined by James Bach. According to Bach,

“To claim that any given thing is Good Enough is to agree with the following propositions:

• It has sufficient benefits.
  
  
• It has no critical problems.
  
  
• The benefits sufficiently outweigh the problems.
  
  
• In the present situation, and all things considered, further improvement would be more harmful than helpful.

Each point is critical. If any one of them is not satisfied, then the product, although perhaps good, cannot be good enough.” [2]

An important point that Bach fails to mention in his “Good Enough” proposition is that, especially with respect to the first three points, it’s the customer’s perception that matters not the developers.

A problem with the “Good Enough” paradigm is that it results in a de facto definition of quality that may or may not be appropriate for your products and customers. It doesn’t take into account the customer’s perspective, nor does it take into account the other indicators of quality I identified above.

I believe that you need a business-appropriate definition of quality in order to apply the “Good Enough” paradigm. In fact, the NIST Report says that:

“The problem is exacerbated because there is disagreement not only on how to define [good] enough, but also on what tests should be run to determine what is enough. For example, commercial software developers use a combination of the following non-analytical methods to decide when a software element is “good enough” to release:

• A ‘sufficient’ percentage of test cases run successfully.
  
  
• Developers execute a test suite while running a code coverage analyzer to gather statistics about what code has been exercised.
  
  
• Defects are classified into different severity categories and numbers and trends within each category are analyzed.
  
  
• Beta testing is conducted, allowing real users to run a product for a certain period of time and report problems; then developers analyze the severity and trends for reported problems.
  
  
• Developers analyze the number of reported problems in a period of time; when the number stabilizes or is below a certain threshold for a period of time, it is considered ‘good enough.’ ” [1]

Wanted: Business-appropriate Models

If we are to reduce the costs associated with what the NIST Report calls “faulty” software, we need business-appropriate models for making more informed and better business decisions about releasing software. Such models need to provide quality indicators throughout the software development cycle – not just during the testing phase.

With respect to quality, the die is cast by the time software is in test. The quality of software is influenced and determined by many factors including the quality of the:

• development process
• requirements
• design
• code
• tests

as well as the skill and professionalism of the staff.

All of these factors contribute to the ultimate quality of the product. Using only one indicator of quality without considering others results in a biased view of the quality of your product.

Organizations need to work on producing higher quality software from the start. Not only will this reduce test time, it will reduce overall cost and enable companies to become more competitive.

The “Good Enough Testing” Model

In looking for a business-appropriate model, I came across Bach’s “Good Enough Testing” approach. He defines “Good Enough Testing” as:

“… the process of developing a sufficient assessment of quality, at a reasonable cost, to enable wise and timely decisions to be made concerning the product.” [3]

He further expands this notion to include the following four activities:

• “Assessment of Product Quality
  
  
  • How accurate and complete is it?
    
    
• Cost of Testing
  
  
  • How reasonable is this cost?
  • Is it within project constraints?
  • Is there are good return on investment, such as the extent of information gained per test?
    
    
• Decisions
  
  
  • How well does the assessment serve the project and the business?
    
    
• Timing of all the above
  
  
  • Is it soon enough to be useful?” [3]

All of these are critically important questions that need to be asked and answered if we expect to get a good return on the investment we make in testing.

While I think that Bach’s “Good Enough Testing” is a good start, more needs to be done if Management is to have a business-appropriate model for making better business decisions.

For now, chew on this and send me your thoughts and comments. Stay tuned for an example of what a business-appropriate model for making better business decisions might look like.