All Software is Defective

Implications for the Software Industry…

Your alarm clock rings and switches to soothing nature sounds that ease you into your morning. As you hit the shower, the thermostat has already raised the temperature setting so that the kitchen will be warm when you come down for breakfast. The coffee maker starts brewing your favorite caffeinated beverage at just the right time. After breakfast, you head out to work. Your car’s on-board computer monitors everything from anti-lock brakes, engine, emissions, airbags, seat belts, tire pressure, audio system, and of course, the GPS navigation system. On the way to work, you stop at an ATM kiosk. You get a call on your cell phone and respond by sending a text message. On the highway, you pay tolls using a transponder that bills your credit card. The on-board navigation system receives updates on traffic conditions and recommends alternate routes so you arrive at your office in the shortest possible time…

By the time you arrive at work, you have already used many millions of lines of code, without even realizing it.

Unless you live under a rock, it is almost impossible to go through a day without coming in contact with software. Often, it is not obvious that we are using software when doing some task. Software is everywhere and affects our lives on a daily basis. And yet, all software is defective.

After over five decades of software development experiences, we do not have the ability to develop software that has zero defects. Nor do we have the ability to prove that software has zero defects.

To be fair, we don’t have the ability to produce anything of similar complexity that has zero defects. Cars, airplanes, medical devices, weapons systems, ATM machines, etc., all have defects. So why should we expect more from software? Well, with many products we can often demonstrate “that we have taken all practical steps and used all available methods to make the product as close to defect free as possible.” [1] We frequently can’t do this for software products.

Just like other complex products, we are not expected to prove that software is defect-free. Unlike other products, however, most customers expect software to have defects.

“The imperfect software that provides computers with intelligence and functionality is purchased because it is affordable and readily available, and has enjoyed the ‘forgiveness’ that comes with being a relatively new technology.” [5]

When it comes to software, we should be expected to take advantage of all that we know about software development in order to produce software that, when delivered, has as few defects as possible. A problem that pervades the software industry is that:

“The current common view that it is impossible to produce defect free software is an excuse for not making the effort to do quality work.” [1]

Why Doesn’t Management Seem to Care About Quality?

When customers demand better quality, Management generally responds. Their actions may not always lead to improvement, but they do take action. The automotive industry is a prime example.

In the software industry, a recent study by the US National Institute of Science and Technology (NIST) [3] estimates that software defects cost the US economy almost $60 billion a year! Given the increasing trends in application complexity and the lack of improvements in software development and testing, this number is certainly likely to increase…

Why Don’t Customers Seem to Care About Quality?

“Because defective software works.” [1] It may not work in every situation or for every user, but it usually works as long as users stay within defined use models. These defined use models represent the best guess of developers and testers as to how users will use the software.

Watts Humphrey calls these defined use models the “testing footprint” [2] as shown below.

The green area represents the use model that the people who developed the application had in mind. Users who work within the green area will encounter a few problems. As these problems are resolved, the software, while far from defect-free, works well enough to satisfy customers whose use model is within the green area.

However, when users start thinking of new ways to use the software, serious problems can arise. These new uses represent the yellow areas that surround the green shaded middle. The yellow area represents uses that were not anticipated by the developers and testers. Since these new uses were not anticipated, the software was not designed to support these uses and was certainly not tested in the context of these new uses. So, it should come as no surprise that there will be many serious problems…

Furthermore, as observed by Humphrey,

“When coupled with the explosive growth of the Internet and the resulting exposure to hackers, criminals, and terrorists, the need for reliable, dependable, and secure software systems will steadily increase. If experience is any guide, as these systems are used to perform more critical functions, they will get more complex and less reliable. Unfortunately, this probably means that it will take a severe, disruptive, and highly public software failure to get people concerned about software quality.” [2]

A “…severe, disruptive, and highly pubic software failure…” – haven’t we already had several of those? In last month’s newsletter, I identified several famous bugs that certainly were severe, disruptive and highly public. But there is no outrage, not yet anyway. In the not too distant future, customers will be demanding that software companies improve quality. As customers have done with other products, they will seek those companies that can and are doing better…

So, given the ubiquitous nature of software and the fact that all software is defective, what are the implications for developers, testers, management, and users…

Implications for Developers…

A problem with current software development methods is that developers have come to rely on tools rather than their own skills to find problems with their code. In the old days, programmers (as they were called then) were more scientific, since most were trained in the discipline of mathematics or physics. Back then, when you developed a program, you spent a considerable amount of time desk-checking your work. The end result was usually programs that not only compiled correctly, but that generally worked the first time.

At a software engineering conference many years ago, an electrical engineer was giving a talk about a circuit analysis program that he had developed that was widely used. No one had ever found a defect in this program and he was asked how did he achieve such an accomplishment? His reply was very insightful. He said, “I didn’t think defects were allowed.”

Today, not only are defects allowed they are expected. Many developers have resigned themselves to the fact that there is little they can do to reduce the number of defects. Today, many applications are released with hundreds or even thousands of known defects and an unknown number of defects that were not found. How big is this unknown number of defects that are not found? Let’s look at the Defect Injection Rate for a sample of 810 experienced developers: [1]

Please try this at work…

The average defect injection rate for all developers in this group was 120, or about 1 defect for every 8 lines of code. Think about the size of your applications – now do the math…

This number probably isn’t something you’ll want to brag about.

The frustrating thing about this is that we have proven methods developers can use to reduce the Defect Injection Rate. These techniques include the Personal Software Process (PSP), Formal Inspections, and Joint Application Development (JAD) just to name a few. We have these techniques yet we choose not to use them.

As a developer, you have the choice to do good work. The quality of your work, be it design or code, should be something that you are passionate about and proud of.

You need to be passionate about quality
if you want to produce a quality product.

Implications for Testers…

Most current software testing models are based on a premise that may no longer be valid. The premise is that we (developers and testers) understand how users are going to use an application. If this premise is true, we can use many testing techniques to come up with a reasonably good set of tests. In the right environment, we can find many defects. Even so, testing is an expensive way to find bugs. The only more expensive way is to let customers find them…

Please try this at work…

The find/fix cycle represents the time required to find and fix one defect. Every software development company should know how long their find/fix cycle is. Use this table to help you estimate your find/fix cycle cost

In reality though, applications have been and will continue to be used in ways not anticipated by developers and testers as illustrated by the testing footprint figure above. Several high profile defects described in last month’s newsletter are prime examples. So what can testers do about this?

Testers depend on information.

• There is information that is known (requirements and domain knowledge, for example).

• There is information testers don’t know (ways individual customers might choose to perform an operation using the application, for example)

• There is also information that you don’t know you don’t know. This is the killer. When customers push the envelope and move that yellow area shown in the testing footprint above – bad things can happen and there’s little that testers can do to anticipate this.

What can be done though is to minimize the impact of what you don’t know. This can be done by using a combination of testing techniques including:

• Exploratory Testing (ET)
  
  James Bach defines exploratory testing as being similar to doing a jig saw puzzle. The techniques and approaches people use to do a jig saw puzzle often change as the puzzle begins to take shape. He says that the “puzzle changes the puzzling.” [6]
  
  Read more about Exploratory Testing…

• Act Like A Customer Testing (ALAC)
  
  Testers need domain knowledge to be effective. Armed with domain knowledge, testers can put themselves in their customers shoes and do what customers do. This is what I call Act Like a Customer Testing.
  
  Read more about ALAC…

• Storytelling
  
  One very effective technique to help testers do ALAC testing is to use storytelling to help identify specific scenarios that need to be tested.
  
  Read more about storytelling…

• Requirements-based testing
  
  Requirement-based testing is testing against the requirements. This is just as important as these other techniques.

Implications for Management…

Until customers demand higher quality, management will not likely pay much attention to this issue. But just like the automobile industry, by the time management decides to finally address this issue, it will be too late. Customers will have found higher quality products elsewhere.

Capers Jones [4] offers a list of things management can do now to be proactive rather than reactive:

• Recognize that no single method is adequate.

• Testing alone is insufficient.

• Formal inspections and tests combined give high efficiency, low costs and short schedules.

• Defect prevention plus inspections and tests give highest cumulative efficiency and best economics.

• “Bad fix” injection needs special solutions.

• Database errors need special solutions.

• Web “content” needs special solutions.

Lastly, many organizations offer incentives to get products released. However, there are often no incentives in to produce code that has fewer defects, This sends the message to developers that getting product released is more important than quality. There needs to be a balance. There needs to be incentives for both – quality and time to market.

Implications for Users…

Users need to demand that software suppliers deliver software with fewer defects. User need to find those suppliers that have a reputation for quality. For those suppliers that deliver shoddy products, users need to threaten legal action if their quality doesn’t improve. Often, that’s the only way to get their attention…

Summary

Everyone involved in the production and consumption of software needs to recognize that all software is defective. Given the pervasive nature of software in business and society, we need to find and use techniques that have a demonstrated impact on reducing defects in order to develop applications that are more robust, safer, and less likely to fail.

Till next time, happy holidays!

p.s. If there is a topic that is of particular interest to you, please let me know and maybe I can address it in a future newsletter.

Pay it Forward

If you haven’t already done so, please consider donating to your favorite charity during this holiday season. There are many people in our neighborhoods who need help all year round but especially at this time of year.

Read more about the Pay It Forward Foundation…