What Do You Do to Ensure Your Software Is High Quality?

In a past life, as Software Quality Manager for a medical device company, I was asked to help the software development group select a real-time operating system for a new embedded product being developed. Working together, we identified a few potential candidates. I was asked to make a recommendation based on quality and business factors. Having never done this before, I was at a loss as to how to make such a recommendation…

As I was thinking about this, our Purchasing Manager happened by and asked if I would like to accompany him on a Supplier Audit. As an ISO-9000 certified company, suppliers were required to be selected based on criteria that were established by Purchasing with input from appropriate departments.

The Purchasing Manager was in a similar situation. Manufacturing asked him to select a supplier for a subsystem that was a critical part of this new product. Since the subsystem included some software, he needed some help and I agreed to go along. It was an eye-opening experience…

The Purchasing Manager had prepared an extensive questionnaire that he had already sent to each potential supplier. By visiting several potential suppliers, we could discuss their responses and have a firsthand look at their processes and the results they achieve. Collecting this information from several potential suppliers and comparing results, Purchasing was then able to select the best possible supplier for this particular subsystem.

After I returned, I became convinced that the same approach could work for selecting software suppliers. I started work on a similar questionnaire I could use for assessing potential software suppliers.

In my research for this questionnaire, I stumbled across an interesting story from Tom Van Vleck. He calls this story - Asking the “nasty question…”

I have a favorite question to ask of people trying to sell me software. I let them go on and on about features, functions, benefits. Then I smile and say, "What do you do to make your product high quality?" I've gotten some interesting answers. "Oh, the programmers are very sincere. They work really hard." or "After we write the code we run a lot of tests we found somewhere." or my favorite one, "Well, you see, I am Swiss, and I am very precise."

When I get an answer like this I smile more, and say, "Yes, but what exactly do you do to make your product high quality?" Some people still don't get it. I asked one prospective supplier my question, and they answered with a lot of charts about the bug rate in their shipped products. They were very proud that as the product matured, the bug rate went down. Another prospective supplier had a better answer: they talked about their systematic testing, generated from an assertion language and validated by a coverage tool. But these are both about how they recover from quality problems, not about how they put quality in. [1]

With Tom’s “nasty question” in mind, I began creating what evolved into a Software Supplier Quality Assessment tool. Over the years, I’ve revised this assessment tool based on actual experiences resulting from problems with third party software suppliers. With each use of the assessment tool, I select appropriate questions from the list and add questions specific to the situation if necessary. The result is a fairly specific questionnaire that can be used to assess the work processes and capabilities of potential software suppliers.

Performing a Software Supplier Quality Assessment may be appropriate when your company is considering acquiring a tool or some other software component that is critical to the success of a software product your company is developing.

More importantly, doing such an assessment of your own company may also be appropriate – given that your potential customers may be doing this…

Back to Tom’s story…

If anybody asks me this nasty question, I want to be ready with a good solid answer. Ideally, everybody on my team should know the answer, believe in it, contribute to it, and be able to evangelize it. Our answer should be true, and way beyond what the competition can claim. It should be obvious to the user that our quality is phenomenal. [1]

When asked the “nasty question”, you need to identify the specific things you do to make sure that you get it right, not what you do when you find problems, or how many problems you find, or what the trends are, etc.

So how would you answer the “Nasty Question”?

Suppose that a significant potential customer approaches your company and asks you the “nasty question”. How would you respond?

You’re probably thinking – “No customer has ever asked us this question so why worry about it?” Well, they may not have ever asked you this question in the past, but the times they are a changin’.

The explosion of software companies in places you’ve probably never heard of is the reason you need to worry about this. You can bet that these hungry companies are following a model that proved highly successful with outsourcing organizations in places like Bangalore. These companies started at CMM Level 5 because they believed that this would give them a competitive advantage. Well guess what, they were right. When you couple a perception of high quality with low labor rates, you can see why so many US companies have outsourced work to places in Asia and the Far East.

Read more about outsourcing…

Startup software development companies recognize that their competitive advantage lies in delivering high quality software – something that many US companies still find elusive. So when customer come knocking on their virtual doorsteps, and ask the “nasty question” they often have a compelling answer… an answer that talks about processes that prevent bugs and not about how many bugs they find.

Back to Tom’s story…

Tom recommends that you should do the following four things:

1. Zero defects: A friend was telling me about the landscaping he was having done. The landscapers had made some mistakes, and he said, "I paid for a perfect job. They'll have to fix it." The customer expects a perfect job from us. Even in a team that already have wonderful commitment to quality, some folks are scared of promising zero defects... but that’s what we expect from others. And any tiny defect can blow the reputation of the whole system, in the user's eyes. (This is the cockroach theory: if you see a roach in a restaurant, you don't say "There's the roach in this place." You say, "Let's get out of here, this place is infested.") [1]

   Even though I don’t believe zero defects is achievable, Tom’s point is well taken. I would express this a bit differently though – Deliver products with as few defects as possible. We frequently ship products with known defects. In some industries, this has become an acceptable practice as long as the defects that remain are not likely to impact customers. Therefore, I would further refine the goal to be – Deliver products with as few defects as possible that customers are likely to encounter.
   
   Being able to deliver products with as few defects as possible that customers are likely to encounter requires that testers have the skills I outlined in my October newsletter…

2. Proceed systematically: Architecture, procedures, well-reviewed designs, high-level languages and discipline for using them, plans for thorough testing, and measurement of results are all evidence for a development process that's under control. In addition, the team should have documented methods for design, defect prevention, and inline testing, and tools to mechanize these. [1]

   Proceeding systematically means that you create a system architecture and document your software design. Why do this? Watts Humphrey [2] says that there are five reasons to document your design:

   • to discipline the design work
     
     
   • to facilitate design reviews
     
     
   • to manage change
     
     
   • to preserve and communicate the design to others
     
     
   • to enable a quality and cost-effective implementation

   Read more about the importance of design…

3. Check everything: Every work product should be checked somehow. Most teams review designs, review source code, and test compiled code. The principle can be applied to other things we do for a "client" anywhere that quality might suffer. If this principle becomes part of our culture, each step can concentrate on preventing or removing its own defects, counting on clean input from the prior step. [1]

   You need to have effective Peer Reviews and, more importantly, individual developers and testers need to be responsible for “desk checking” their own work. “Desk checking” is one of those skills that seems to have been deemed old fashioned. Significant benefit can be derived from desk checking your work. Using the buddy system (I desk check your work, you desk check my work) is another technique that can be very effective in finding defects earlier rather than later.

   Read more about Peer Reviews - see Karl Wiegers book, Peer Reviews in Software: A Practical Guide

4. Improve continuously: If we do all the above, it won't be enough. We need to learn from our mistakes, look for better ways, question authority. Our methods and processes have to keep evolving to meet changing conditions. [1]

   Continuous improvement and learning from past mistakes is a very important objective. The organization needs to have a process focus and have process effectiveness measures in place in order to affect process improvement.

   Organizations that are not able to continually review and adapt processes to changing requirements and customer needs are likely to become less competitive over time.

   Learning from past mistakes can be accomplished through the Project Retrospective process that I described in an earlier newsletter.

   Read more about Project Retrospectives…

Summary  

If it hasn’t happened already, at some point in the not too distant future, your company is going to be asked tough questions by potential customers. Tom’s “nasty question” is but one example. You need to have a compelling answer – an answer based on doing things that prevent problems rather than just doing what everyone else does, which is reacting to problems found. Being proactive is always better than being reactive…

Pay it Forward

If you find this newsletter of value, please consider the following:

Norm Kerth is a highly respected consultant who developed the Project Retrospective techniques discussed in the July-Aug newsletter. He was in a serious car accident and suffered a disabling brain injury. As a result, he cannot work and lives on a very limited income. You can help recognize his contribution to our industry by sending a small donation. Checks can be made payable to Norm Kerth Benefit Fund and sent to Norm Kerth Benefit Fund c/o Process Impact, 11491 SE 119th Drive, Clackamas, OR 97015-8778. You can also visit Karl Weiger’s website (Process Impact) for more details about contributing to the fund. Thanks.

Read more about the Pay It Forward foundation…