Software Quality Assurance…

…Leveraging Your Resources 

The topic of my Jan 2005 newsletter was “What is Software Quality Assurance”? This month, I would like to expand on this theme and illustrate ways you can leverage your SQA resources to improve productivity and quality, while at the same time, getting products released on time…

An interesting aspect of working with many companies is that I get to see how different companies define the role of SQA. The result - there is no standard way that SQA is defined. Every company seems to have a slightly different spin on SQA. In fact, a cross the software industry, there is significant variation in key factors that directly influence the effectiveness of SQA. These factors include:

• degree of independence
• mission statement
• authority and responsibility
• tasks performed
• skill level
• relationship with development
• support from management
• training...

As a result, there is a corresponding variation in SQA effectiveness.

What is meant by SQA effectiveness and how can we measure it?

The SQA Spectrum

When you look at how SQA is performed across a broad cross-section of companies, some patterns begin to appear. These are illustrated below:

First, I have found that there are two distinctly different SQA roles. The blue line represents the traditional product assessment role while the red line represents a process assessment role.

Process Assessment Role

This role is often found in high discipline companies, especially those that are following CMM SM or CMMi SM models. In this role, SQA audits the software development process by reviewing development artifacts to ensure that prescribed processes have been followed. SQA is not involved in testing or other development activities, but rather, is tasked with checking to see if these activities occur.

Given that this is essentially an internal auditing function, assessing SQA effectiveness is difficult at best.

Product Assessment Role

The product assessment role is by far the most common role for SQA. Here SQA is tasked with assessing product quality.

As shown above on the blue line, there is wide variation in what SQA does in this product assessment role. On the left hand side are those organizations where SQA’s role is limited primarily to testing. Often these organizations are small and relatively low discipline. For these organizations, measuring SQA effectiveness is also difficult since there often isn’t a definition of software quality and as a result, the mission of SQA isn’t clearly defined.

In my Dec 2004 newsletter I said that there is no single definition for software quality. As a result, it is difficult to assess SQA effectiveness in this role.

• Does your organization have a definition of quality?

• Is your definition measurable?

Low discipline organizations often produce minimum documentation. The kinds of projects they work on are usually relatively lower risk.

As shown above, the right hand side of the blue line represents those organizations where SQA’s role includes more than testing. These organizations tend to be mid-sized to large companies producing products that are more likely to be mission-critical and business-critical. These organizations also typically have a measurable definition of software quality and clear goals and objectives for SQA. As a result, SQA effectiveness can be measured.

SQA Umbrella

So what can SQA do besides testing? Well, there are many ways SQA can add value to the organization. Roger Pressman [1] describes SQA as an umbrella for a collection of activities that may include those shown below:

Watts Humphrey’s view of SQA is that:

“SQA is a valid discipline in its own right and people can be SQA experts without being software design experts. This SQA expertise is what is required to establish a strong quality program. It includes knowledge of statistical methods, quality control principles, the software process, and an ability to deal effectively with people in contentious situations.” [2]

Leveraging SQA Resources

By learning how to leverage your SQA resources, you can help reduce typical testing cycles since there would be fewer problems to find and fix. This means that products can get to market sooner, rather than later, and that customers will be generally more satisfied with the results.

Here are three examples of areas beyond testing where you can leverage your SQA resources in a manner that will yield tangible results…

1. Peer Review the Requirements
   
   One activity that has a huge return on investment is peer review of requirements. Involve SQA in this activity so they can ask the question:

   “Is every requirement testable?”

   You will probably find many requirements that are ambiguous and as a result, not testable as written. Use the Design for Testability techniques I discussed in my March 2006 newsletter to express requirements in a manner that is less ambiguous and most importantly, testable.

   Remember that:
   
   Every hour of effort spent removing ambiguity in the requirements during the early stage of a project can result in saving potentially 100 hours of effort during the testing stage!
   
   
2. Triage
   
   Most organizations have many more defects than resources to fix them. And developers generally are want to fix those defects that are “interesting” or are related to their own code, rather than those defects that impact customers most directly.
   
   A Triage Team is empowered to review all defects and decide which ones need to be fixed – because they impact customers – and which defects can be deferred. This approach helps make more effective use of resources - for fixing defects and for regression testing.
   
   SQA can play a key role as part of the Triage Team by bringing the customer’s perspective to the discussion. Of course, this requires that SQA have “domain knowledge”, that is knowledge of how your customers are using your products.
   
   
3. Root Cause Analysis
   
   Customer Reported Problems (CRPs) are the most important type of defects that you have. You need to understand what you did not understand about how customers are using your products that has led to CRPs.
   
   This understanding can result in changes that can prevent recurrence in future releases. Clearly, this is an important role for SQA since it represents potential gaps in your “domain knowledge”.
   
   Learn more about Root Cause Analysis…

Action Plan

If your SQA team is involved with just testing, here’s what you can do to leverage their effectiveness:

• Get them involved in reviewing requirements for clarity and testability…

• Start testing as soon as possible – don’t wait till all of the code is ready…

• Make sure SQA is represented on the Triage Team…

• Make sure SQA has domain knowledge…

• Get SQA involved in the Root Cause Analysis of CRPs…

Summary

The role SQA plays clearly needs to be commensurate with the business risk associated with product development as well as consumer risk in using the product. To find the SQA role appropriate for your organization:

• Define SQA Goals and Objectives
  
  Align these goals and objectives with overall business goals and objectives. Factor in business risk and consumer risk. Remember that:

  “It is a mistake to assume that SQA people themselves can do anything about quality.” [2]

• Define SQA’s Role Within the Organization
  
  Based on the goals and objectives, define the specific role SQA will play. Remember, that:

  “If SQA fulfills its responsibilities and if senior management refuses to allow line management to ship products until the SQA issues have been addressed, then SQA can help management improve product quality.” [2]

• Staff SQA With Talented People
  
  Management needs to ensure that the best people are found to staff the SQA function. These people require training and support just like any other critical function within the organization.

  “Getting good people into SQA is one of the most difficult problems software managers face.” [2]

• Establish Effective SQA Processes
  
  Just like development, to be effective SQA needs good processes. The processes developed by SQA should be focused on achieving the following broad goals:
  
  
• To help development improve software quality by assessing the quality of software products as well as the quality of the processes used to produce those products.
  
  
• To determine the level of compliance with established development processes and procedures.
  
  
• To bring to Management’s attention any deficiencies in either the product or the process so that Management can take appropriate corrective action.

Your feedback is important…

Each month, this newsletter tries to provide you with some useful nugget of information. This is a two-way street and your feedback is important. Please send me your thoughts and comments at steve@swqual.com