Welcome to Food for Thought™, an e-newsletter from Software Quality Consulting. I've created free subscriptions for my valued business contacts. If you find this newsletter informative, I encourage you to continue reading. Feel free to pass this newsletter along to colleagues by clicking this Forward Email link. If you’ve received this newsletter from a colleague and would like to subscribe, please click this Enter New Subscription link. If you don't wish to receive this newsletter, click the SafeUnSubscribe™ at the bottom of this newsletter, and you won’t be bothered again.

Your continued feedback on this newsletter is most welcome. Please send your comments and suggestions to info@swqual.com.

In This Months’ Topic, we continue the discussion of software quality begun last month with a discussion of software quality assurance.

Regular features to look for each month are:

• Monthly Morsel: Hints, tips, techniques and reference info related to this month’s topic.

• Calendar: Conferences, workshops, and meetings of interest to software engineers, QA engineers and anyone interested in software development.

What is Software Quality Assurance?

Ever wonder where Software Quality Assurance (SQA) came from? The activity we call SQA evolved from Independent Verification & Validation (IV&V) – a collection of activities associated with large, mission-critical projects. Understanding IV&V is important because it represents the “heritage of SQA”. Many of the tasks that we typically associate with SQA originated with IV&V. This month, we’ll explore the evolution of SQA from IV&V as a backdrop for a discussion on “What is SQA?”

Historical Perspective

Back in the late 1950's, software began to find its way into systems procured by the US Dept. of Defense (DoD). Not surprisingly, these projects were consistently behind schedule, over budget, and had many technical problems. Frequently, software never worked as intended and many projects were cancelled before anything was delivered. Throughout this period, software development contractors often gave overly optimistic assessments of the software development status to the DoD. The DoD was frequently unaware of schedule, budget, and technical problems until late into the program – when they were often unable to understand them and assess their impact.

The IV&V role was established to address this problem. The first program to use IV&V was the Atlas Missile Program in the late 1950's. An “independent software tester” was hired to “perform additional, unbiased testing of the software”. [1] By employing someone who was totally separate from the software development contractor, the DoD hoped to get a more accurate and objective technical assessment of the project's status.

Over time, the role of the “IV&V contractor” became critically important. IV&V has been and still is used on many large, mission-critical projects for DoD, NASA, FAA, HUD, and DEA. Today, the set of tasks performed by IV&V contractors is quite comprehensive. See the draft IEEE Standard 1012 Software Verification & Validation for specifics. (Note this is a draft standard and has not yet been approved.)

Since Atlas, much data has been collected to support the assertion that projects with IV&V perform much better than similar projects without IV&V. [2], [3] As a result of this data, NASA now requires IV&V to be applied on applicable NASA projects. [13]

Much of the success of IV&V is attributable to the fact that IV&V contractors are completely independent of the software development organization. Working for and reporting to the procuring entity, IV&V contractors provide an unbiased, objective technical and managerial assessment of a project. As a result, the procuring entity is in a much better position to identify and resolve issues that could otherwise easily be overlooked (intentionally or unintentionally) by the software development contractor. By raising these issues in a timely manner, they are more likely to be resolved in a way that does not negatively affect the project.

Emergence of SQA

During the 1970’s, as software development activity began to expand, software development companies were experiencing the same poor results as did government agencies a decade earlier. Companies had difficulty delivering software within the constraints of schedule, budget, and quality. Many projects undertaken in the 1980’s and 90’s were disasters – either because they failed to deliver anything, they grossly exceeded budget and schedule parameters, or they delivered software of such poor quality that it was unusable.

During the 1980’s, we experienced what became known as the “software crisis” – the point in time when spending on software maintenance exceeded spending on creating new software products. The advent of the “software crisis” brought with it a host of changes - not the least of which was the emergence of SQA.

Drawing on its roots in IV&V, SQA evolved into an effective tool that software development companies have used to help identify quality problems earlier in the development process. While SQA was viewed as the “poor stepchild” of software development, many enlightened managers of the day saw measurable benefit from integrating SQA into the software development process.

By the 1990’s, many software companies had SQA functions within their organizations. Yet, high profile software failures continued to occur. (see [4, 5, 14]) Was SQA not living up to expectations? Hard to say. But there were several differences in the nature of software being developed during this time that are worth noting:

• Complexity of software developed during the 90’s increased significantly.

• Competitive business pressures also increased significantly.

• Software was being used in many new areas – especially areas that were life-threatening.

• Many people working in SQA received little formal training in SQA. SQA engineers were expected to learn their craft primarily from on-the-job training.

• Universities failed to recognize that SQA is a legitimate discipline unto itself and that it requires specialized training.

As observed by Watts Humphrey:

“SQA is a valid discipline in its own right and people can be SQA experts without being software design experts. This SQA expertise is what is required to establish a strong quality program. It includes knowledge of statistical methods, quality control principles, the software process, and an ability to deal effectively with people in contentious situations.” [6]

What is SQA?

Just like quality, there are many definitions of SQA:

• The “confidence” definitions:
  
  The IEEE defines quality assurance (QA) as:

  “A planned and systematic pattern of all actions necessary to provide adequate confidence that an item or product conforms to established technical requirements.
  
  A set of activities designed to evaluate the process by which products are developed or manufactured”. [7]

  Daniel Galin defines SQA as:

“A systematic, planned set of actions necessary to provide adequate confidence that the software development process or the maintenance process of a software system product conforms to established functional technical requirements as well as with the managerial requirements of keeping the schedule and operating within budgetary confines”. [8]

• The “visibility” definition:
  
  The Software Engineering Institute (SEI) defines SQA as:

  “Software Quality Assurance provides management with appropriate visibility into the process being used by the software project and of the products being built”. [9]

• The “assurance” definition:
  
  Don Reifer says:

  “Software quality assurance is the system of methods and procedures used to assure that the software product meets its requirements. The system involves planning, measuring and monitoring developmental activities performed by others.” [10]

• The “fitness for use” definition:
  
  Schulmeyer and McManus define SQA as:

  “The systematic activities providing evidence of the fitness for use of the total software product”. [11]

• The “umbrella” definition:
  
  Pressman defines SQA as an “umbrella activity that is applied throughout the software process”. [12] He further states that “SQA encompasses:
  
  
  1. A quality management approach
  2. Effective software engineering technology (methods and tools)
  3. Formal technical reviews applied throughout the software process
  4. A multi-tiered testing strategy
  5. Control of software documentation and the changes made to it
  6. A procedure to measure compliance with software development standards
  7. Measurement and reporting mechanisms” [12

• The “management tool” definition:
  
  Watts Humphrey defines SQA as a “management tool that must be properly used to be effective.” [6]

The Many Faces of SQA

Before I give you my definition of SQA, I first want to share with you some of the many different roles that SQA often plays in organizations.

• The “Process Police”
  
  
  • SQA’s motto might be:
    
    
    • “Our job is to make sure Development follows the process.”
      
      
  • SQA’s role:
    
    
    • Audits work products to identify deficiencies
    • Determines compliance with project development plans and software development process
    • Focus on process assessment rather than product assessment
      
      
  • What often happens:
    
    
    • SQA is perceived as nitpickers obsessed with minutia
    • SQA insists on inordinate amounts of mostly unproductive testing
    • SQA becomes obsessed with finding the “last defect”

• The “Customer Advocate”
  
  
  • SQA’s motto might be:
    
    
    • “You must please us because we represent the customer.”
      
      
  • SQA’s role:
    
    
    • Identifies issues customers would likely find
    • Helps organization become sensitive to customer needs
    • Actively involved in performing “Act Like a Customer Testing” which results in higher customer satisfaction
      
      
  • What often happens:
    
    
    • SQA may not have sufficient “domain knowledge” to accurately represent customers
    • SQA may inject their own preferences or biases in the name of customers
      

• The “Analyst”
  
  
  • SQA’s motto might be:
    
    
    • “The more data we collect the more ammunition we have.”
      
      
  • SQA’s role:
    
    
    • Collects lots of data on all aspects of product and process
    • Good data can be used to help improve processes and products
    • Examples: defect data, test coverage data, requirements traceability data, etc...
      
      
  • What often happens:
    
    
    • SQA can’t see the forest for the data – analysis paralysis sets in
    • SQA becomes obsessed with meaningless metrics
    • Customers are more concerned with not finding defects than with metrics

Confused by all these different definitions?

Ok, so what is SQA?

First, some points to consider:

• To be effective, SQA must be independent of the development organization.
  
  IV&V is effective because IV&V contractors are independent (managerially, technically, and financially) of the software development contractor. For SQA to be most effective, a similar degree of independence is needed. The SQA function should report to the same executive that the most senior development manager reports to. The SQA budget should not be contingent upon development’s budget. And SQA should attempt to collect as much technical information as possible independently of the development organization.
  
  
• SQA provides Management with timely, factual information.
  
  The key here is that SQA provides Management with objective, technical information. Management needs to view SQA as a source of meaningful information that can help them make difficult decisions. SQA should not make the release decision. That decision rightly belongs with Management.
  
  
• Management uses this information to make appropriate business decisions.
  
  SQA should provide information on risk, potential customer impact, etc. By using information from SQA, development, and other sources Management can then make informed decisions about releasing products. The information SQA provides needs to be considered in this decision, but ultimately, the decision rests with Management. Sometimes SQA may not agree with the decisions Management makes. (Get over it – it’s their job not yours!). If Management never agrees with SQA, there’s a problem…

Ok, here’s my definition of SQA:

SQA as a “Provider of Information”

The most effective role that SQA can play is as a provider of useful, timely information.

In this role:

• SQA’s motto would be:
  
  
  • “We review what was done and provide an objective technical assessment supported by facts so that management can make better business decisions.”
    
    
• SQA’s role would include:
  
  
  • Providing objective technical information Management can use to make better business decisions
    
    
  • Providing information appropriate for the nature of products and risks associated with those products
    
    
  • Being more concerned with reducing risks than insisting on 100% compliance with process

What Role is Best for Your Organization?

The role SQA plays clearly needs to be commensurate with the business risk associated with product development as well as consumer risk in using the product. To find the SQA role suitable for your organization:

• Define SQA Goals and Objectives
  
  Align these goals and objectives with overall business goals and objectives. Fctor in business risk and consumer risk. Remember that:

  “It is a mistake to assume that SQA people themselves can do anything about quality.” [6]

• Define SQA’s Role Within the Organization
  
  Based on the goals and objectives, define the specific role SQA will play. Remember, that:

  “If SQA fulfills its responsibilities and if senior management refuses to allow line management to ship products until the SQA issues have been addressed, then SQA can help management improve product quality.” [6]

• Staff SQA With Talented People
  
  Management needs to ensure that the best people are found to staff the SQA function. These people require training and support just like any other critical function within the organization.

  “Getting good people into SQA is one of the most difficult problems software managers face.” [6]

• Establish Effective SQA Processes
  
  Just like development, to be effective SQA needs good processes. The processes developed by SQA should be focused on achieving the following broad goals:
  
  
  • To help development improve software quality by assessing the quality of software products as well as the quality of the processes used to produce those products.
    
    
  • To determine the level of compliance with established development processes and procedures.
    
    
  • To bring to Management’s attention any deficiencies in either the product or the process so that Management can take appropriate corrective action.

I’d like to leave you with the following thoughts:

• “When software projects fail, it’s usually because a manager didn’t insist that the work be done the right way.” [15]
  
  
• SQA isn’t responsible for the quality of your software – Development is.
  
  
• To change the quality of your software, you need to change the process used to create it.

Every month in this space you’ll find additional information related to this month’s topic.

• Software Quality Assurance Organizations:
  
  ASQ Software Division Website

  Software Quality Group of New England Website

  SEI Software Process Improvement Network (SPIN)

  Boston SPIN

  Quality Assurance Institute (QAI)

  And on a lighter note… finally, a good reason for document reviews

• Books and articles:

  [1] Nelson, J. G., "Software Testing in Computer-Driven Systems", in Software Quality Management, ed. Fisher, Matthew J., and Cooper, John D., Petrocelli Books, 1979

  [2] Arthur, J. D. and Nance, R. E., “Verification and Validation Without Independence: A Recipe for Failure”, Proc. 2000 Winter Simulation Conference, Orlando FL, 2000.

  [3] Wallace, D. R., and Fuji, R. U., “Software Verification and Validation: Its Role in Computer Assurance and Its Relationship with Software Project Management Standards”, National Institute of Standards and Technology, Special Publication 500-165, May 1989

  [4] Glass, R., Software Runaways: Lessons Learned from Massive Software Project Failures, Prentice-Hall PTR, 1997

  [5] Weiner, L., Digital Woes: Why We Should Not Depend On Software, Addison-Wesley, 1993.

  [6] Humphrey, W. S., Managing the Software Process, Addison-Wesley, 1989

  [7] IEEE Glossary of Software Engineering Terminology, IEEE Standard 610.12-1990, IEEE, NY.

  [8] Galin, D., Software Quality Assurance – From Theory to Implementation, Pearson Education Limited, 2004.

  [9] Paulk, M., et. al., The Capability Maturity Model: Guidelines for Improving the Software Process, Addison-Wesley, 1995

  [10] Reifer, D., State of the Art in Software Quality Management, Reifer Consultants, 1985.

  [11] Schulmeyer, G. G. and McManus, J. I., eds. Handbook of Software Quality Assurance, 2nd ed, Van Nostrand Rheinhold, 1992

  [12] Pressman, R. S., Software Engineering: A Practitioner's Approach, McGraw-Hill, 4th edition, 1997.

  [13] NASA Policy Directive NPD 8730.4A, Effective August 1, 2001.

  [14] Johnson, J., “Chaos: The Dollar Drain of IT Project Failures”, Application Development Trends, January 1995, p. 41-47.

  [15] Humphrey, W., Winning with Software – An Executive Strategy, Addison Wesley, 2002.

Every month, you’ll find news here about local and national events that are of interest to the software community …

• Software Quality Calendar
  
  There are many organizations that sponsor monthly meetings, workshops, and conferences of interest to software professionals. Find out what’s happening…

• Workshops Offered by Software Quality Consulting
  
  Software Quality Consulting offers workshops in many topics related to software process improvement. Get more info...

Software Quality Consulting provides consulting, training, and auditing services tailored to meet the specific needs of clients. We help clients fine-tune their software development processes and improve the quality of their software products. The overall goal is to help clients achieve Predictable Software Development™ – so that organizations can consistently deliver quality software with promised features in the promised timeframe.

To learn more about us and how we can help your organization, visit our web site or send us an email.

I hope this newsletter has been informative and helpful. Your comments and feedback are most welcome. Send me your feedback…

Thanks,

Steve Rakitin

info@swqual.com