What’s your Testing ROI?

Have you ever thought about how expensive software testing is? Even if your testing is outsourced, testing is an expensive, time-consuming activity. Like coding, testing is labor-intensive. Tests have to be planned, developed, debugged, maintained and executed. Defects, once found, need to be tracked and triaged. Decisions need to be made about which defects should be fixed. When fixed, defects need to be verified and regression testing needs to be performed to determine if fixes have introduced new defects.

Like most other expensive, time-consuming business processes, it’s important to know what the Testing Return on Investment (ROI) is and how it could be improved. Many software companies have no idea how much testing costs, much less their Testing ROI. You’d think that an activity so critical to the business and so expensive would be constantly measured, monitored, and managed…

What is Testing ROI?

The most basic definition of ROI is:

ROI = [(Payback - Investment)/Investment)]*100

where: Payback is the amount earned from your investment

Investment is the cost of resources required to generate the payback

To calculate Testing ROI, we need to understand both the investment and the payback.

The Investment

Testing is performed by developers (unit testing) and QA (functional, system, validation testing are several terms used for testing performed by QA). For this discussion, we’re focusing only on testing performed by QA.

The testing investment can be determined by looking at testing costs, such as:

• Cost of resources (labor) spent on all testing and testing-related activities
  
  
• Cost of resources (equipment and infrastructure) to support testing activities
  
  
• Costs of resources (labor) required to deal with defects found from testing

Companies may have additional costs that may be specific to their business. Quantifying these costs is certainly feasible.

The Payback

The payback from testing is determined by looking at:

• Customer satisfaction is higher as a result of encountering far fewer defects
  
  
• Employee satisfaction is higher as a result of greater pride in producing higher quality products
  
  
• Sales and revenue are higher because higher quality products compete more successfully against products perceived to be of lesser quality
  
  
• Development and support costs are lower because higher quality products are less expensive to develop and support
  
  
• Maintenance costs are lower because higher quality products are easier to maintain and enhance
  
  
• etc…

Capers Jones [1] surveyed hundreds of software development companies. Of those that produce higher quality software, he found that they have:

• shorter development schedules
• lower development and maintenance costs
• better reliability and longer mean time to failure
• larger market share
• better user satisfaction
• better employee morale
• less chance of ending up in court

Quantifying testing payback is difficult, not because we can’t identify what it is, but because it is difficult to measure. We all know that testing identifies defects and that there’s a strong correlation between defects and quality. When defects are fixed properly, quality improves.

As a result, it is difficult to measure the Testing ROI because many of the Paybacks are not conducive to measurement or represent intangibles. For example, if you are among the few software companies who measure customer satisfaction, can you determine what portion of customer satisfaction is attributable to product quality? Can you determine what portion of product quality is attributable to testing (as opposed to using good engineering practices)? This is hard to do…

So, while we all agree that testing is a good thing to do, we can’t realistically measure the Testing ROI. But, since we all agree testing is a good thing to do, we can identify ways to make testing more effective. Taking steps to make testing more effective will clearly improve the Testing ROI, even though we may not be able to directly measure it.

Here are some examples of what I mean when I say we can improve testing effectiveness:

• Are you testing the most important features? How do you know?
• Are there features you’re not testing? I bet you are…
• Are you testing the same features many times? I bet you are…
• Are you testing against vague and ambiguous requirements? I bet you are…
• Does your regression suite have tests that are not relevant? I bet it does…

The Problem

Testing requires a significant investment in time and effort. Each year, companies spend hundreds of thousands of hours testing software. Typical test suites often number into the thousands of tests, many of which require hundreds of hours to develop, maintain, and execute. Often, tests are written against requirements that are vague and ambiguous. Regression test suites evolve over time and often include large numbers of what I call “non-productive” tests. These tests often are looking for problems in areas where there aren’t problems. Couple this with the fact that testers are inclined to develop more tests in the areas of the application they are most familiar with, leaving other areas under-tested or not tested at all.

We need ways to assess testing effectiveness. And, we need to assess effectiveness BEFORE we start testing so that we increase the Testing ROI.

Measuring Testing Effectiveness

Very few techniques have been developed to measure testing effectiveness. Here are some examples of what has been done:

Chernak [2] defined a measure called test case effectiveness (TCE) as:

TCE = N_tc / N_tot * 100%

where:

N_tc = defects found by test cases
N_tot = sum of defects found by test cases and found by other means

Huber [3] at Hewlett-Packard developed a set of metrics to measure test effectiveness. The H-P metrics illustrate the importance of measuring the effectiveness of the testing activity.

Factors that Impact Testing Effectiveness

In looking at the testing effectiveness measures from Huber and Chernak, I think we need to focus on productive vs. non-productive testing as one way to improve testing effectiveness.

Productive tests are tests that have a much higher probability of finding defects. Since the whole purpose of testing is to find defects, we would obviously want to develop as many productive tests as possible.

Non-productive tests are tests that will likely never find a defect, no matter how many times the tests are executed.

If you accept the premise that we need to focus on developing as many productive tests as possible and that having more productive tests will surely improve testing effectiveness, we can then focus on identifying factors that affect our ability to write productive tests. My list includes:

• Testing against Poorly written requirements
• Unrealistic development and testing schedules
• Lack of measurement capability
• Lack of testing infrastructure (staff, hardware, systems, automation tools)
• Lack of domain knowledge
• Lack of training in basic testing techniques

Given these factors, we can now look at ways to assess Testing Effectiveness.

Testing Effectiveness Assessment

A Testing Effectiveness Assessment is a process that can improve the Testing ROI and includes the following five steps:

1. Scrub the Requirements
   
   Tests can only be as good as the requirements they are intended to test. If requirements are vague and ambiguous, the tests will not be very effective. Therefore, the first step in the assessment process is a thorough review of the requirements.
   
   Based on this review, a Requirements Scrub may be necessary. For more information on writing requirements that are unambiguous, check out the Sept 2004 Food for Thought
   
   
2. Update the Requirements Trace Matrix (RTM)
   
   Once the requirements have been scrubbed, the next step is to review and update the RTM. The RTM is an extremely important document. In its simplest form, it provides a way to determine if all requirements are tested. However, the RTM can do much more.
   
   

   Example RTM

   Requirement	Est. Tests Req'd	Types of Tests	Automated or Manual?	Re-use Existing Tests?
   4.1.2 Change Password	22	10 Functional 3 Positive 4 Negative 5 Boundary	7 Auto 5 Manual	T0167 T0045
   4.1.4 Calculation Precision	5	2 Functional 3 Boundary	5 Manual	T0056 T6069
   …		…
   …		…
   Total Estimated Tests	150		90-A 60-M	50

   
   
   In this example, the RTM is used as a test planning tool to help determine how many tests are required, what types of tests are required, whether tests can be automated or manual, and if any existing tests can be re-used. Using the RTM in this way helps ensure that the resulting tests are most effective.
   
   The RTM can serve many purposes over the course of a development project. Initially, it can be used as a planning tool (as illustrated above). Once the tests are developed and Validation Testing has begun, the RTM can be used to help determine the extent of regression test required based on the relationship be requirements, design, code, and tests as illustrated below.
   
   When it becomes necessary to perform regression testing, the accurate information included in the RTM will be invaluable in helping to select a reasonable set of tests to run.
   
   
3. Analyze the Existing Test Suite
   
   The RTM can also be used to help analyze the existing test suite. For example, an analysis of test types can be performed to determine if a variety of test types have been associated with each requirement. By using a wide variety of types of tests, the tests are likely to be more productive.
   
   

   Examples of Test Types
   Functional Algorithmic Positive Negative Usability Boundary Act Like a Customer Startup Shutdown Configuration Platform Load/Stress Security	Performance Data Flow Safety-related Compatibility Documentation Timing Error Checking Power Failure Resources Installation Upgradeability Volume Scalability Throughput

   
   
   In addition to looking for a variety of test types, there are specific test strategies that are also important to identify. For example:
   
   
   • Data Flow Testing
     
     Data flow testing is a strategy based on known patterns of data usage and data flow. Data flow tests are often derived from data flow diagrams.
     
     
   • Equivalence Classes
     
     A group of tests can form an equivalence class if:
     
     
     • they all test the same thing
     • if one test passes, all tests will likely pass
     • if one test fails, all tests will likely fail

     Identifying potential equivalence classes allows redundant, non-productive tests to be identified.

   • Boundary Analysis
     
     By identifying ranges and boundaries, tests can be created that are:

     • well within boundary or range
     • one less than boundary
     • equal to boundary
     • one greater than boundary
     • well beyond boundary

     Boundary analysis ensures that all boundary conditions are covered. Again, this increases the likelihood that these tests are productive.

   • Act Like a Customer Testing
     
     Act Like a Customer (ALAC) testing is based on how users actually use your product. These tests are critical in helping to uncover the kinds of defects that users are likely to encounter. ALAC tests can be easily created from Use Case/Workflow diagrams.
     
     

     Act Like A Customer Testing
     	Software has lots of defects… Users typically find only a small percentage of the total… Act Like A Customer Testing helps focus on finding those defects your customers are likely to find.

     
     
4. Scrub the Test Suite
   
   The next step in the Testing Effectiveness process is to remove redundant, non-productive tests from the test suite.

5. Identify Additional Training for Testers
   
   Training for testers can help improve skills in areas such as:
   
   
   • Test Planning
   • Act Like a Customer Testing
   • Equivalence Classes
   • Data Flow Testing
   • Boundary Analysis
   • Requirements Management
   • Test Automation

The Bottom Line

So, while it may not be practical to measure Testing ROI, improving testing effectiveness will certainly improve the Testing ROI whether we can measure it or not.

Till next time…