Food for Thought: What’s Your Testing ROI? An e-newsletter published by Software Quality Consulting, Inc. April 2005, Vol. 2 No. 4 To view a web version of this newsletter, click on the following link: http://www.swqual.com/newsletter/vol2/no4/vol2no4.html. -------------------------------------------------------------------------------- Welcome to Food for Thought(TM), an e-newsletter from Software Quality Consulting (http://www.swqual.com/?Intro). I've created free subscriptions for my valued business contacts. If you find this newsletter informative, I encourage you to continue reading. Feel free to pass this newsletter along to colleagues by clicking this Forward Email link (http://ui.constantcontact.com/roving/sa/fp.jsp?plat=i&p=f&m=sctz69n6). If you’ve received this newsletter from a colleague and would like to subscribe, please click this Enter New Subscription link (http://www.swqual.com/newsletter/Subscribe.htm). If you don't wish to receive this newsletter, click the SafeUnSubscribe(TM) link at the bottom of this newsletter, and you won’t be bothered again. Your continued feedback on this newsletter is most welcome. Please send your comments and suggestions to info@swqual.com. -------------------------------------------------------------------------------- In This Months’ Topic, we we discuss the notion of Testing ROI... Regular features to look for each month are: Monthly Morsels - Hints, tips, techniques and reference info related to this month’s topic Calendar - Conferences, workshops, and meetings of interest to software engineers, QA engineers and anyone interested in software development -------------------------------------------------------------------------------- ***This Month’s Topic*** WHAT’S YOUR TESTING ROI? Have you ever thought about how expensive software testing is? Even if your testing is outsourced, testing is an expensive, time-consuming activity. Like coding, testing is labor-intensive. Tests have to be planned, developed, debugged, maintained and executed. Defects, once found, need to be tracked and triaged. Decisions need to be made about which defects should be fixed. When fixed, defects need to be verified and regression testing needs to be performed to determine if fixes have introduced new defects. Like most other expensive, time-consuming business processes, it’s important to know what the Testing Return on Investment (ROI) is and how it could be improved. Many software companies have no idea how much testing costs, much less their Testing ROI. You’d think that an activity so critical to the business and so expensive would be constantly measured, monitored, and managed... WHAT IS TESTING ROI? The most basic definition of ROI is: ROI = [(Payback - Investment)/Investment)]*100 where: Payback is the amount earned from your investment Investment is the cost of resources required to generate the payback To calculate Testing ROI, we need to understand both the investment and the payback. THE INVESTMENT Testing is performed by developers (unit testing) and QA (functional, system, validation testing are several terms used for testing performed by QA). For this discussion, we’re focusing only on testing performed by QA. The testing investment can be determined by looking at testing costs, such as: - Cost of resources (labor) spent on all testing and testing-related activities - Cost of resources (equipment and infrastructure) to support testing activities - Costs of resources (labor) required to deal with defects found from testing Companies may have additional costs that may be specific to their business. Quantifying these costs is certainly feasible. THE PAYBACK The payback from testing is determined by looking at: - Customer satisfaction is higher as a result of encountering far fewer defects - Employee satisfaction is higher as a result of greater pride in producing higher quality products - Sales and revenue are higher because higher quality products compete more successfully against products perceived to be of lesser quality - Development and support costs are lower because higher quality products are less expensive to develop and support - Maintenance costs are lower because higher quality products are easier to maintain and enhance - etc... Capers Jones [1] surveyed hundreds of software development companies. Of those that produce higher quality software, he found that they have: - shorter development schedules - lower development and maintenance costs - better reliability and longer mean time to failure - larger market share - better user satisfaction - better employee morale - less chance of ending up in court Quantifying testing payback is difficult, not because we can’t identify what it is, but because it is difficult to measure. We all know that testing identifies defects and that there’s a strong correlation between defects and quality. When defects are fixed properly, quality improves. As a result, it is difficult to measure the Testing ROI because many of the Paybacks are not conducive to measurement or represent intangibles. For example, if you are among the few software companies who measure customer satisfaction, can you determine what portion of customer satisfaction is attributable to product quality? Can you determine what portion of product quality is attributable to testing (as opposed to using good engineering practices)? This is hard to do... So, while we all agree that testing is a good thing to do, we can’t realistically measure the Testing ROI. But, since we all agree testing is a good thing to do, we can identify ways to make testing more effective. Taking steps to make testing more effective will clearly improve the Testing ROI, even though we may not be able to directly measure it. Here are some examples of what I mean when I say we can improve testing effectiveness: - Are you testing the most important features? How do you know? - Are there features you’re not testing? I bet you are... - Are you testing the same features many times? I bet you are... - Are you testing against vague and ambiguous requirements? I bet you are... - Does your regression suite have tests that are not relevant? I bet it does... THE PROBLEM Testing requires a significant investment in time and effort. Each year, companies spend hundreds of thousands of hours testing software. Typical test suites often number into the thousands of tests, many of which require hundreds of hours to develop, maintain, and execute. Often, tests are written against requirements that are vague and ambiguous. Regression test suites evolve over time and often include large numbers of what I call “non-productive” tests. These tests often are looking for problems in areas where there aren’t problems. Couple this with the fact that testers are inclined to develop more tests in the areas of the application they are most familiar with, leaving other areas under-tested or not tested at all. We need ways to assess testing effectiveness. And, we need to assess effectiveness BEFORE we start testing so that we increase the Testing ROI. MEASURING TESTING EFFECTIVENESS Very few techniques have been developed to measure testing effectiveness. Here are some examples of what has been done: Chernak [2] defined a measure called test case effectiveness (TCE) as: TCE = Ntc / Ntot * 100% where: Ntc = defects found by test cases Ntot = sum of defects found by test cases and found by other means Huber [3] at Hewlett-Packard developed a set of metrics (http://www.swqual.com/articles/HPMetricDescription.pdf) to measure test effectiveness. The H-P metrics illustrate the importance of measuring the effectiveness of the testing activity. FACTORS THAT IMPACT TESTING EFFECTIVENESS In looking at the testing effectiveness measures from Huber and Chernak, I think we need to focus on productive vs. non-productive testing as one way to improve testing effectiveness. Productive tests are tests that have a much higher probability of finding defects. Since the whole purpose of testing is to find defects, we would obviously want to develop as many productive tests as possible. Non-productive tests are tests that will likely never find a defect, no matter how many times the tests are executed. If you accept the premise that we need to focus on developing as many productive tests as possible and that having more productive tests will surely improve testing effectiveness, we can then focus on identifying factors that affect our ability to write productive tests. My list includes: - Testing against Poorly written requirements - Unrealistic development and testing schedules - Lack of measurement capability - Lack of testing infrastructure (staff, hardware, systems, automation tools) - Lack of domain knowledge - Lack of training in basic testing techniques Given these factors, we can now look at ways to assess Testing Effectiveness. TESTING EFFECTIVENESS ASSESSMENT A Testing Effectiveness Assessment is a process that can improve the Testing ROI and includes the following five steps: 1 Scrub the Requirements Tests can only be as good as the requirements they are intended to test. If requirements are vague and ambiguous, the tests will not be very effective. Therefore, the first step in the assessment process is a thorough review of the requirements. Based on this review, a Requirements Scrub may be necessary. For more information on writing requirements that are unambiguous, check out the Sept 2004 Food for Thought (http://www.swqual.com/newsletter/vol1/no1/vol1no1.html). 2 Update the Requirements Trace Matrix (RTM) Once the requirements have been scrubbed, the next step is to review and update the RTM. The RTM is an extremely important document. In its simplest form, it provides a way to determine if all requirements are tested. However, the RTM can do much more. In this example, the RTM is used as a test planning tool to help determine how many tests are required, what types of tests are required, whether tests can be automated or manual, and if any existing tests can be re-used. Using the RTM in this way helps ensure that the resulting tests are most effective. The RTM can serve many purposes over the course of a development project. Initially, it can be used as a planning tool (as illustrated above). Once the tests are developed and Validation Testing has begun, the RTM can be used to help determine the extent of regression test required based on the relationship be requirements, design, code, and tests as illustrated below. When it becomes necessary to perform regression testing, the accurate information included in the RTM will be invaluable in helping to select a reasonable set of tests to run. 3 Analyze the Existing Test Suite The RTM can also be used to help analyze the existing test suite. For example, an analysis of test types can be performed to determine if a variety of test types have been associated with each requirement. By using a wide variety of types of tests, the tests are likely to be more productive. Examples of Test Types Functional Algorithmic Positive Negative Usability Boundary Act Like a Customer Startup Shutdown Configuration Platform Load/Stress Security Performance Data Flow Safety-related Compatibility Documentation Timing Error Checking Power Failure Resources Installation Upgradeability Volume Scalability Throughput In addition to looking for a variety of test types, there are specific test strategies that are also important to identify. For example: - Data Flow Testing Data flow testing is a strategy based on known patterns of data usage and data flow. Data flow tests are often derived from data flow diagrams. - Equivalence Classes A group of tests can form an equivalence class if: - they all test the same thing - if one test passes, all tests will likely pass - if one test fails, all tests will likely fail Identifying potential equivalence classes allows redundant, non-productive tests to be identified. - Boundary Analysis By identifying ranges and boundaries, tests can be created that are: - well within boundary or range - one less than boundary - equal to boundary - one greater than boundary - well beyond boundary Boundary analysis ensures that all boundary conditions are covered. Again, this increases the likelihood that these tests are productive. - Act Like a Customer Testing Act Like a Customer (ALAC) testing is based on how users actually use your product. These tests are critical in helping to uncover the kinds of defects that users are likely to encounter. ALAC tests can be easily created from Use Case/Workflow diagrams. 4 Scrub the Test Suite The next step in the Testing Effectiveness process is to remove redundant, non-productive tests from the test suite. 5 Identify Additional Training for Testers Training for testers can help improve skills in areas such as: - Test Planning - Act Like a Customer Testing - Equivalence Classes - Data Flow Testing - Boundary Analysis - Requirements Management - Test Automation THE BOTTOM LINE So, while it may not be practical to measure Testing ROI, improving testing effectiveness will certainly improve the Testing ROI whether we can measure it or not. Till next time... -------------------------------------------------------------------------------- ***Monthly Morsel*** Every month in this space you’ll find additional information related to this month’s topic. References: [1] Jones, C., Software Quality: Analysis and Guidelines for Success, ITP, 1997 [2] Chernak, Y., “Validating and Improving Test-Case Effectiveness”, IEEE Software, January-February 2001. [3] Huber, J., “Efficiency and Effectiveness Measures To Help Guide the Business of Software Testing”, SM/ASM Conference, 1999. On-line Resources: - StickyMinds: Articles of interest to testing professionals (http://www.stickyminds.com/) - Software Quality Professional: ASQ Software Division’s quarterly journal (http://www.asq.org/pub/sqp/) - Better Software Magazine (http://www.stickyminds.com/BetterSoftware) Testing Resources: Some of my favorite books on testing: - Kaner, C., et. al, Testing Computer Software, 2nd ed, Thomson Computer Press, 1993 - Marick, B., The Craft of Software Testing, Prentice-Hall PTR, 1995 - Copeland, L., A Practitioner’s Guide to Software Test Design, Artech House, 2003 - Kit, E., Software Testing in the Real World, Addison-Wesley, 1995 -------------------------------------------------------------------------------- ***Calendar*** Every month, you’ll find news here about local and national events that are of interest to the software community ... - Software Quality Calendar There are many organizations that sponsor monthly meetings, workshops, and conferences of interest to software professionals. Find out what’s happening... (http://www.swqual.com/links/upcoming.html) - Workshops Offered by Software Quality Consulting Software Quality Consulting offers workshops in many topics related to software process improvement. Get more info... http://www.swqual.com/seminars/courses.html) -------------------------------------------------------------------------------- ***About SQC*** Software Quality Consulting provides consulting, training, and auditing services tailored to meet the specific needs of clients. We help clients fine-tune their software development processes and improve the quality of their software products. The overall goal is to help clients achieve Predictable Software Development(TM) – so that organizations can consistently deliver quality software with promised features in the promised timeframe. To learn more about how we can help your organization, visit our web site (http://www.swqual.com/?AboutSQC) or send us an email (info@swqual.com). -------------------------------------------------------------------------------- I hope this newsletter has been informative and helpful. Your comments and feedback are most welcome. Send me your feedback... (info@swqual.com) Thanks, Steve Rakitin info@swqual.com Food for Thought and Predictable Software Development are trademarks of Software Quality Consulting, Inc. Copyright © 2005. Software Quality Consulting, Inc. All rights reserved. Graphic design by Sage Studio