Design for Testability

Many software quality problems are the result of poorly written requirements. We know that 40-60 percent of all software defects are caused by errors in requirements. [1] What this means is that on a typical product development cycle that has say, 500 defects - as many as 300 are caused by poorly written requirements.

While most companies recognize the importance of having requirements, many haven’t recognized the importance of having good requirements - requirements that are well-written, unambiguous, and testable. Since so many defects are caused by poorly written requirements, companies should be motivated to improve the quality of their requirements…

What can be done to improve requirements?

There are a couple of ways to do this:

• First, provide training for those people who typically write requirements. We can’t expect people to do a good job of writing requirements if they haven’t been trained in what that means.
  
  Read more about writing good requirements…
  
  Check the Monthly Morsels section for references on requirements…

• Second, conduct effective peer reviews of requirements to increase the likelihood of finding requirements problems early. Some training may be required here as well since most people have not been taught how to perform effective peer reviews on requirements documents. For example, when reviewing requirements, one question that should be asked is “Is every requirement testable?”
  
  Read more about peer reviews…
  
  Check the Monthly Morsels section for additional references on peer reviews…

• And third, introduce Design for Testability concepts into the organization...

What is Design for Testability?

Design for Testability (DFT) is a concept developed by electrical engineers to improve the designs of circuit boards so that these boards could be tested using conventional automated testing equipment (ATE). By bringing certain test points out to the edge, the test engineer writing the ATE program is able to test more of the functions the board is intended to perform, thus lowering the failure rate in the field and improving overall quality.

Wikipedia defines Design for Testability as:

Tests are applied at several steps in the hardware manufacturing flow and, for certain products, may also be used for hardware maintenance in the customer’s environment. The tests generally are driven by test programs that execute in Automatic Test Equipment (ATE) or, in the case of system maintenance, inside the assembled system itself. In addition to finding and indicating the presence of defects (i.e., the test fails), tests may be able to log diagnostic information about the nature of the encountered test fails. The diagnostic information can be used to locate the source of the failure.

DFT principles long used by hardware engineers can be applied to both software design and software testing…

What is software testability?

James Bach [4] and David Gelperin [5] define testability as anything that makes software easier to test by making it easier to design and execute tests.

Bach [4] describes software testability in terms of:

• Control. The better we can control it, the more the testing can be automated and optimized.

• Visibility. What we see is what we test.

• Operability. The better it works, the more efficiently it can be tested.

• Simplicity. The less there is to test, the more quickly we can test it.

• Understandability. The more information we have, the smarter we test.

• Suitability. The more we know about the intended use of the software, the better we can organize our testing to find important bugs.

• Stability. The fewer the changes, the fewer the disruptions to testing.

Testability, as described by Bret Pettichord, has more of a design component to it:

“Testability is a design issue and needs to be addressed with the design of the rest of the system. Projects that defer testing to the later stages of a project will find their programmers unwilling to consider testability changes by the time testers actually roll onto the project and start making suggestions. Testability takes cooperation, appreciation and a team commitment to reliability.” [2]

Improving software testability is clearly an important objective in order to reduce the number of defects that result both from poorly written requirements as well as poorly designed software. Let’s look at how we can apply the principles of DFT to software testing…

Applying DFT principles to software testing

Design for Testability principles can be adapted for software testing in at least two ways. First, we can use DFT principles to help clarify requirements that may be poorly written so that they can be more easily tested. Second, we can use DFT principles to improve the testability when using automated testing tools.

Let’s look at what this means…

DFT for Manual Testing

Poorly written requirements create problems for developers and testers. Not only can such requirements be difficult to understand, they can be interpreted in several ways. Rewriting requirements to remove ambiguity often isn’t done for a variety of reasons – there’s no time, the author doesn’t see the value, etc. So the tester is left to guess - How did the developer interpret this? How should I interpret this? Getting clarification from the author of the requirements document often is an exercise in frustration. Asking the developer how they interpreted the requirement can often raise more questions than it resolves. There’s a better way…

Testers can rewrite ambiguous requirements – using alternative techniques - so that the requirements become clear and more importantly, testable. After rewriting an ambiguous requirement, the tester can tactfully present the rewritten requirement to both the author - like this…

“I had difficulty understanding this section of the requirements the way it was written so I rewrote it like this. Is this what you meant?”

Once the author agrees to the rewritten requirement, the tester can share this with the developer.

Some techniques for rewriting requirements to improve testability include using:

• Work Flow Diagrams
• Flowcharts
• Structured English
• Truth Tables

Flowcharts

Work flow diagrams and flowcharts are excellent tools for expressing requirements in a way that leads to better understanding.

Expressing portions of the requirements using a flowchart enables testers to see all of the paths that need to be tested. This technique also makes it easy to see if all paths were included in the English text. Frequently, rewriting requirements using a flowchart helps identify missing information. Here’s a simple example:

As a tester, having the requirements expressed in a flowchart makes it much easier to visualize tests that are required. Using a flowchart in this example also makes it easier to ensure that all paths are defined, including error handling conditions – something that is easy to overlook in English.

Testers can also use colored highlighters to mark the paths covered by each test, ensuring that all paths are in fact covered.

For an example of rewriting requirements using Structured English, see the September 2004 newsletter...

Truth Tables

Here is an interesting example of comparing the testability of requirements written in English and the same requirements expressed in a truth table. Truth tables are useful when dealing with several discrete variables that are related.

Which is easier to understand, implement, and test?

Now here’s the same requirements rewritten using a truth table.

Messages:

1. “Password successfully changed”
2. “The password entered does not conform to the format specified by your sys admin. Enter a valid password.”
3. “New Password and Confirm New Password entries do not match. Try again.”
4. “The password you entered has been used recently. Try again.”
5. “The Old Password you entered is invalid. Try again.”

Not only is the truth table easier to understand than the text, each row of the truth table represents a set of conditions that need to be tested.

DFT for Automated Testing

The principle of rewriting requirements to remove ambiguity applies regardless of whether you’re doing manual or automated testing.

Applying DFT principles is a bit different for automated testing. Brian LeSuer [3] says that applications need to be designed for automated testing. Specifically, he recommends that application developers follow these rules to help improve the testability of the application:

• Expose application data
• Choose automation-friendly third-party controls
• Uniquely name application pages and objects
• Add “hidden” controls
• Use standard objects
• Modify custom controls
• Externalize functions in an API
• Use proper naming conventions
• Use only standard widgets
• Expose “hidden” controls
• Use unique page names

Bret Pettichord [2] says that testability is a critical aspect of successful test automation efforts:

• Automation requires testability. Whether testing via an API or a GUI, some accommodations to testability are necessary for successful automation. If you’re going to have to do it, why not plan for it up front?

• Successful automation requires a focus on testability. Again, regardless of approach, the greater the emphasis on testability within the development team, the more likely automated testing will successful.

• Test automation always affects test veracity to some degree. Automating tests always makes them somewhat different from actual customer usage. Wise testers will understand these deviations and develop compensating testing strategies, rather than act as though it can be eliminated.

Bottom Line…

Improving testability affects requirements and design. Rewriting ambiguous requirements using alternative techniques leads to requirements that are less ambiguous, more understandable and as a result, more testable. When the requirements are more testable, your customers will find fewer bugs.

For test automation to be successful, testers and developers need a common understanding of what is required in order to make most effective use of the test automation tools.