What’s Your Story?

Storytelling as a Critical Testing Skill

nce upon a time, there was a king who instructed his most trusted developers to build a new system for tracking taxes the king collected from the those who lived in his kingdom. The king had his Tax Collector prepare a Software Requirements Specification and hold a roundtable review. Questions were discussed and the developers started coding. To save some gold, the king decided to outsource the testing. He found a group of testers in a neighboring realm and issued a proclamation allowing these testers to work on this project from their realm. Testers would use carrier pigeons to report bugs back to the developers.

The developers worked hard and eventually delivered the application to the testers several fortnights later than planned. Testers started testing and a few problems were found. The carrier pigeons were bored most of the time. Finally, when the moon was full, the testers were instructed to stop testing and deliver the application to the Tax Collector. The Tax Collector was very demanding.

The Tax Collector tried to import existing tax rolls from his Excelsior 97 Spreadsheet into the new application – that didn’t work. He tried changing the number of goats owned by one of the peasants who lived in the realm – that didn’t work either. Needless to say, he found many problems and was not pleased.

The Tax Collector spoke to the king about the situation. The king was irate and asked the testers why they missed so many of these problems. The king soon discovered that the realm the testers were from (New Hamp Shire, I believe) had no taxes. The testers, the king surmised, were not aware of his tax rules:

• The king exacted additional taxes on Knights and Lords who lived within the realm but who owned land in other realms.

• Peasants were taxed at one rate on their land and at a different rates on the number of cows and goats they owned, the number of wagons they owned, and the amount of money they made selling their meager wares.

• Senior peasants could grovel before the king and ask for tax abatements. If the king was in a good mood that day, they would pay far less in taxes.

All of this was foreign to the testers, since in their realm, there were no such taxes. The testers were thus not able to imagine ways to test this software since they lacked realm-knowledge. All they could do was test against the Tax Collector’s incomplete requirements specification. Clearly, this was not sufficient.

Luckily, the king was able to find new testers from within his realm who were all too familiar with the vagaries of the king’s tax system. The new testers had first-hand knowledge about many different situations the Tax Collector would face. They even knew of some peasants who grazed their goats in New Hamp Shire to avoid the king’s taxes. Their knowledge was turned into tests and as a result, many more bugs were found. Eventually, the bugs were repaired and the Tax Collector was happy. The new testers however, were not, since the new Tax Collection software they just tested requires them to hand over more of their meager income to the king.

The moral -- in addition to basic testing skills, testers need the following:

• domain knowledge – testers need to be intimately familiar with how customers use the software they are testing…

• the ability to tell stories – stories that illustrate specific scenarios that need to be tested…

Let’s look at how domain knowledge and storytelling can help add value to your testing.

Domain Knowledge

To be effective as a tester, you need to have domain knowledge. To further illustrate this point, Brian Marick [1] tells the following (probably true) story:

A customer rents a car for a three-day business trip. Midway through the rental, he extends the rental for another week. This, by the way, gives him enough rental points to reach Preferred status. Several days later, he calls to report that the car has been stolen. He insists that the Preferred benefit of on-site replacement applies, even though he was not a Preferred customer at the start of the rental. The company agrees and a new car is delivered to him. Two days after that, he calls to report that the “stolen” car has been found. It turns out he’d forgotten where he’d parked it. He wants one of the cars picked up and the appropriate transaction closed. Oh, and one other thing, the way he discovered the missing car was by backing into it with its replacement, so they’re both damaged.

This story illustrates the importance of domain knowledge. Imagine if you had never rented a car in your life and were asked to test a rental car reservation system. You couldn’t possibly anticipate the kinds of problems in Brian’s story. As a result, those situations would not be tested and would only be found when a customer like Brian came along.

Testers who have had experiences like Brian’s, can express their domain knowledge in the form of realistic stories – stories that illustrate specific scenarios that need to be tested…

Storytelling

Brian’s car rental experience is an example of a “story” that can be used to identifyconditions in the rental car application that need to be tested [1]:

• Upgrading to Preferred status (during rental period)
• Extending a rental period
• Handling a “stolen” car
• On-site replacement
• Undoing an on-site replacement
• Undoing handling of reported “stolen” car
• Return of a damaged car

Using stories as the basis for developing tests is not a new idea. It’s been around for some time and has many different names. It’s been called scenario testing, Act Like a Customer (ALAC) testing, and most recently, soap operatesting [1].

So what are scenario-based testing, ALAC testing, and soap opera testing? And how are these methods different from the traditional approach of testing against the spec?

Scenario-based Testing

Cem Kaner [2] defines a scenario as:

“… a hypothetical story, used to help a person think through a complex problem or system. A scenario test is a test based on a scenario. The ideal scenario test has several characteristics:

• The test is based on a story about how the program is used, including information about the motivations of the people involved.

• The story is motivating . A stakeholder with influence would push to fix a program that failed this test. (Anyone affected by a program is a stakeholder. A person who can influence development decisions is a stakeholder with influence.)

• The story is credible . It not only could happen in the real world; stakeholders believe that something like it probably will happen.

• The story involves a complex use of the program or a complex environment or a complex set of data.

• The test results are easy to evaluate . This is valuable for all tests, but is especially important for scenarios because they are complex.”

Read more about ways to create good scenarios…

Act Like a Customer Testing™ (ALAC™)

Act Like a Customer Testing™ is based upon the following simple principles:

• All software has defects

• When customers use your software, they typically only find a very small percentage of the total number defects that are present

• Finding and fixing those defects customers are likely to find and ignoring the rest will result in very satisfied customers

One caveat, ALAC™ Testing is not appropriate for software that is life-threatening or life-supporting…

In addition to the three principles, ALAC™ requires testers to have domain knowledge and to have the ability to translate their domain knowledge into test cases. Often, when I’m helping testing groups improve effectiveness, the biggest obstacle I encounter is the lack of domain knowledge.

So how does one acquire domain knowledge? This is not easy and can’t be done in a short period of time. Here are some suggestions:

• Visit customers
  
  Many testers I’ve met have never visited or even talked to a real customer. Testers need to be given the opportunity to spend a day in your customer’s shoes. Testers need to experience first hand, your customer’s pain, to observe what they do with your software, and how that differs from the way in which they’ve been testing…

• Spend time (a lot of time) with your technical support team
  
  Testers need to become best friends with technical support. Testers should make it a point to take a support person to lunch at least once a month. This way, testers can find out what wacky things customers are doing with your software so that they can create stories, scenarios, and tests for them. I encourage testers to spend time working in support answering calls from customers. Again, this serves to help gain important first-hand experiences…

• Recruit testers from Technical Support
  
  It is easier to train someone who has extensive domain knowledge in good testing practices than it is to provide domain knowledge to someone who has good testing skills. The QA Group should be a possible career path for your Technical Support staff. Both groups should work closely with each other.

• Attend industry-specific conferences
  
  Industry-specific conferences are a good way to learn more about your industry and the people who work in the industry, your potential customers.

Like I said, acquiring domain knowledge is not easy and can’t happen overnight. But it is definitely worth the effort if you are serious about improving your effectiveness as a tester and if your company is serious about reducing the number of defects your customers find…

Soap Opera Testing

Hans Buwalda defines Soap Opera Testing as follows:

“As many readers know, soap operas are dramatic daytime television shows that were originally sponsored by soap vendors. They depict life in a way that viewers can relate to, but the situations portrayed are typically condensed and exaggerated. In one episode, more things happen to the characters than most of us will experience in a lifetime. Opinions may differ about whether soap operas are fun to watch, but it must be great fun to write them. Soap opera testing is similar to a soap opera in that tests are based on real life, exaggerated, and condensed.” [1]

Buwalda devised this approach by inviting people who had extensive domain knowledge to work with testers in small teams to create stories based on extreme or exaggerate situations. By working together, the testers were able to create several stories, which were then turned into tests using the scenario-based testing approach described above.

Testing Against the Spec…

Testing against the spec is an important kind of testing to do. However, for most complex applications, it’s never sufficient. You always need to do scenario testing, Act Like a Customer Testing™, Soap Opera Testing and even some lightly scripted exploratory testing if you are truly concerned with delivering value to your customers.

Buwalda [1] illustrates the major difference between what he calls “mechanical testing” (testing against the spec) and Soap Opera Testing (using stories, scenarios, and ALAC™).

The mechanical tests are often derived solely from one requirement in the Software Requirements Spec (SRS) or Functional Spec. This results in a one-to-one mapping between requirements and tests. This is good because it represents the place to start. And, for those test teams that have testers with little or no domain knowledge, this is the kind of testing that those folks can do. Figure 1 below illustrates the mapping between requirements (or objectives) and tests when testers are focused on “testing against the spec”.

from [1]

When testers use stories, soap operas, scenarios, ALAC™, and exploratory testing techniques, they create many-to-many relationships between requirements and tests, as shown in Figure 2. This helps uncover problems that would clearly be missed if we only did the mechanical type of testing against the spec.

from [1]

At the end of the day, testers need to write test cases…

Whether you’re testing against the spec or doing scenario-based testing using stories, ALAC™ tests, or soap operas, at the end of the day, testers need to write test cases. So what is a test case? Well, like most things in software engineering, there is no universally accepted definition of what constitutes a “test case”.

Cem Kaner has a very insightful definition. He says:

“In my view, a test case is a question that you ask of the program. The point of running the test is to gain information, for example whether the program will pass or fail the test.

An important implication of defining a test case as a question is that a test case must be reasonably capable of revealing information .” [3]

Brian Marick uses another term to describe a “lightly documented test case.” He calls this a test idea .

“A test idea is a brief statement of something that should be tested. For example, if you're testing a square root function, one idea for a test would be ‘test a number less than zero’. The idea is to check if the code handles an error case.” [4]

Read more about test case definitions…

We also need a definition of what constitutes a “good” test case. I often refer to test cases as being either “productive” or “non-productive”. I define these terms as follows:

• a “productive” test case is a test that has a high probability of finding defects

• a “non-productive” test case has a very low probability of finding defects

Given these definitions, we would want testers to write as many “productive” test cases as possible. Since test suites are rarely reviewed, many companies have very large test suites that contain mostly non-productive tests.

How can you tell if your scenario tests are productive or not? One way is to review them with your peers and ask the question, is this test likely to find a defect that a customer would find?

Another way is to use Caper Jones’ defect removal efficiency metric. This metric measures how good a job your testers are doing at Acting Like a Customer when they test…

Read more about defect-removal efficiency metric…

The higher the defect removal efficiency metric, the better your test cases are at finding those defects that customers are likely to find.

So what’s your story?

To improve effectiveness, testers need domain knowledge and storytelling skills. Testers can acquire domain knowledge in several ways. Storytelling skills, well that’s another story…

Have you heard any good stories lately? More importantly, have you told any good stories lately?

Pay it Forward

If you find this newsletter of value, please consider the following:

Norm Kerth is a highly respected consultant who developed the Project Retrospective techniques discussed in the July-Aug newsletter. He was in a serious car accident and suffered a disabling brain injury. As a result, he cannot work and lives on a very limited income. You can help recognize his contribution to our industry by sending a small donation. Checks can be made payable to Norm Kerth Benefit Fund and sent to Norm Kerth Benefit Fund c/o Process Impact, 11491 SE 119th Drive, Clackamas, OR 97015-8778. You can also visit Karl Weiger’s website (Process Impact) for more details about contributing to the fund. Thanks.