Software Verification and Validation
for Practitioners and Managers
|
We never have the time to do it right but always have time to do it over and over. -Anonymous- This theme is unfortunately common for many companies developing software. While this mode of operation may have been adequate in the past, as the saying goes, "the times they are a changin’ ". Those organizations who are unable or unwilling to learn how to "do it right the first time" are going to be left in the dust. Software development in the 21st century is changing dramatically. New paradigms such as "Extreme Programming" [1] and developing software on "Internet time" [2] and putting increasing pressure on organizations to get better software developed faster with fewer resources. Do more with less. Software development organizations are developing more products that are more complex and more difficult to develop. To staff up for new projects, organizations are competing, fiercely at times, for a limited pool of experienced people. The tight labor market for software developers and QA professionals is expected to continue well into the first decade of the 21st century since the demand for qualified people is expected to far exceed available resources. [3], [4]. Doing more with less means that there are fewer people available for testing. So the testing that is done must be that much more effective. Do it faster. As a result of the Global Economy, many organizations are now facing competition from new players in other parts of the world. These new players are developing products faster and at lower cost. Further, the ability to develop products on "Internet Time" is affecting all businesses, not just those that are developing software for the Web. Product development, testing, and release cycles that once took years are being compressed to months and sometimes weeks. So, not only do organizations have to Do more with less, they also have to Do it faster. Doing it faster means that the verification and validation activities have to be completed in less time. Do it with higher Quality. Customers are becoming more demanding and more selective in deciding which software products to buy. Having products with great features that are unreliable is becoming less and less attractive when compared to products that have the features customers actually need and are very reliable. Compressed schedules, understaffed projects, coupled with the need to release very reliable products all result in extraordinary pressures on people within development organizations. Organizations need a way to cope with these conflicting demands. Doing more with less, Doing it faster and with higher Quality all mean that there is little room for error. More than ever before, organizations must learn how to "get it right the first time". From a validation testing perspective, the focus has to be on finding and fixing those defects that customers are likely to find. A Predictable Development Process is crucial to getting it right the first time. The goal of Predictable Software Development is simply: To delight your customers by consistently delivering what you promised when you promised it. Now let’s look at why this is so important. 12.1 INTRODUCTION TO PREDICTABLE SOFTWARE DEVELOPMENT For every software project, Management wants to know:
These are legitimate questions for Management to ask. Unfortunately, when an organization is unpredictable, it is very difficult to provide accurate answers to these questions. Why? Well, in order to know with certainty when the product will be released requires, among other things:
New projects begin with great optimism – a naive expectation that somehow this project is going to be different, that it will be successful. For some reason, Management expects that the outcome for this project will be different, even though the organization is using the same unpredictable processes that resulted in failure on earlier projects. After the initial euphoria ends, the hard reality sets in. The Project Team quickly recognizes that the same unpredictable processes used on the last unsuccessful project are being applied yet again to this new project. Lessons learned from the previous failure have not resulted in any changes to the process because there was no Management commitment to improve. Managers and Executives need to understand that having a predictable software development process is vitally important to the long-term success of their business. Many software development organizations lack discipline. These organizations either do not have or have but do not follow, a written software development process. As a result, they are not able to accurately predict when key events (such as code complete, test complete, product release, etc.) will occur. What many companies fail to recognize is that other parts of the organization need to know when things will happen so that:
As a result of the inability to predict when things will happen, many software development organizations suffer from a lack of credibility. No one believes dates from the Development group because they’ve never met a date. As a result, Management frequently sets (or allows to be set) unreachable release dates for products and makes (or allows to be made) unreasonable commitments to customers. Further, the unreachable release dates and unreasonable commitments are usually provided to customers or potential customers to help sell product. As you might expect, the dates set by Management are also rarely met. This creates a lose-lose situation - customers lose since their plans may be predicated on your unrealistic schedules and unreasonable commitments, and your employees lose since no one wants to be associated with a project that fails. On projects with unrealistic schedules, the schedule can only be reduced so much (the time required to get the minimum feature set coded and tested). Developers, being eternal optimists, fail to anticipate problems and coding frequently takes longer than expected. QA engineers fail to accurately estimate how many tests are needed and how long it will take to write and execute them. This problem occurs because most people have never been trained in how to develop accurate estimates and build realistic schedules. When development time expands, time for software verification and validation activities is reduced (usually because someone made a commitment to a key customer to ship on a certain date). When software verification and validation tasks are cut from projects, the organization will find fewer bugs and customers will find more bugs. This makes the QA staff frustrated and customers unhappy. It also means the organization will need to do unplanned bug fix releases. Simply stated, the more predictable an organization is, the more likely the software verification and validation activities discussed in Parts I-III will be performed. When those activities are performed, they will significantly increase the ability of the organization to find and fix bugs before the software is shipped to customers, thereby decreasing the need for unplanned, expensive bug fix releases. Lack of predictability impacts your bottom line. Unplanned bug fix releases represent a significant cost to the organization as shown in Figure 12-1. Management determines how to use scarce and expensive resources. You can decide to use these resources to deliver bug fix releases, which typically don’t generate any revenue or you can decide to use those scarce resources to work on new features and new products, which do generate revenue. The choice is yours to make. Figure 12-1. Bug Fix Releases are not Free! 
Lack of predictability negatively impacts customers. In unpredictable organizations, customers are unsure of when new products and updates to existing products will be released. This makes it hard for customers to plan to migrate to new software releases. Further, since unpredictable organizations are unable to develop accurate schedules, they tend to release software with far too many bugs, adding to customer dissatisfaction. Lack of predictability negatively impacts employees. No one wants to be associated with projects that fail. In unpredictable organizations, failed projects are the norm. And experience has shown that there is a very strong correlation between customer satisfaction and employee satisfaction. Predictable Software Development can be achieved when Management takes the lead to change the behavior of the organization. Management needs to be focused on the elements identified in Figure 12-2. Figure 12-2. The Elements of Predictable Software Development 
To become predictable, organizations need to learn how to balance Quality, Features, and Schedule. While tradeoffs are made all the time, organizations need to understand and assess the implications of these tradeoffs. Further, organizations also need to learn how to balance the needs of People, Process, and Product. Tradeoffs here affect productivity, customer and employee satisfaction. Also required is the ability to manage commitments and to manage risks. Managing commitments (internal and especially external) is essential so that the organization can consistently exceed commitments. Many complex software projects are fraught with risks. Risks on many software projects may be tacitly understood but all too often, are not actively managed. Effective risk management skills can make the difference between success and failure. 12.2 CHARACTERISTICS OF UNPREDICTABLE ORGANIZATIONS How can you tell if your organization is unpredictable? Listed below are some characteristics of unpredictable organizations.
From working with dozens of companies, I’ve identified several root causes of unpredictable behavior. These include:
Unrealistic schedules can result from several causes including: lack of training in estimating and scheduling, allowing other organizations to set development and validation schedules, not keeping or using information from past projects to help develop more accurate estimates, etc.
Software Project Management is a difficult and unrewarding job. Project managers frequently become scapegoats for failed projects. Good project managers are rare and worth their weight in gold. Without a doubt, one of the most frequent reasons that projects fail is due to poor project management. In many cases, the root cause of this problem is not the project manager, but rather, how Management measures the project manager's performance. Usually, project managers are measured on their ability to get products released. Therefore, they will do whatever it takes to get product released including cutting features and reducing quality.
For many organizations, working in "fire fighting" mode is the norm. These organizations move from one crisis to the next. How anything gets done is a real mystery. What we should have learned by now is that working from crisis to crisis is not the most effective way to use scarce, expensive resources. This mode of working frequently leads to burnout and frustration.
In many organizations, Management's goals and objectives are not aligned with the individual performance goals and objectives of the staff. For example, in organizations where Management complains about poor product quality, you would likely not find any mention of the word "quality" in the performance plans for the staff. Rather than encouraging the behavior that is desired, Management knowingly or unknowingly does the opposite. For example, Management inadvertently encourages "fire-fighting" behavior by rewarding "heroes" who resolve the "crisis-du-jour". In fact, in organizations that work in "fire fighting" mode most of the time, in many instances, it is these so-called "heroes" that frequently cause the crises in the first place.
Many organizations are unable to measure the amount of effort required to develop, document, and test a software release. Some organizations are unable to answer basic questions like: how big is the product (using whatever size metric you want), how long did it take to develop and release it, and how many bugs were found? 12.3 CHARACTERISTICS OF PREDICTBLE ORGANIZATIONS Listed below are some characteristics of predictable organizations:
These topics will be discussed further in subsequent chapters.
12.4 MANAGEMENT CAN CHANGE THE ORGANIZATION Being a parent who has helped raised two children, I learned the value of positive reinforcement - rewarding and recognizing good behavior and providing negative incentives for bad behavior. I adapted these principles to managing technical people and found that they worked just as well with adults as they did with children. Management has the ability to change the organization. Management controls the organization’s human resources, determine how resources are allocated to projects, and most importantly, determine how people are evaluated and measured. What Management often fails to recognize however is that to change the culture, you need to change the way people behave. In most organizations, the way people behave is directly related to how they are measured. This is particularly true for software engineering organizations where technical challenge and peer recognition are very important. The notion that Management can change the culture of an organization by changing the way people are measured is not new. It’s been known from many years [6], [7]. Unfortunately, in many organizations, Management hasn't recognized this fact. The following are some specific actions Management can take to help create an organization that behaves in a more predictable manner:
Most organizations have overall corporate goals such as increase market share, improve quality and customer satisfaction, etc. Frequently, Management fails to define specific goals and objectives for individuals that are based on the corporate goals. Setting individual goals and objectives is vitally important since we know from experience that people will do those things that they are being measured on. A client having "quality problems" asked me to come up with recommendations for improvements. I asked to see the Performance Plans for the software engineering staff. Not one person had "improve code quality" or "reduce defects" as an objective. My recommendation was to incorporate such objectives into the Performance Plans. Over time, the "quality problem" disappeared.
The literature is full of horror stories about project failures, cost overruns and missed schedules [8]. The ability of an organization to define an accurate and realistic schedule is critical. Yet, few organizations know how to do this. Why? Well, there are probably many reasons but what I’ve observed is that most people have never been trained, while in school or on the job, in how to accurately estimate a task, build an accurate schedule, and then meet that schedule. Even in those organizations that conduct project "post-mortems", there is little or no improvement in the organization’s ability to develop accurate schedules that can be met. Skills required to develop accurate estimates and build realistic schedules are discussed in Chapter 14.
One of the reasons organizations are unpredictable is that they either don’t have a documented software development process, or don’t follow the process they do have. We know from experience that to successfully develop complex products you need to rely on a proven process. This is as true for software just as it is for any complex product. For software, the process must address both development activities and verification and validation activities, as described in Chapter 3 and Appendices G and H. Without a documented process, people will spend a significant amount of time arguing about what to do. It will be harder to develop accurate estimates and schedules. Groups like Software QA and Documentation that need to have specific information will not know if or when that information will be available. Further compounding the problem is the claim from some software engineers that having to follow a process will diminish their ability to be creative. Nothing could be further from the truth. Ask a hardware engineer if following a process diminishes their ability to be creative. It doesn’t. I’ve found that software engineers who resist following a process do so out of fear that they will actually be held accountable to do something (like write a Design Specification) when they would rather spend all of their time writing code. More on this topic in Chapter 15.
Accountability is a concept that is totally foreign to many in the software industry. However, it is essential for those organizations that want to become predictable. Everyone in the organization, from the CEO on down, needs to be held accountable for doing his or her job. On a project team, people need to be held accountable for meeting their schedule, assuming of course that the people doing the work set their schedules. Development should be held accountable for delivering what was promised, not just to the external customers but to internal customers (such as Software QA and Technical Documentation) as well. Software QA should be held accountable for writing and executing tests based on the software requirements spec (SRS) and for performing the testing as defined in the schedule. Management needs to give people the responsibility for defining what they will do and when they will do it by. Once this happens, people need to expect to be held accountable for delivering what they promised. When people miss their commitments, Managers need to understand the reason and determine how to get things back on track by working collaboratively and cooperatively with everyone involved. Note that accountability extends to Managers as well. Oftentimes Management is responsible for providing resources, buying equipment and tools, etc. that project teams need to meet their commitments. More on this topic in Chapter 16.
Most software development projects are fraught with risk. Yet few project managers know how to proactively manage risk. On projects where risk management is ignored, I frequently see several "re-planning" activities. Re-planning is often just a euphemism for something happened that wasn’t expected but that could have been avoided. Many times these "re-planning" activities can be prevented if proactive risk management is used from the outset. Proactively managing risk not only helps ensure that Development and Software QA will meet their schedule, it increases the likelihood that the project will be successful. More on Risk Management in Chapter 16.
In many organizations, sales people frequently make unrealistic commitments to customers in order to sell product. This often results in undue pressure on project teams to deliver. When they can’t deliver (because the commitments were unrealistic), customers become unhappy and may look elsewhere. Meanwhile, the sales person has received their commission check and is off with other customers making more unrealistic commitments. The underlying problem here is an organizational one. There’s a disconnect between Sales and Development. And it all goes back to how Sales people are measured. In many organizations, sales people are measured on the dollar amount of sales they book not on meeting commitments to customers. In fact, some customers have finally smartened up and have signed agreements that have penalty clauses in them based on agreed to delivery dates and feature sets. Surprisingly, even with these financial incentives in place, Management still has failed to hold sales people accountable. Management needs to manage commitments made to external and internal customers. One way to do this is to set expectations lower. By doing this, the organization has a much better chance of meeting or exceeding expectations. All too often, organizations set expectations unrealistically high and frequently fail to meet them. As a customer, we are generally very satisfied when we receive exactly was we expect. If we receive more, we are very happy. If we receive less, we are usually not happy. Therefore, Management should encourage the organization to under-commit and over-deliver. In addition, people who make commitments to customers need to be held accountable for meeting those commitments. The same is true for commitments made to internal customers. Managing commitments is discussed in more detail in Chapter 16.
It goes without saying that in order to become more predictable, organizations need to measure what happens. A few simple measures that are tied to the overall corporate goals should be sufficient. For example: What is the average amount of schedule slippage on projects? How many known defects are products being shipped with? How many unplanned bug fix releases were there last year? Management must recognize that they have the ability to change the behavior of their organizations. By learning how to do this, Management can significantly improve their organization's predictability. Measurement is discussed in Chapters 7, 10, 13, and 14. SUMMARY A Predictable Software Development organization has a significant competitive advantage over their unpredictable competitors. This point is become even more important in the face of a tight labor market. And, on this issue, Management must take responsibility. As observed by Jones [5]: "One of the most important topics in the entire software quality domain is the relationship between software quality and software project management. Indeed, Deming, Juran, and Crosby have asserted that the main source of quality problems can be assigned to management rather than to technical workers." Management must provide leadership to help organizations behave in a more predictable manner. Management must understand that they can change how people behave by changing the way people are measured. If you want to improve schedule accuracy, measure people on their ability to develop accurate schedules and then meet them. If you want to improve product quality, measure people on product quality. If you want to improve customer satisfaction, reward people based on achieving improvements in customer satisfaction. Don’t believe that this works? Jay Bertelli, CEO of Mercury Computer in Chelmsford, Mass. challenged his Management Team in January 1999 to double the company’s stock price by the end of the year. As incentive to meet the goal, each executive would get a new Porsche. By the end of 1999, the company’s stock price had tripled. Bertelli had 20 Porsche’s delivered to the company’s headquarters – one for each executive and two loaners for top performers among the staff. [9] REFERENCES [1] Beck, K., "Embracing Change with Extreme Programming", IEEE Computer, October 1999, p. 70-78. [2] Cusumano, M. A., "Software Development on Internet Time", IEEE Computer, October 1999, p. 60-70. [3] Schafer, M., "Hiring for Keeps (or at least for a while)", Software Magazine, April/May 2000. [4] Wilkinson, S., "Hired Guns – Beating the IT Staffing Shortage with Contract Workers", Datamation, May 1999. [5] Jones, C., Software Quality: Analysis and Guidelines for Success, ITP, 1997. [6] Weinberg, G. M., The Psychology of Computer Programming: Silver Anniversary Edition, Dorset House, 1998. [7] Lister, T. and DeMarco, T., Peopleware: Productive Projects and Teams, 2nd edition, Dorset House, 1999. [8] Yourdon, E., Death March: The Complete Software Developer’s Guide to Surviving ‘Mission Impossible’ Projects, Prentice-Hall, 1999. [9] Boston Globe, Business Section, April 27, 2000, page C4. |
For further information,
call Steve Rakitin at 508.529.4282
or e-mail him at
steve@swqual.com
| Home |
Company Info |
Contact Info |
Food for Thought and Predictable Software Development are trademarks of Software Quality Consulting, Inc.
Updated January 2008
Copyright ©2008 Software Quality Consulting, Inc. All rights reserved.