Agile Methods - Beyond the Hype

Agile Methods came to the forefront with the publication of the so-called Agile Manifesto [1] in 2001. Not since Dijkstra’s [2] classic paper “Go To Statement Considered Harmful” in 1968 where he stated “… the go to statement should be abolished from all ‘higher level’ languages…” has there been such an impassioned debate about software development methodology. There has been so much hoopla about the perceived benefits of “lightweight” vs. “heavyweight” development processes that camps have formed on both sides of this issue, each with their own zealots and their own books…

I distinctly remember reading the “manifesto” in 2001. It was the first technical article I ever read that elicited a visceral reaction. It made me mad. My reaction was something like this:

“What were these guys doing? We have been working so hard to improve the perception of software engineering and instill discipline in developers and managers. And then this… Are they nuts?

And why would anyone use the word manifesto in an article on software development?”

The term “manifesto” evoked visions of the “Communist Manifesto” [3] written in 1848 by Karl Marx and Frederick Engels – the acknowledged beginning of the Communist Movement. Here is a short snippet from an introduction by G. S. Jones [3]:

“… the Manifesto sketches a vision of reality that, at the start of a new millennium and against a backdrop of endless chatter about globalization and deregulation, looks as powerful and contemporary a picture of our own world as it might have appeared to those reading it in 1848.”

Wow. I immediately started writing a letter to the editor of IEEE Computer. Something I had never done before. My frustration was evident in my letter:

“Please excuse my cynicism. But I’ve been waiting a long time for software engineering to become a respected engineering discipline. While progress has been made, it has been painfully slow and frequently impeded by people within the software community.” [4]

When the December 2001 issue of IEEE Computer, I was surprised to see my letter was selected for publication. Good, I thought, someone needs to speak up for process! I received a few e-mails in response to the letter, all supporting my views.

The Four Values

Agile Methods are a collection of several methodologies that include:

• Extreme Programming (XP)
• Crystal
• Scrum
• Dynamic Systems Development
• Adaptive Software Development
• Lean Software Development

While each is different, they all share common values as expressed by the “manifesto”. To refresh your memory, the essence of the “manifesto” are the four values:

“We are uncovering better ways of developing software by doing it and helping others do it. Through this work we have come to value:

• individuals and interactions over processes and tools,
• working software over comprehensive documentation,
• customer collaboration over contract negotiation,
• responding to change over following a plan.

That is, while there is value in the items on the right, we value the items on the left more.” [1]

In the subsequent “endless chatter” about Agile Methods, I unwittingly became the poster boy for the process camp. I was mentioned in an IEEE Computer article written by Barry Boehm and Tom DeMarco: [6]

“Those on the pro side—Jim Highsmith and others—argued that the current approaches are broken and should be replaced. Those on the con side—Steven Rakitin, for example—said that agile approaches are just hacking by another name and that we shouldn’t abandon our disciplined processes now that we’re finally getting them right.”

And a sidebar to another IEEE Computer article written by Barry Boehm [5] stated that:

“Although their advocates have used each of these agile methods successfully in practice, critics who prefer process-based methods remain skeptical. In a letter [4] to Computer, Steven Rakitin indicates that, in his experience, the items on the right are essential, while those on the left serve only as easy excuses for hackers to keep on irresponsibly throwing code together with no regard for engineering discipline. He provides ‘hacker interpretations’ that turn agile value propositions such as ‘responding to change over following a plan’ into chaos generators. Rakitin’s hacker interpretation of ‘responding to change over following a plan’ is roughly ‘Great! Now I have a reason to avoid planning and to just code up whatever comes next.’ ”

So much for my five minutes of fame.

A short history lesson…

The publication of Dijkstra’s article on go to statements [2] in 1968 led to what has come be known in software engineering lore as the Methodology Wars. (See [7]). People like Ed Coad, James Martin, Ed Yourdon, Tom DeMarco, Michael Jackson (no, not that Michael Jackson), and others developed and then began pushing their own software development “methodology”. Each of these gurus had their own cult following and detractors.

They all promised, “if only those lazy, good-for-nothing programmers would adopt their methodology, all of our problems would be solved”! (Does any of this sound familiar?)

The advent of “structured methodologies” gradually evolved into Object-Oriented (OO) methods, mostly due to the work of the infamous Three Amigos (Grady Booch, Ivar Jacobson, and James Rumbaugh).

So, to continue our brief history, OO begat UML which begat RUP (Rational Unified Process), which begat lots of money for Rational from books, Case Tools, and training. For a more complete view of the history of software development, visit Wikipedia.

The timeline below is my attempt at putting the Methodology Wars in perspective…

Software Development Methodology Timeline

As you can see, many methodologies have come and gone over the past three decades. Having lived through all of these, I’ve become a bit cynical when yet another methodology is purported to be the Silver Bullet because we should remember what Fred Brooks said – there is no Silver Bullet. [8]

In looking back over the past three decades, I believe there have been cases where each methodology shown above has performed well and cases where each has failed miserably. This is true for every methodology from Agile Methods to CMM and everything in between.

Why has there been so much hype?

One of the things I believe the Agile zealots learned quickly was that there’s a lot of money to be made as a result of publishing books on methodologies. Today, amazon.com lists at least 27 books with “Agile Methods” in the title! Anyone remotely affiliated with an Agile project it seems has written a book on the virtues of Agile Methods.

The C3 project at Chrysler was the first large project where many Agile principles and methods were applied collectively and intentionally. The small team working on C3 included Kent Beck, Ron Jeffries, Chet Hendrickson and others. What is not so well known is that the project was a failure.

In spite of the fact that the project failed to deliver what the customer expected…

“By the time C3 was cancelled, XP was well on its way to the phenomenon that it is today… Beck, Jeffries, Hendrickson and others had book contracts in hand.” [9]

I am constantly amazed by the ability of those of us in the software industry to put positive spins on projects that would otherwise be viewed as dismal failures.

Consumer advocates constantly remind us that:

If it sounds too good to be true, it probably is.

This applies to telephone solicitation schemes as well as software development methodologies.

My bottom line…

Agile Methods can work given a specific set of circumstances. In my opinion, those circumstances are projects where:

• software is being developed for use solely within the company
• a real, live “customer” (end user) is available to the project team 24x7
• the risk of failure is low
• the project team is highly motivated and led by an excellent leader
• long term maintenance of software is not a concern

Under these circumstances, I believe Agile Methods may work well. But so will other methods. And this is the source of my frustration with all of the hype about Agile Methods. Agile Methods can be effective but so can other methods.

For projects that don’t meet the circumstances above, read what Roger Pressman observed:

“I contend that software engineering principles always work. It’s never inappropriate to stress solid problem solving, good design, and thorough testing (not to mention the control of change, an emphasis on quality,...). A specific software process might fail because it is overkill, or the work products it requires are unnecessary or burdensome, or a person or team becomes overly dogmatic in the application of the process. But history is on the side of a solid engineering approach.” [10]

And for projects that are high-risk or life-threatening, read what Mark Paulk said:

“Should organizations use XP, as published, for life-critical or high-reliability systems? Probably not. XP’s lack of design documentation and de-emphasis on architecture are risky.” [11]

Those Agile Methods like XP that rely on verbal communications rather than written documents appeals to some people. The people this doesn’t appeal to are sustaining engineers. I pity the poor slobs who have to maintain applications developed using Agile Methods. With no design documentation or requirements to refer to, you’re left with only the code. I contend that even incomplete or outdated design documentation is better than no documentation. How many times have you seen cases where maintenance is so difficult because of a lack of design documentation that we decide it would be easier to start over than to figure out how the existing code works?

Like the CMM, many Agile Methods include practices that can be applied regardless of methodology. For example, test first design is an XP practice. Test first design is definitely a good thing to do whether you are using an Agile Method or not.

Deciding what methodology to use is a difficult task. Before committing to a specific method, become as informed as you can on both the advantages and disadvantages. Remember, there is no silver bullet. Developing complex software is a difficult and challenging task. There are many pitfalls along the way. Use what will work best for the situation you are in. Leverage the skills of the people that will be working on the project. And most of all, do what makes sense.

That’s my story and I’m sticking to it…

This issue was inspired by a spirited debate titled “Agile Methods Considered Harmful?” held at a monthly meeting of the Software Quality Group of New England between myself and my friend and colleague Scott Andersen.

Pay it Forward

If you find this newsletter of value, please consider the following:

Norm Kerth, a highly respected consultant, suffered a disabling brain injury in an automobile accident. As a result, he cannot work and lives on a limited income. You can help him by sending a donation. Checks can be made payable to Norm Kerth Benefit Fund and sent to Norm Kerth Benefit Fund c/o Process Impact, 11491 SE 119th Drive, Clackamas, OR 97015-8778. You can also visit Karl Weiger’s website (Process Impact) for more details about contributing to the fund. Thanks.

More information about the Pay It Forward foundation…

Some unfinished business…

In the June newsletter on Project Retrospectives, I neglected to acknowledge my colleague and bike-riding friend, Mike McGonagle, who introduced me to Norman Kerth and the project retrospective process. My humble apologies Mike.