Food for Thought: Agile Methods - Beyond the Hype An e-newsletter published by Software Quality Consulting, Inc. July/August 2005, Vol. 2 No. 7 To view a web version of this newsletter, click on the following link: http://www.swqual.com/newsletter/vol2/no7/vol2no7.html. -------------------------------------------------------------------------------- Welcome to Food for Thought(TM), an e-newsletter from Software Quality Consulting (http://www.swqual.com/index.html?Intro). I've created free subscriptions for my valued business contacts. If you find this newsletter informative, I encourage you to continue reading. Feel free to pass this newsletter along to colleagues by clicking th Forward Email link at the bottom of this email. If you’ve received this newsletter from a colleague and would like to subscribe, please click this Enter New Subscription link (http://www.swqual.com/newsletter/Subscribe.htm). If you don't wish to receive this newsletter, click the SafeUnSubscribe(TM) link at the bottom of this newsletter, and you won’t be bothered again. Your continued feedback on this newsletter is most welcome. Please send your comments and suggestions to info@swqual.com. -------------------------------------------------------------------------------- In This Months’ Topic, I discuss the raging debate over Agile Methods... Regular features to look for each month are: - Monthly Morsels Hints, tips, techniques and reference info related to this month’s topic - Calendar Conferences, workshops, and meetings of interest to software engineers, QA engineers and anyone interested in software development -------------------------------------------------------------------------------- *** Agile Methods - Beyond the Hype*** Agile Methods came to the forefront with the publication of the so-called Agile Manifesto [1] in 2001. Not since Dijkstra’s [2] classic paper “Go To Statement Considered Harmful” in 1968 where he stated “... the go to statement should be abolished from all ‘higher level’ languages...” has there been such an impassioned debate about software development methodology. There has been so much hoopla about the perceived benefits of “lightweight” vs. “heavyweight” development processes that camps have formed on both sides of this issue, each with their own zealots and their own books... I distinctly remember reading the “manifesto” in 2001. It was the first technical article I ever read that elicited a visceral reaction. It made me mad. My reaction was something like this: “What were these guys doing? We have been working so hard to improve the perception of software engineering and instill discipline in developers and managers. And then this... Are they nuts? And why would anyone use the word manifesto in an article on software development?” The term “manifesto” evoked visions of the “Communist Manifesto” [3] written in 1848 by Karl Marx and Frederick Engels – the acknowledged beginning of the Communist Movement. Here is a short snippet from an introduction by G. S. Jones [3]: “... the Manifesto sketches a vision of reality that, at the start of a new millennium and against a backdrop of endless chatter about globalization and deregulation, looks as powerful and contemporary a picture of our own world as it might have appeared to those reading it in 1848.” Wow. I immediately started writing a letter to the editor of IEEE Computer. Something I had never done before. My frustration was evident in my letter: “Please excuse my cynicism. But I’ve been waiting a long time for software engineering to become a respected engineering discipline. While progress has been made, it has been painfully slow and frequently impeded by people within the software community.” [4] When the December 2001 issue of IEEE Computer, I was surprised to see my letter was selected for publication. Good, I thought, someone needs to speak up for process! I received a few e-mails in response to the letter, all supporting my views. THE FOUR VALUES Agile Methods are a collection of several methodologies that include: - Extreme Programming (XP) - Crystal - Scrum - Dynamic Systems Development - Adaptive Software Development - Lean Software Development While each is different, they all share common values as expressed by the “manifesto”. To refresh your memory, the essence of the “manifesto” are the four values: “We are uncovering better ways of developing software by doing it and helping others do it. Through this work we have come to value: - individuals and interactions over processes and tools, - working software over comprehensive documentation, - customer collaboration over contract negotiation, - responding to change over following a plan. That is, while there is value in the items on the right, we value the items on the left more.” [1] In the subsequent “endless chatter” about Agile Methods, I unwittingly became the poster boy for the process camp. I was mentioned in an IEEE Computer article written by Barry Boehm and Tom DeMarco: [6] “Those on the pro side—Jim Highsmith and others—argued that the current approaches are broken and should be replaced. Those on the con side—Steven Rakitin, for example—said that agile approaches are just hacking by another name and that we shouldn’t abandon our disciplined processes now that we’re finally getting them right.” And a sidebar to another IEEE Computer article written by Barry Boehm [5] stated that: “Although their advocates have used each of these agile methods successfully in practice, critics who prefer process-based methods remain skeptical. In a letter [4] to Computer, Steven Rakitin indicates that, in his experience, the items on the right are essential, while those on the left serve only as easy excuses for hackers to keep on irresponsibly throwing code together with no regard for engineering discipline. He provides ‘hacker interpretations’ that turn agile value propositions such as ‘responding to change over following a plan’ into chaos generators. Rakitin’s hacker interpretation of ‘responding to change over following a plan’ is roughly ‘Great! Now I have a reason to avoid planning and to just code up whatever comes next.’ ” So much for my five minutes of fame. A SHORT HISTORY LESSON... The publication of Dijkstra’s article on go to statements [2] in 1968 led to what has come be known in software engineering lore as the Methodology Wars. (See [7]). People like Ed Coad, James Martin, Ed Yourdon, Tom DeMarco, Michael Jackson (no, not that Michael Jackson), and others developed and then began pushing their own software development “methodology”. Each of these gurus had their own cult following and detractors. They all promised, “if only those lazy, good-for-nothing programmers would adopt their methodology, all of our problems would be solved”! (Does any of this sound familiar?) The Three Amigos The advent of “structured methodologies” gradually evolved into Object-Oriented (OO) methods, mostly due to the work of the infamous Three Amigos (Grady Booch, Ivar Jacobson, and James Rumbaugh). So, to continue our brief history, OO begat UML which begat RUP (Rational Unified Process), which begat lots of money for Rational from books, Case Tools, and training. For a more complete view of the history of software development, visit Wikipedia (http://en.wikipedia.org/wiki/Software_engineering). The timeline below is my attempt at putting the Methodology Wars in perspective... Software Development Methodology Timeline [for timeline image, see the HTML version – http://www.swqual.com/newsletter/vol2/no7/vol2no7.html] As you can see, many methodologies have come and gone over the past three decades. Having lived through all of these, I’ve become a bit cynical when yet another methodology is purported to be the Silver Bullet because we should remember what Fred Brooks said – there is no Silver Bullet. [8] In looking back over the past three decades, I believe there have been cases where each methodology shown above has performed well and cases where each has failed miserably. This is true for every methodology from Agile Methods to CMM and everything in between. WHY HAS THERE BEEN SO MUCH HYPE? One of the things I believe the Agile zealots learned quickly was that there’s a lot of money to be made as a result of publishing books on methodologies. Today, amazon.com lists at least 27 books with “Agile Methods” in the title! Anyone remotely affiliated with an Agile project it seems has written a book on the virtues of Agile Methods. The C3 project at Chrysler (http://c2.com/cgi/wiki?ChryslerComprehensiveCompensation) was the first large project where many Agile principles and methods were applied collectively and intentionally. The small team working on C3 included Kent Beck, Ron Jeffries, Chet Hendrickson and others. What is not so well known is that the project was a failure (http://c2.com/cgi/wiki?CthreeProjectTerminated). In spite of the fact that the project failed to deliver what the customer expected... “By the time C3 was cancelled, XP was well on its way to the phenomenon that it is today... Beck, Jeffries, Hendrickson and others had book contracts in hand.” [9] I am constantly amazed by the ability of those of us in the software industry to put positive spins on projects that would otherwise be viewed as dismal failures. Consumer advocates constantly remind us that: If it sounds too good to be true, it probably is. This applies to telephone solicitation schemes as well as software development methodologies. MY BOTTOM LINE... Agile Methods can work given a specific set of circumstances. In my opinion, those circumstances are projects where: - software is being developed for use solely within the company - a real, live “customer” (end user) is available to the project team 24x7 - the risk of failure is low - the project team is highly motivated and led by an excellent leader - long term maintenance of software is not a concern Under these circumstances, I believe Agile Methods may work well. But so will other methods. And this is the source of my frustration with all of the hype about Agile Methods. Agile Methods can be effective but so can other methods. For projects that don’t meet the circumstances above, read what Roger Pressman observed: “I contend that software engineering principles always work. It’s never inappropriate to stress solid problem solving, good design, and thorough testing (not to mention the control of change, an emphasis on quality,...). A specific software process might fail because it is overkill, or the work products it requires are unnecessary or burdensome, or a person or team becomes overly dogmatic in the application of the process. But history is on the side of a solid engineering approach.” [10] And for projects that are high-risk or life-threatening, read what Mark Paulk said: “Should organizations use XP, as published, for life-critical or high-reliability systems? Probably not. XP’s lack of design documentation and de-emphasis on architecture are risky.” [11] Those Agile Methods like XP that rely on verbal communications rather than written documents appeals to some people. The people this doesn’t appeal to are sustaining engineers. I pity the poor slobs who have to maintain applications developed using Agile Methods. With no design documentation or requirements to refer to, you’re left with only the code. I contend that even incomplete or outdated design documentation is better than no documentation. How many times have you seen cases where maintenance is so difficult because of a lack of design documentation that we decide it would be easier to start over than to figure out how the existing code works? Like the CMM, many Agile Methods include practices that can be applied regardless of methodology. For example, test first design is an XP practice. Test first design is definitely a good thing to do whether you are using an Agile Method or not. Deciding what methodology to use is a difficult task. Before committing to a specific method, become as informed as you can on both the advantages and disadvantages. Remember, there is no silver bullet. Developing complex software is a difficult and challenging task. There are many pitfalls along the way. Use what will work best for the situation you are in. Leverage the skills of the people that will be working on the project. And most of all, do what makes sense. That’s my story and I’m sticking to it... This issue was inspired by a spirited debate titled “Agile Methods Considered Harmful?” held at a monthly meeting of the Software Quality Group of New England between myself and my friend and colleague Scott Andersen. PAY IT FORWARD If you find this newsletter of value, please consider the following: Norm Kerth, a highly respected consultant, suffered a disabling brain injury in an automobile accident. As a result, he cannot work and lives on a limited income. You can help him by sending a donation. Checks can be made payable to Norm Kerth Benefit Fund and sent to Norm Kerth Benefit Fund c/o Process Impact, 11491 SE 119th Drive, Clackamas, OR 97015-8778. You can also visit Karl Weiger’s website (Process Impact http://www.processimpact.com/goodies.shtml)for more details about contributing to the fund. Thanks. More information about the Pay It Forward foundation... (http://www.payitforwardfoundation.org/) Some unfinished business... In the June newsletter on Project Retrospectives, I neglected to acknowledge my colleague and bike-riding friend, Mike McGonagle, who introduced me to Norman Kerth and the project retrospective process. My humble apologies Mike. Please note: There will be no issue in August – see you again in September. -------------------------------------------------------------------------------- ***Monthly Morsels*** Every month in this space you’ll find additional information related to this month’s topic. - References: [1] Highsmith, J. and Cockburn, A., “Agile Software Development: The Business of Innovation”, IEEE Computer, September 2001, p. 120-122 [2] Dijkstra, E., “Go To Statement Considered Harmful”, Comm. ACM, Vol. 11, No. 3, March 1968, p. 147-148. [3] Marks, K. and Engels, F., The Communist Manifesto, Penguin Classics, 2002. [4] Rakitin, S., “Manifesto Elicits Cynicism”, Letters to the Editor, IEEE Computer, December 2001, p. 4. [5] DeMarco, T. and Boehm, B., “The Agile Methods Fray”, IEEE Computer, June 2002, p. 90-92. [6] Boehm, B., “Get Ready for Agile Methods, With Care”, IEEE Computer, January 2002, p. 64-69. [7] Software Methodologies: The Battle of the Gurus, Info-tech Research Group, London, Ontario, Canada, 2002 [8] Brooks, F. P., "No Silver Bullet: Essence and Accidents of Software Engineering," IEEE Computer, Vol. 20, No. 4 (April 1987) pp. 10-19. [9] Stephens, M. and Rosenberg, D., Extreme Programming Refactored: The Case Against XP, Apress, 2003. [10] Pressman, R., “What A Tangled Web We Weave”, IEEE Software, Jan/Feb 2000, pp. 18-21. [11] Paulk, M., “Extreme Programming from a CMM Perspective”, IEEE Software Nov/Dec 2001, p. 19-26. - Books: There is a plethora of books on Agile Methods. Just visit your favorite on-line bookstore and search on Agile Methods. Here’s a sampling: Books on Agile Methods: - Extreme Programming (XP) Beck, K., Extreme Programming Explained, Addison-Wesley, 2000. - Crystal Cockburn, A., Crystal Clear : A Human-Powered Methodology for Small Teams, Addison-Wesley, 2004. - Scrum Schwaber, K. and Beedle, M., Agile Software Development with SCRUM, Prentice-Hall, 2001 - Dynamic Systems Development Coleman and Verbruggen, A quality software process for rapid application development, Software Quality Journal 7, p. 107-1222 (1998) - Adaptive Software Development Highsmith, J., Adapive Software Development, Dorset House, 2000 - Lean Software Development Poppendick, T. and Poppendick, M., Lean Software Development: An Agile Toolkit, Addison-Wesley Professional, 2003. Books critical of Agile Methods: - Boehm, B. and Turner, R., Balancing Agility and Discipline: A Guide for the Perplexed, Addison-Wesley, 2003 - Stephens, M. and Rosenberg , D., Extreme Programming Refactored: The Case Against XP, Apress, 2003 - McBreen, P. Questioning Extreme Programming. Addison-Wesley, Boston. 2003. - On-line Resources: - Unbiased information from Wikipedia on Agile Development Methods... (http://en.wikipedia.org/wiki/Agile_software_development#References) Chrysler Comprehensive Compensation (C3) Project Wiki Sites - This site provides interesting commentary on the project... (http://c2.com/cgi/wiki?ChryslerComprehensiveCompensation) - And, after the project was terminated... (http://c2.com/cgi/wiki?CthreeProjectTerminated) Sites supportive of Agile Methods: - Agile Alliance (http://www.agilealliance.org/) - Extreme Programming (http://www.extremeprogramming.org/) Sites critical of Agile Methods: - Software Reality (http://www.softwarereality.com/ExtremeProgramming.jsp) -------------------------------------------------------------------------------- ***Calendar*** Every month, you’ll find news here about local and national events that are of interest to the software community ... - Software Quality Calendar There are many organizations that sponsor monthly meetings, workshops, and conferences of interest to software professionals. Find out what’s happening... (http://www.swqual.com/links/upcoming.html) - Workshops Offered by Software Quality Consulting Software Quality Consulting offers workshops in many topics related to software process improvement. Get more info... (http://www.swqual.com/training/on_site.html) -------------------------------------------------------------------------------- ***About SQC*** Software Quality Consulting provides consulting, training, and auditing services tailored to meet the specific needs of clients. We help clients fine-tune their software development processes and improve the quality of their software products. The overall goal is to help clients achieve Predictable Software Development(TM) – so that organizations can consistently deliver quality software with promised features in the promised timeframe. To learn more about how we can help your organization, visit our web site (http://www.swqual.com/index.html?AboutSQC) or send us an email (info@swqual.com). -------------------------------------------------------------------------------- I hope this newsletter has been informative and helpful. Your comments and feedback are most welcome. Send me your feedback... (info@swqual.com) Thanks, Steve Rakitin info@swqual.com Food for Thought and Predictable Software Development are trademarks of Software Quality Consulting, Inc. Copyright © 2005. Software Quality Consulting, Inc. All rights reserved. Graphic design by Sage Studio (sage_studio@yahoo.com)