Are We There Yet?

Every parent has heard this question a million times. When my kids were little, I would hear this question as we were backing out of the driveway. “No, not yet.” I would reply…

I have thought about this question with regard to software engineering. The software engineering discipline has experienced major changes since the late 1950’s. In contrast, changes to more mature engineering disciplines have been much more subtle. The journey has been anything but smooth – there have been many bumps along the road. To understand where software engineering is heading, we need to first understand from whence it came…

When did it become software engineering?

The term software engineering was was first used in the late 1950s. However, it wasn’t until 1968 at a NATO Science Committee conference that software engineering was recognized as an emerging engineering discipline.

An interesting historical footnote - During the 1950’s and 60’s the job of programming computers was performed mostly by women. Many women mathematicians, led by Admiral Grace Hopper, were instrumental in proving that computers could be used to solve real-world problems. During this same time, men were primarily focused on hardware engineering – which was then considered to be a more prestigious job...

Universities initially offered computer programming courses through the Mathematics or Physics Departments. This is partly a result of early computer users – who were primarily mathematicians and physicists. It wasn’t until the 1980’s that universities established departments and curricula specific to Computer Science at the undergraduate level.

Back in the day, developing programs involved using what now seems like stone age tools such as pencil and paper, flow chart templates, and coding sheets. Programs were “designed” by drawing detailed flowcharts. Programs were “written” by typing statements on computer punch cards using a keypunch machine.

After spending countless hours at the keypunch machine, the “card deck” was submitted to the computing center. Large mainframes were often hidden from view behind smoke-colored windows - accessible only to the first generation of computer nerds. Card readers would read your card deck along with dozens of others. Often, the card readers would mangle your cards and the computer operator occasionally got card decks mixed up. This required more hours of re-punching and re-ordering of your card deck. Once your card deck was successfully read, the first pass would often result in many compiler errors. After several iterations (which could take hours or even days), the program would finally compile cleanly. Only then would it be possible to begin the task of determining if it worked properly…

The Emerging Software Crisis

During the 1980’s, “programming” began its evolution into “software engineering”. But the fledgling software industry was facing its first significant challenge. By the end of the 1980’s, companies were spending more money on software maintenance activities than they were on new development. Quality was abysmal. Productivity was low. This was the infamous “software crisis” of the 1980’s – the first significant bump in the road. High profile software failures started appearing in the news. These failures resulted in economic losses and in a few cases, loss of life.

The “software crisis” gave rise to the Methodology Wars. Software gurus of the day advocated their own software development paradigms that were sure to get us out of this mess – what Fred Brooks called the Silver Bullet. They all promised, “if only those lazy, good-for-nothing programmers would adopt their methodology, all of our problems would be solved!” Ed Yourdon, James Martin, Tom DeMarco, and many others published their own methodologies and established cult-like followings.

Hindsight has shown that none of these methodologies ever lived up to their hype.

Another interesting historical footnote: The object-oriented methodology evolved from many sources including smalltalk, Simula 67, and Modula. As O-O methods started to become popular, many developers learned O-O programming languages like C++ before learning O-O Analysis and Design techniques. As a result, many early C++ programs thought to be “object-oriented” really were not because the developers had not learned to think differently.  

Chief Programmer Teams

In the 1970’s when Harlan Mills created the first Chief Programmer Team [4], he acknowledged a fact of human nature:

all programmers are not created equal – some are more adept than others at specific tasks such as architecture, design, coding, or testing…

(Do you see the similarity between Chief Programmer Teams and current Agile Methods?)

Another important contribution from Chief Programmer Teams was the notion of conceptual integrity. According to Fred Brooks:

“I will contend that conceptual integrity is the most important consideration in system design. It is better to have a system omit anomalous features and improvements, but to reflect one set of design ideas, than to have one that contains many good but independent and uncoordinated ideas.” [2]

During the 1980’s some people believed that the Silver Bullet could be found in better tools. CASE Tool companies developed and marketed all sorts of tools that promised significant improvements in productivity and quality. Not surprisingly, many of these tools never lived up to their hype either.

Others believed that more process was the solution. This eventually led to the establishment of the Software Engineering Institute (SEI) and the Capability Maturity Model ® (CMM)– one of the heaviest of the heavyweight processes.

The CMM ® and CMMi ® have been widely used for large government development projects. Outside of this arena, they are criticized as being overly bureaucratic and inefficient. This criticism begat the evolution of lightweight processes, commonly referred to as Agile Methods.

The Bubble

During the 1990’s, the software engineering journey shifted into high gear. As Internet access spread across the globe, it created explosive growth in software development. While software was being applied to more and more problems (as well as to things that were not problems), it also caused more and more problems. High profile software failures were becoming commonplace.

The dot.com debacle of the late 1990’s negatively impacted the reputation of software engineering. As thousands of dot.com companies sprang up over night, they competed fiercely for a very limited number of experienced software engineers. Many dot.coms hired people to develop web applications who had little or no training in software engineering principles. As a result, many of these applications became major disasters…

The proliferation of software that was fueled by the Internet has created new problems for software engineers and new challenges for software engineering. Hackers, SPAM, virtual terrorists, security leaks, identity theft, etc. are just some of the problems that have resulted from the proliferation of software since 2000.

So, are we there yet?

The Methodology Wars that started in the 1980’s are still raging. Today, it’s lightweight vs. heavyweight processes, Agile vs. traditional methods. One thing we should have learned by now is that…

There still is no silver bullet and there probably will never be…

As Roger Pressman said:

“I contend that software engineering principles always work. It’s never inappropriate to stress solid problem solving, good design, and thorough testing (not to mention the control of change, an emphasis on quality,...). A specific software process might fail because it is overkill, or the work products it requires are unnecessary or burdensome, or a person or team becomes overly dogmatic in the application of the process. But history is on the side of a solid engineering approach.” [1]

I couldn’t agree more. If software engineering is ever to become a respected engineering discipline, it has to be based on sound engineering principles – principles that have been shown to work. So what are some examples of such principles? Most of us probably have many examples…

Apply Sound Engineering Principles…

There have been many outstanding software engineering books written in the past five decades. Of all the computer science and software engineering books that have been published in the past 50 years, only a few have published special anniversary editions – more than two decades after they first appeared. Why is that? It’s because their message still rings as true today as it did when they were first published. They embody many sound engineering principles…

What are these very special books?

The Mythical Man-Month, by Fred Brooks [2]

A few examples of the lessons described by Brooks that he observed more than 30 years ago and still apply today include:

Adding staff to a late project will only make it later.

Hard to believe that there are managers out there that still don’t get this…

Conceptual Integrity 

This is probability the most important aspect of designing systems that isn’t taught in colleges and universities. When I meet with young software engineers, I often ask them about this and usually get a puzzled look.

Plan to throw one away

As the complexity of software has increased and the number of new systems grows, companies need to recognize that product marketing is not likely to get it right the first time. Planning to throw the first one away means you trash the code not the knowledge gained. And, we need to remember that the knowledge is far more valuable than the code.

Learn to develop accurate project estimates and realistic schedules

I can’t tell you how many times I’ve seen project teams give management the answer Management wants rather than the truth. Haven’t these people read Death March Projects? If you want to develop better estimates and realistic schedules, you need to first train people how to do this, and then, require that they do it. Managers, you need to be prepared for schedules that may not fit with your goals. The answer is to not force the schedule to fit, but rather, negotiate with the project team to see what can reasonably be done with the given resources and requirements…

Communication 

Since software engineering is an inherently human process, communication between humans developing software is probably pretty important. Yet, software engineers are notoriously bad communicators. Ever read a good requirements spec? They are few and far between. Product marketers are just as bad at communicating as software engineers. The common thread – English – one of the worst languages there is for stating requirements for software. English is inherently vague and ambiguous. Software must be deterministic. Do you see why there are so many communication problems here? Read more…

The other special book is from Gerry Weinberg and addresses some key behavioral characteristics of developers. He observed these issues over 35 years ago and yes, they still apply today:

The Psychology of Computer Programming, by Gerald Weinberg [3]

Weinberg identified several key personality traits that are critical for success as a software developer. These traits include:

• Ability to adapt to rapid change

• Organizational skills and the ability to keep track of lots of important bits of information

• Humility – your code will have problems that will be found by others

• Some level of assertiveness – needed to get things done often in the face of many obstacles

• A sense of humor, since the computer “doth make fools of us all”.

The take away message from these books is that software development is a complex human activity that is influenced by many factors. To be successful, its my contention that software engineering requires three critical ingredients:

I’ve observed that many organizations have one of the three, a few have two of the three, and only best in class companies have all three.

Summary

Lewis Carroll said that if you don’t know where you are going, any road will get you there. Software engineering is still a relatively young engineering discipline. The journey has had its ups and downs. I don’t know where things will end up but being a passenger on this journey has enabled me to make some observations and draw some conclusions. I’m enjoying the trip but sometimes it seems like We’re All Bozos on This Bus.