In the last few years software development teams have begun using a philosophy called "Agile", achieving better results by reducing reliance on documents, having more effective meetings, and thinking about development in a new way, says contributor Daryl Kulak. Interestingly, much of the Agile approach has roots in Dr. W. Edwards Deming’s management and quality work of the 1980's.
For many years, those of us in the field of computer software development have based our processes on what we learned from industrial engineering. Our solution to most problems we encountered was to create a bit more documentation, have a few more meetings, and institute more structure and roles. This approach has tended to increase costs, decrease productivity, and sacrifice speed and joy in work.
But an amazing thing has happened in the past few years. Software development teams have begun using a philosophy called "Agile" and have been achieving better results by reducing reliance on documents, meetings and roles.
The growth of Agile has been strong in North America and has spread quickly to South America, Europe and Asia. The sheer number of conferences, such as Agile Open in Belgium and Agile Eastern Europe in Kiev, shows the tremendous enthusiasm for Agile around the world. Information on the Agile approach is readily available on the web. One key element is that the Agile Manifesto makes it clear that agile processes promote sustainable development. The sponsors, developers and users should be able to maintain a constant pace indefinitely.
Agile software development can be viewed as having four main concepts:
-
Team-based decisions
-
Breaking down silos
-
Pursuit of business value
-
Iterative and incremental progress
Team-based decisions means that decision-making occurs as closely to the work as possible. Breaking down silos means that managers’ central function is to reduce the communication barriers between teams and also, importantly, the barriers that teams have in believing what is possible to achieve. The software manager’s central function is to coach the developers, assure communication is effective, and deal with any organizational barriers run into by the team. Often there will be daily meetings where any roadblocks are voiced. Either the lead developer or the manager will take responsibility for addressing these and report back on progress as soon as possible (certainly in the next daily meeting).
Pursuing business value means that Agile teams relentlessly link their work back to something that clearly benefits the entire organization, which might be more revenue or less expense in a for-profit company or tangible improvement for benefactors in a non-profit. Any team member is able to question whether the activity in front of the team is actually providing the business value or not. Finally, iterative progress means doing (and quickly redoing) activities to increase quality, and incremental progress means taking on bite-size chunks at a time, to ensure that people can hold the current work in their heads easily. A key part of this is that working software is delivered quickly to allow users to see it in action. This will often be stripped down and missing bells and whistles but working software none-the-less. Users (and business managers) are much more effective at giving comments based on working software, even with limitations, than just abstractly creating requirements for a huge number of features in advance.
If you have studied the work of Dr. W. Edwards Deming, the insightful management consultant who wrote extensively on quality issues as well, you may recognize many of those ideas from his work. Although the proponents of Agile do not often link their ideas back to Deming, per se, the connection is clear to people familiar with both worlds.
Leaders as Systems Scientists
The role of the leader or manager is similar in Agile and Deming’s work. In Agile we view leaders as people who are facilitating the progress of the team, and who are helping to tweak the process to improve things. The leaders focus on changing the interaction between team members on any given team as well as the interaction between a team and other teams. Leaders do not dive into the details of what each team member is doing, that is for the team members themselves to decide.
Deming saw the power of the manager as being responsible for improving the system in which people work. He wanted leaders to view themselves as systems scientists, looking at the system and changing it, and avoiding placing blame on individuals. Deming saw, in almost every case of a problem or mistake, that the system, as designed by leaders, was to blame, not the individual working within the system.
Time-Series Graphs
Agile teams use a simple, useful chart called a "burndown chart." This shows the progress of the team so far at regular intervals (each week, perhaps) and then projects what the team might continue to achieve in the future, given the historical levels. This has been a dramatic improvement over the typical detailed (and ineffective and inaccurate) estimates software teams have tried to use in the past. Deming was in favor of using time-series graphs, similar to burndown charts, to reveal information about a system. He asked managers to pay attention to the variation without necessarily trying to "manage the variation away" by intervention and tampering. Knowing about variation that was simply part of the system (common cause) and variation that was caused by some unique circumstance (special cause), managers could take appropriate actions based on the type of variation to change the system in ways that might be useful.
Drive Out Fear
In another comparison, more advanced Agile teams try to create a culture that reduces the fear people have about doing the wrong thing. They feel that team members need latitude and the absence of pressure to be at their most creative and most efficient and effective. Agile team members are asked to be courageous in trying new things and taking risks in pursuit of continual improvement and productivity.
Deming said it more succinctly: Drive out fear. He saw that managers should have a central goal of taking fear out of the equation for their teams. This has always been very strange advice to managers who are used to using fear as a "motivational tool." In my experience, driving out fear is the only way to get the very best performance from team members.
One of the chronic situations with many software development work teams is to respond to unreasonable demands by just working harder. There is a great deal of evidence working long hours actually slows down progress (for many reasons, not the least of which is bugs and sloppy code introduced in late hour and weekend sessions). One of the key agile principles is to work at a sustainable pace. Long work hours are seen as failure in agile not evidence of dedicated employees.
Agile and Deming - Not the Same Yet
It’s true that there are many similarities between Deming’s philosophy and Agile software development. It is also worth looking at some areas where Agile and Deming are not yet in alignment.
Deming pushed companies to drop their "incentive pay" schemes because he saw them as damaging to individuals and to the organization. Agile does not specifically address anything about incentive pay, nor about performance reviews, pay-for-performance, or the like, although true agility is enhanced when agile software development is not subjected to such practices.
Deming also warned companies about trying to manage via a focus on numerical goals and quotas. In fact, he named "management by visible numbers alone" as a deadly disease. Agile, however, too often easily falls into the trap of judging progress by numerical goals because progress is measured in "storypoints" (functionality delivered). This can become a dangerous management tool if used without understanding that storypoints are not only about getting a read on quantitative progress, and are intended to provide opportunities to discuss ways to achieve greater quality, creativity and continual improvement. The best agile implementations understand the idea that software estimates include a great deal of variation. Understanding Variation is a foundation of Deming’s teachings. Thus, when applied to Agile leaders do not react to variation with blame and demands to work harder. Alas, not all Agile efforts have this level of understanding.
Agile software development has taken many, many pointers from Deming’s work, and has made a great deal of progress. Agile simply hasn’t caught up with all of Deming’s management methods. And, as we are catching up, we need to make sure we do not stray from the Deming path and subjugate Agile development to the counter-productive but common management practices that create fear, misunderstand and misuse data on variation, and prevent systems thinking –practices that have invaded organizations and limited their progress. I am optimistic that we in the software world can continue on our path towards applying Deming’s view of systems thinking and his ways to manage to unleash innovation and productivity.
Author’s Note: Thank you to Kelly Allan and John Hunter for their suggestions and insightful editing.
Editor’s Note: The columns published in THE DEMING FILES have been written under the Editorial Guidelines set by The W. Edwards Deming Institute. The Institute views these columns as opportunities to enhance, extend, and illustrate Dr. Deming’s theories. The authors have knowledge of Dr. Deming’s body of work, and the content of each column is the expression of each author’s interpretation of the subject matter.