Welcome to GlobalLogic Blogs

It used to be a common cliché that an artist could not be truly “great” unless he or she had “suffered”. I suppose this expression has fallen into disuse because there are many examples of great artists who enjoyed happy childhoods, were rewarded with early success and a reasonable degree of material prosperity, and who do not appear to have suffered beyond the norm for the human condition. Sculptor August Rodin, painter Edouard Manet and composer Johan Sebastian Bach are a few apparently well-adjusted and happy individuals who come to mind. Our contemporary, brilliant billionaire filmmaker Stephen Spielberg, does not strike one as a tortured soul either.

Still, the image of the suffering or tormented artist is a strong and compelling one. One does not need to think too hard to come up with examples—Beethoven, Van Gogh, the suicidal Ernest Hemingway—the list is long. It’s not too hard to figure out why, either: One has the feeling that people who have gone through challenging experiences are deeper and have more understanding than those who have lived “easy” lives. Or maybe it’s a cultural or self-esteem issue that makes us think that one needs to work really hard, or pay a terrible cost, to achieve something really exceptional. The belief that achievement, knowledge and progress comes only at a price is a deep-seated and wide-ranging one, featuring prominently in sources as diverse as the Greek myth of Prometheus, the Biblical book of Genesis, and Mary Shelley’s Frankenstein.

Knowing all this, I nevertheless find myself feeling skeptical when I hear someone say: “Agile / Scrum is not so tough. It’s actually not that different from what we were already doing. We’ve been doing iterative development for years…” and so on. When I hear someone say this, I inevitably catch myself thinking: “If you haven’t suffered, you’re really not using Agile”.

Am I really that masochistic / moralistic? Or is there some other reason for this reaction?

On the one hand, Agile is not so different than pre-Agile software development methodologies. As someone who lived through that era, I do see Agile as the current manifestation of a trend in software development that has been around at least since the early 1980’s. A key feature of that trend has consistently been: (a) To produce early and visible results, (b) to evaluate those results, and (c) to immediately take corrective action. I see this essential thread running through the various 1990’s era methodologies such as round-trip iterative development, RUP, RAD, “iterative waterfall” and others. These, in turn, grew from seeds planted by the 1980’s era “structured programming” paradigm, early rapid prototyping tools like “Dan Briklin’s demo”, the first IDE’s from Borland and others, and the invention of “drag-and-drop” user interface paradigms by NeXT Software and other companies—all having the goal of making results visible faster. Other trends embodied by Agile, such as the empowerment of individual team members and the equal importance of both development and quality-focused activities, have similarly deep roots.

So in one sense, Agile is nothing new—the principles Agile embodies are those many of us in the software industry have been working with and toward for many years, through a variety of means.

On the other hand, I still believe that saying Agile is “more of the same” widely misses the mark. An airplane accelerating from a speed of Mach 0.8 to a speed of Mach 1.1 is indeed “just traveling faster”—but I would question whether someone who describes their journey that way was really paying attention when they broke the sound barrier. Water at sea level and a temperature of 220 degrees Fahrenheit (104 degrees C) is truly “just hotter” than it was at 200 degrees (93 degrees C). But if someone described it that way you’d have to wonder whether they’d ever really boiled water!

I believe adopting Agile is a similar “change of state”, where not only are you altering things in degree, but also altering them in kind. Taking an organization through a successful Agile adoption process is to take it through a phase change, like heating water to above its boiling point: a different kind of order and energy exists at the higher temperature than you had at the lower one.

My own reaction when I first read Kent Beck’s book “Extreme Programming Explained: Embrace Change” back in 2001 was to get angry. At that point in my career I had worked as or managed software architects for over a decade, and was an alumni of Rational, probably one of the more architecturally focused companies in the history of the software industry. Though I was already a firm believer in iterative development, Beck’s advocacy of iterative architecture made me want to throw his book across the room—and I’m not the kind of person who is generally inclined to do that sort of thing. Though I was never an advocate of the waterfall model for implementation, I had never even thought to question the premise that you must fully architect a system before you begin work on it. How else would you know what to do?

Reading further and doing some painful introspection, however, I realized that Beck was correct: Even though I had played a role in roughly two hundred commercial software releases by that point, no detailed architecture I had ever created or had knowledge of had ever been implemented as planned. There were always changes and tweaks due to things discovered along the way, or for other reasons. Project management considerations aside, 99 times out of 100 making these changes was the right thing to do. The high level architecture was indeed useful—but in every real case I knew of detailed architectural documents and designs consumed more time than the value they added, given that later learning would and should cause many of those details to change. I also knew of at least one occasion where rigid adherence to a detailed and overly complex initial architecture (not mine, thank goodness) actually caused a company to fail. It was a painful realization, but I had to admit that Beck was correct, and I had been wrong about my—or anyone’s—architectural omniscience. Change was inevitable and required. The only reasonable stance to take was embrace it, not fight it.

I mention my own initial negative response to portions of Agile to make a point: Agile stretches the cultural, organizational and process envelope in so many ways that it’s basically guaranteed to conflict with one or more cherished beliefs held by you or your organization. In other words, if you haven’t suffered—and changed and grown—in some way, I suggest you seriously question whether you’ve truly adopted Agile.

If Agile is implemented fully, its adoption will inevitably surface deep issues within the organization. Agile is a very radical process, with a constant focus on results. The need to produce working, potentially shippable software every couple of WEEKS means that people cannot hide behind long schedules anymore; they are constantly accountable. Likewise, any tooling or other issues that get in the way of producing results are surfaced almost immediately.

Generally when a company adopts Agile, the immediate impact is to make everything look worse than people thought it was. This is not because things are worse, but because problems that were formerly masked by process inefficiencies are now exposed. For this reason, even in the presence of “bottom-up” advocacy, a “top-down” familiarity with and commitment to Agile is also essential for success. Otherwise, the easy way out is to blame all the problems being exposed on Agile itself and to revert to the old way of doing things. Management commitment is essential to persevere beyond the exposure and resolution of these existing problems.

Management commitment is also essential to establish a trust relationship with employees. Because Agile will immediately surface problems that formerly remained hidden for long periods of time, people’s mistakes will surface very soon after they make them. If people are attacked for those mistakes, they will take a very conservative approach and their performance will suffer. If, on the other hand, mistakes are welcomed as a part of the learning process, then organizations will begin seeing better results than they ever thought possible, in part because the sooner mistakes are exposed, the easier they are to fix. The Agile mantra of “fail fast” is an invitation to try innovative approaches and rapidly discard those that don’t work. Being open about your mistakes and learning from them requires a high degree of trust “up”, “down” and “horizontally” within the organizational hierarchy.

Because Agile surfaces problems early, engineering management must also be trusted and supported by executive and senior management. In Agile, problems which surface early in a release are not signs of a failing project, as they might be in some other project management methodologies, and senior management must be made aware of this so results are interpreted correctly. Because of Agile’s emphasis on a product that is continually at a “potentially shippable” level of quality, small early problems are not indicative of bigger problems yet to come. Agile projects have no “big bang” integration at the tail end so progress is, by definition, linear. In a well functioning Agile project, “technical debt” such as the defect backlog is kept to a minimum level or completely eliminated throughout a project. A well-run Agile project is never, at any time, more than a few weeks away from being ready to ship commercially in terms of the quality of its completed features.

Agile has a huge impact on line managers, especially at the Lead and Manager level. This is because those levels of management no longer do task assignment within Agile projects. Agile is a “pull” methodology, where individual contributors choose the tasks they will do, rather than a “push” model where an authoritative boss assigns the tasks. This can be a challenge initially for the individual contributors, who must step up and be engaged at a new level—though in general once they get used to the “pull” model, most team members find this approach highly empowering and welcome it. The Leads and Managers, however, must learn to assume a coaching and mentorship role for project work, though their other management tasks, such as project oversight, obstacle removal, hiring, firing, training and career development, remain largely unchanged. Letting go of responsibility for day-to-day project task assignments can be a tough transition, especially for those recently promoted to supervisor.

When implemented properly, Agile is one of the most disciplined software processes imaginable. Because the emphasis is less on documentation and more on results, many jump to the conclusion that all you have to do to be Agile is to stop writing documentation and start writing code. And, I think, many claim to be Agile on the basis of lack of documentation alone! The key to Agile, however, is the emphasis on a product that is brought to a demonstrably shippable or “potentially shippable” condition at frequent short intervals, often one or two weeks. Imagine doing that literally, without compromise to product-level quality, test coverage, user and technical documentation, packaging or other standards—in fact, with heightened standards. To produce an truly commercially shippable product at such short intervals requires a high degree of discipline, co-ordination, quality focus and systems support. This, together with empowering teams with the ownership, processes, tools and frameworks necessary to achieve this goal, is in my view the essence of an Agile approach.

When an organization becomes truly Agile, the Agile teams themselves will begin to collaborate to spread best practices and optimize their level of contribution. In other words, success means that Agile becomes viral, spreading from group to group rather than only from the top down. Management still has a vital role in such an organization, removing obstacles, providing infrastructure, and setting product direction. Empowering the individuals within an organization tends to bring their full set of skills and creativity to bear on their work, achieving a higher level of productivity than is generally possible in a strictly top-down organizational structure. A real Agile organization is a fun place to work at every level. Being able to ship great products reliably and consistently allows an organization to focus its energy on future opportunities, rather than on a day-to-day struggle to simply get a product out the door and keep it alive. Even though the transformation to becoming an Agile organization is not an easy one, it brings huge benefits. Those who have achieved Agile success have found it well worth the investment—and even a little suffering.

Popularity: 28% [?]