Are you "doing" agile or "being" agile?

One of the most colorful statements that I have heard lately says "A critical predictor of success in Agile transitions is how people define reality". When people transition into the scrum they tend to see a lot of bad mojo creep out of the woodwork. Value and technical debt, managers and teams wrestling with new roles and conflicts between teams. People now are no longer able to hide behind the fogs of the waterfall.

The whole idea behind scrum and agile methods in general is to make these impediments visible to the management so that requisite actions can be taken early and in time to make projects successful. You can think of the:

• The organizational reality as the Matrix (like the movie)
• Somebody in the management who is not satisfied with the status quo, who knows something is wrong as Neo, the transition sponsor
• The agile solution provider or coach as Morpheus

Neo is given a chance to choose either one of the two pills:

• The blue pill which will make him wake up the next morning in the current reality believing whatever he wants.
• The red pill which will make him learn the truth and see how deep the rabbit hole really is.

People who swallow the red pill are those who choose to confront the crimson problems head on. There is a fundamental shift in the mindset triggering a change in behavior as prescribed by agile methodologies. Scrum is used as a strategy to create a better and more sustainable organization. On the other hand people who swallow the blue pill are in a tactical agile situation. This might produce some local results in terms of team productivity and morale but eventually will lead to an organisational backlash which reverses the agile processes or a place where people will tell you "We are agile, but ..."; a condition referred to as Scrumbutt.  Of course there is no place for Morpheus in this situation!

Organizational reality is really about how things are done to succeed. I found this little graphic which puts organizational cultures into some perspective:

Agile methods talk about using a more people oriented culture to achieve competence by promoting collaboration between people and cultivating individuals and teams to fulfill a vision. This is the real shift in mindset that is needed to move to a strategically agile condition and here is where three out every five agile implementations fail. The graph below from a survey by VersionOne in 2010 comprehensively relays the story.

At the end of the day if you are trying to be the champion of agile methods to a "Neo", think about how he perceives reality and how he looks at the situation because the final choice of the Red or the Blue pill belongs to him!

Messy Documents!

One of the tenants of the agile manifesto is working software over comprehensive documentation. A lot of people misunderstand this to writing no documentation at all. The manifesto itself goes on to say "That is, while there is value in the items on the right, we value the items on the left more". Every project invariably needs documentation and not writing any is a recipe for doom. That said, it must noted that document creation and their maintenance are expensive and need to be managed carefully.

Documents are ideal for maintaining organizational memory but are ineffective as a means of communication during project implementation. The primary issue of communication is one of understanding and not that of documentation. Transferring knowledge or ideas through documents is a skill, you might even consider it an art and this skill is definitely not easy to find. Even the best of developers find it difficult to write good documents simply because our educational systems don't allow for students to gain this skill. Effectively the question that needs to be asked is "do you need documentation" and not "do you want documentation". Documentation is needed if that's the best way to achieve the relevant goals, but there often proves to be better ways to achieve those goals than writing static documentation.   

I tend to agree with an agile modelling approach where a document is any artifact external to source code whose purpose is to convey information in a persistent manner.  This is different from the concept of a model, which is an abstraction that describes one or more aspects of a problem or a potential solution addressing a problem.  Some models will become documents, or be included as a part of them, although many more will simply be discarded once they have fulfilled their purpose.  Some models will be used  to drive the development of source code, although some models may simply be used to drive the development of other models.

A low-fidelity prototype of a system's user interface or a picture of a block diagram sketched during a meeting between developers or a short article on an internal wiki are real "information radiators" as compared to a twenty five page word document stuck on a share point site. Furthermore the sheer thought of the cost for laying out and formatting documents and getting them formally reviewed when they are just models which can be thrown out should be repulsing!

Traditional development methodologies emphasize greatly on a lot of documentation early on in the SDLC in the form of requirement specifications, detailed design and architecture documents and project plans. If you look at the list, most of these documents are speculative and would require a lot of effort to create and maintain since most of the project parameters would change substantially during the project execution. There is no real and  solid relationship between project success and writing comprehensive documentation, and in fact it is more likely that the more documentation that you write the greater the chance of project failure. Statics actually show that agile teams produce more quality documentation than traditional teams at a fraction of the cost.

The following list enumerates some best practices that are suggested by agile modelling techniques to use documentation effectively:

• Prefer executable specifications over static documentation. A Test Driven Design (TDD) approach or a Behavior Driven Design (BDD) approach to requirement gathering, architecture and design can go a long in producing Just-In-Time specifications for an effective implementation. With TDD you write a test, either at the customer acceptance level or the developer level (unit tests), before writing sufficient functionality to fulfill that test.  The tests are used for two purposes: they specify the requirements/architecture/design and they validate your work.  This is an example of the practice of Single Source Information.
• Document stable concepts not speculative ideas. The agile strategy is to defer the creation of all documents as late as possible, creating them just before you need them. System overviews are best written towards the end of the development because you know what you’ve actually built.  Similarly, a majority of user and support documentation is also best written towards the end of the lifecycle.  However, this doesn't mean that all documentation should be left towards the end.  It is a good idea to take notes for these sorts of documents throughout development so that critical information is not lost.  These notes may be nothing more than point-form information in code coments as there is no need to layout and format documents until just before final delivery.
• Generate System Documentation. Modern software-based modeling tools can reverse-engineer existing code and present a multitude of views into it. Doxygen is just one example!
• Keep Documentation Simple.The best documentation is the simplest that gets the job done.  Don’t create a fifty-page document when a five page one will do.  Don’t create a five-page document when five bullet points will do.  Don’t create an elaborate and intricately detailed diagram when a sketch will do.  Don’t repeat information found elsewhere when a reference will do.  Write in point form.  Document only enough to provide a useful context. Start with a document that's minimal enough for the needs of its customers then augment it as needed
• Write few documents with least overlap. This is the same as the principle of "Don't Repeat Yourself" or DRY. Strive to travel as light as possible, writing on just enough documentation for the situation at hand which is just barely good enough to fulfill it's purpose. Use wikis and hyperlinking to connect small documents to create a whole instead of rewriting.
• Display Information publicly. Create information radiators considering issues such as indexing, linking, and accessibility when writing documentation because the end customers of the documents are not always known. The greater the communication on your project the less need for detailed documentation because people already know what you’re doing.
• Treat documentation as a requirement. Always attach a business value to a document and prioritize them like any other feature in the project. Any time spent on documentation could have been potentially spent on feature development. By treating documentation as a requirement you make its creation a visible to the stakeholders to consider.  Fundamentally, the investment in documentation is a business decision, not a technical one: you shouldn't create documentation because your process says you should.
• Require that stakeholders justify document requests. Everybody involved must understand the total cost of ownership (TCO) for a document, and strive to maximize stakeholder ROI to provide the best value possible to the project. Stakeholders must explicitly choose to make an informed investment in the documentation. The benefit of having documentation must be greater than the cost of creating and maintaining it.
• Recognize That You Need Some Documentation. Documentation is as much a part of the system as the source code.  In addition to working software, projects need to minimally deliver user manuals, support documentation and system overview documentation. The agile team’s primary goal is to develop software but its secondary goal is to enable the next effort. Building high-quality working software which meets the needs of the stakeholders is important, but ensuring that the people who come after you can maintain and enhance it, operate it, and support it is also equally important. Documents still serve a purpose of capturing important information permanently.

Documentation is critical to the success of any project, too much is as bad as too little. Success hinges on getting just that right amount!

Discipline Envy: Building software worlds with blueprints?!

In many engineering disciplines, civil engineering is a perfect example,  implementation is preceded by a distinct design phase which produces a defined architecture; a blueprint for implementation. For all practical purposes it can be said that design is an activity which spawns the architecture and the corresponding implementation. We know that this paradigm works extremely well when building bridges but does it work when constructing software.

I don’t think that in the software world design can be compartmentalized into a phase. The interactivity inherent in software systems creates severe dynamism in requirements. Moreover the swiftness with which tools and technologies evolve do not allow for a Big Design Up Front (BDUF). The fundamental assumption that BDUF makes is that architects and designers can visualize pitfalls without prototyping and some implementation. This unfortunately does not apply to the software development process and to me it looks like a classic case of Discipline Envy.

Ward Cunningham (of Wiki fame) wrote that the programming discipline has always had an identity crisis. It never seems happy to be itself but instead pretends to be the discipline it serves at the moment, or the discipline from which it has drawn the most members. First our discipline wanted to be like mathematics and then we were a science, off on a voyage of discovery with the scientific method at our side. Now we have matured to engineers, liberated by certified processes and writing our handbook.

I think Ward is absolutely right; what we seem to be doing now is borrowing practices from other engineering disciplines to write our process handbooks. Engineering Envy! This may work sometimes but comparing practices in software engineering with those in civil engineering is going to fail most of the times. Instead of looking at established practices, we need to invest into looking at our own patterns and create paradigms which will help us better optimize the age old iron triangle of speed, cost and correctness.

There needs to be a balance struck between the time spent on planning and the time needed to refactor to fix a defect. In most software projects there is always a dearth of requirements which will make any analysis or design phase stretch out and lead to an anti-pattern called "analysis paralysis", which I have already spoken about in an earlier post. With the inherent dynamism, the cost of planning is almost always higher than the cost of fixing, effectively laying waste to the time that was put into planning.  Continuous Deployment, Automatic Updates, Fault Tolerance and related ideas substantially reduce the cost of defects in production so that they become cheaper to fix at run-time than to plan out at the beginning. These concepts are far more suited to the software development environment.

Another undeniable feature of any software development life cycle is change. Customers normally don't really know what exactly they want. Most of the time, given a prototype, they will be able to say definitely what they don’t want. Given this situation if you are making a BDUF, a lot of assumptions are inevitable that later will prove to be false but these are already designed and possibly coded. This brings me to the concept of YAGNI or You Ain't Going to Need It. The idea is to design, code and test only when a feature is absolutely needed and not when it is foreseen. A kind of Just-In-Time approach which eventually reduces a lot of wasted work and time.

Design has to be an iterative activity with no real start and end points stretching from inception all the way up to maintenance. Software design or architecture cannot be completed in a "phase"; design and architecture need to be emergent across the span of the product lifecycle which will in turn create a sustainable development environment.

I am not advocating "cowboy coding" or saying that there is absolutely nothing needed upfront. There definitely has to be an approach and a vision for the development process but it need not be big and grand. Nothing is going to evolve unless it is seeded. All I am questioning is the depth of the design made upfront and saying that it looks that we will be much better off at a point three quarters down the line below:

I sometimes wonder though is this too darwinian for a creationistic world!

Mushrooming Clouds!

Mushroom cloud from a hydrogen bomb test

All hierarchical and process-bound organisations are guilty of "mushroom management" at some point or another. Mushroom management is a satirical metaphor for a situation when access to information and decisions are restricted to employees and furthermore when the information is conveyed, it is inaccurate, incomplete and full of hyperbole. It is like being kept in the dark and having the dirty stuff thrown at you. Even if you are mildly ambitious this style of leadership will curtail your creativity and hold you back.

Business today is about change and adapting to it. Nowhere else is this more evident than in the software industry. Free flowing and ubiquitous communication are the corner stones to innovation, good customer service, high retention and change. You would think that with the Internet, email and mobile phones communications would have improved by leaps and bounds. But it turns out this is not really true. If you ask most people what they would like their supervisors or managers to do to make the work environment better, you will hear "communicate better" most of the times. In my opinion this is the case because, email and mobile phones have substantially increased the quantity of communication but they have had a damning effect on the quality. The tools are improving but the content is deteriorating creating a rude deception!

Keeping a tight rein on information has never proved to be a good management strategy. The most famous example of this was the Teamster's strike at UPS in 1997. It is estimated that UPS lost nearly $700 million in revenue apart from the loss of credibility with its customers and employees. Many commentators argue that as part of its negotiations the management held a lot of information close to it's chest. It turned out that this flawed strategy caused many loyal employees to feel betrayed because they simply did not know what was going on. As a consequence they simply decided to leave. The lack of information caused a backlash with anger, resentment and  loss of revenue. 

Functional groups in organizations need to be "high-access" with communication at the center. Good communication will invariably lead to high retention. On the other hand "low-access" groups are characterized by constricted communication flows where people find themselves kept in the dark like mushrooms, stuffed in silos of job descriptions, rankings and the organizational chart. It is but natural that these groups have trouble responding to change and wavering customer needs which is a reality of the modern software producing work environment. The mantra has to be "when in doubt, over communicate!".    

All in all mushroom management in the software product development world is an anti-pattern that will only end up producing mushroom clouds. 

  

  

The Dangerous Chicken

Angry Birds

Chicken is a fairly well modelled conflict pattern in game theory circles. It is a simple game really where each player tries not to yield to the others. The worst possible outcome of the game though occurs when none of the players yields. You would have seen this situation depicted in many movies. Two gang brats with their snappy cars set out to prove something to their cronies or settle a dispute. They get into their cars and a skimpily clad hottie flags them off at warp speed towards each other. The brat who blinks first and veers off the path is the "chicken" while the other one gets the bragging rights and probably the flag bearing hottie as a bonus. If neither of the hooligans swerves of the path, the result is a crash which will probably be deadly producing the worst outcome of the game.

In the software industry especially in project management the game of chicken manifests itself in scheduling and is often referred to as "Schedule Chicken". The condition occurs when teams start committing to unrealistic schedules assuming that other dependent teams are crunching their schedules even more.

Project delivery dates, especially so for new product development, are ridiculously difficult to estimate primarily because:

• Uncertainty is inherent and inevitable in software development processes and products - Ziv’s Uncertainty Principle 
• For a new software system the requirements will not be completely known until after the users have used it - Humphrey’s Requirements Uncertainty Principle 
• It is not possible to  completely specify an interactive system – Wegner’s Lemma
• Ambiguous and changing requirements, combined with evolving tools and technologies make implementation strategies unpredictable.

It might seem like I am stating the obvious but these tenants do not seem to be very well accepted. "The management" is always looking for a "date" for obvious reasons, irrespective of the uncertainty and the ambiguity. They are in most cases pulling in an untenable date that can be sold to the bosses which in turn is sold to their bosses and so on. To arrive at a date there is bound to be a lot of analysis with:

1. Requirements being gathered and formally written 
2. Big designs being made upfront
3. Project estimation being made and
4. Complex interrelated plans being drawn up. 

With all the requirements being far from clear since customers themselves only have a vague idea of the final product, the analysis phase in most cases will turn into an "analysis paralysis". There is absolutely no tangible functional value created with the steps mentioned above. There may be some organizational value but that can almost be equated to bureaucracy! It is not surprising then that analysis paralysis is commonly referred to as an anti-pattern. All the analysis though takes a lot of time and since it is not producing tangible results is, in effect, creating a "value" debt for the project which will always be looming large.

Even if you were to be brutally honest with every dependency analyzed (which is pragmatically impossible), the date you come up with is never going to be accepted because somebody up the hill won't be able to fit it into his presentation. The inevitable advice will be to slash the estimations which will end up making the already shaky estimates absolutely and firmly unattainable. That the dates don't work is now a open secret in every sense of the word. That said, we live in a dysfunctional corporate culture where you are discouraged from being forthright with the messenger always being the first guy to be shot. The value debt and the culture together setup this cult-like delusion where the dates are committed to in unanimity.

Now the question is how long is this charade going to last? Most likely till the the first significant milestone in the project. The next obvious question is who is the guy who is going to blink; who is the chicken? Most likely it is going to be a "not so experienced" project manager or a service vendor. The chicken in most cases is crucified and accused of every incompetence conceivable under the sun. Once the chicken has been labeled an outlier, shot and killed; everyone else readjusts the dates and the game starts over to find the next chicken.

The real problem with Schedule Chicken is that it delays the inevitable, hurting the project schedule and driving up the cost. It prevents the stakeholders from looking at the root cause of the problems and taking corrective action early on.

The bottom line is that the game of chicken will not only disembowel the chicken but will also nuke the project. Schedule Chicken is a dangerous game to play!