Messy Documents!

One of the tenants of the agile manifesto is working software over comprehensive documentation. A lot of people misunderstand this to writing no documentation at all. The manifesto itself goes on to say "That is, while there is value in the items on the right, we value the items on the left more". Every project invariably needs documentation and not writing any is a recipe for doom. That said, it must noted that document creation and their maintenance are expensive and need to be managed carefully.

Documents are ideal for maintaining organizational memory but are ineffective as a means of communication during project implementation. The primary issue of communication is one of understanding and not that of documentation. Transferring knowledge or ideas through documents is a skill, you might even consider it an art and this skill is definitely not easy to find. Even the best of developers find it difficult to write good documents simply because our educational systems don't allow for students to gain this skill. Effectively the question that needs to be asked is "do you need documentation" and not "do you want documentation". Documentation is needed if that's the best way to achieve the relevant goals, but there often proves to be better ways to achieve those goals than writing static documentation.   

I tend to agree with an agile modelling approach where a document is any artifact external to source code whose purpose is to convey information in a persistent manner.  This is different from the concept of a model, which is an abstraction that describes one or more aspects of a problem or a potential solution addressing a problem.  Some models will become documents, or be included as a part of them, although many more will simply be discarded once they have fulfilled their purpose.  Some models will be used  to drive the development of source code, although some models may simply be used to drive the development of other models.

A low-fidelity prototype of a system's user interface or a picture of a block diagram sketched during a meeting between developers or a short article on an internal wiki are real "information radiators" as compared to a twenty five page word document stuck on a share point site. Furthermore the sheer thought of the cost for laying out and formatting documents and getting them formally reviewed when they are just models which can be thrown out should be repulsing!

Traditional development methodologies emphasize greatly on a lot of documentation early on in the SDLC in the form of requirement specifications, detailed design and architecture documents and project plans. If you look at the list, most of these documents are speculative and would require a lot of effort to create and maintain since most of the project parameters would change substantially during the project execution. There is no real and  solid relationship between project success and writing comprehensive documentation, and in fact it is more likely that the more documentation that you write the greater the chance of project failure. Statics actually show that agile teams produce more quality documentation than traditional teams at a fraction of the cost.

The following list enumerates some best practices that are suggested by agile modelling techniques to use documentation effectively:

• Prefer executable specifications over static documentation. A Test Driven Design (TDD) approach or a Behavior Driven Design (BDD) approach to requirement gathering, architecture and design can go a long in producing Just-In-Time specifications for an effective implementation. With TDD you write a test, either at the customer acceptance level or the developer level (unit tests), before writing sufficient functionality to fulfill that test.  The tests are used for two purposes: they specify the requirements/architecture/design and they validate your work.  This is an example of the practice of Single Source Information.
• Document stable concepts not speculative ideas. The agile strategy is to defer the creation of all documents as late as possible, creating them just before you need them. System overviews are best written towards the end of the development because you know what you’ve actually built.  Similarly, a majority of user and support documentation is also best written towards the end of the lifecycle.  However, this doesn't mean that all documentation should be left towards the end.  It is a good idea to take notes for these sorts of documents throughout development so that critical information is not lost.  These notes may be nothing more than point-form information in code coments as there is no need to layout and format documents until just before final delivery.
• Generate System Documentation. Modern software-based modeling tools can reverse-engineer existing code and present a multitude of views into it. Doxygen is just one example!
• Keep Documentation Simple.The best documentation is the simplest that gets the job done.  Don’t create a fifty-page document when a five page one will do.  Don’t create a five-page document when five bullet points will do.  Don’t create an elaborate and intricately detailed diagram when a sketch will do.  Don’t repeat information found elsewhere when a reference will do.  Write in point form.  Document only enough to provide a useful context. Start with a document that's minimal enough for the needs of its customers then augment it as needed
• Write few documents with least overlap. This is the same as the principle of "Don't Repeat Yourself" or DRY. Strive to travel as light as possible, writing on just enough documentation for the situation at hand which is just barely good enough to fulfill it's purpose. Use wikis and hyperlinking to connect small documents to create a whole instead of rewriting.
• Display Information publicly. Create information radiators considering issues such as indexing, linking, and accessibility when writing documentation because the end customers of the documents are not always known. The greater the communication on your project the less need for detailed documentation because people already know what you’re doing.
• Treat documentation as a requirement. Always attach a business value to a document and prioritize them like any other feature in the project. Any time spent on documentation could have been potentially spent on feature development. By treating documentation as a requirement you make its creation a visible to the stakeholders to consider.  Fundamentally, the investment in documentation is a business decision, not a technical one: you shouldn't create documentation because your process says you should.
• Require that stakeholders justify document requests. Everybody involved must understand the total cost of ownership (TCO) for a document, and strive to maximize stakeholder ROI to provide the best value possible to the project. Stakeholders must explicitly choose to make an informed investment in the documentation. The benefit of having documentation must be greater than the cost of creating and maintaining it.
• Recognize That You Need Some Documentation. Documentation is as much a part of the system as the source code.  In addition to working software, projects need to minimally deliver user manuals, support documentation and system overview documentation. The agile team’s primary goal is to develop software but its secondary goal is to enable the next effort. Building high-quality working software which meets the needs of the stakeholders is important, but ensuring that the people who come after you can maintain and enhance it, operate it, and support it is also equally important. Documents still serve a purpose of capturing important information permanently.

Documentation is critical to the success of any project, too much is as bad as too little. Success hinges on getting just that right amount!