May 10, 2010

Agile Needs Documents for Visualization

In the course of trying to make people aware of my software documentation product, I sent out emails to authors of software development books offering a free evaluation download. One of the authors was not interested, and replied: "Our team communicates by word of mouth and shared experience".

In my twenty plus years of software development, I've seen some changes—most of them good—but when we skip documentation, I believe we're abandoning something important.

Every technological success of the modern world has been accompanied by a modeling notation. Electronics, architecture, civil engineering, electrical engineering, mechanical engineering—all have their sets of symbols and diagrams. The diagrams are used to communicate, and, perhaps more importantly, to visualize. There would be no television without circuit diagrams.

Software development has its modeling notations also, and diagrams can be used to engage the special ability of our brains to visualize. Documents containing diagrams let us share the design visions we create.

Moving from stories to pictures

I went to the store, bought some milk, drove home, and poured the milk on a bowl of cereal.

That's a story. Lines of code are also a story. When a design is communicated only by conversation and experience, it can only be a story—you can't tell a picture!

As soon as we put our lines of code into methods we have something we can draw pictures of: the calls between methods. Object-oriented development introduces class hierarchies—also something pictures are useful for.

Some code has so little structure that there's not much to draw pictures of. That's too bad, because structure is what we need to control complexity, and complexity doesn't scale well.

Here's a pretty obvious example that compares a story and a picture version of an org chart.

It's naturally much easier for us to comprehend the picture. And this is just a small example—imagine trying to understand the story version of an org chart three times this size.

I've included this simple example to demonstrate that our brains have the ability to quickly grasp the relationships expressed in a picture in a way they cannot from text or conversation. Let's call this special ability visualization.

Visualization

When I think of Microsoft, I get a mental image of a US map that's focused on Seattle, where their headquarters are. When I think of Apple, my brain places them in Colorado. Don't know why—I know their headquarters are in California—but that's the image I get.

Our brains automatically create mental images of abstract things and we hang our thoughts onto these images. We want to exploit this powerful mental faculty as much as possible when developing software.

When I'm working complex code, with lots of data structures, class hierarchies, and interactions between them, I take out a sheet of 8 1/2 by 11 paper and, with a pencil, sketch out data structures, class hierarchies, state diagrams, whatever. Then I add lines for relationships, and sometimes step through an algorithm, modeling the changes on the paper as I go.

This is not so much a document as it is a thinking paper. It's a power multiplier for my brain, allowing me to see relationships, and track states that would otherwise be vague images in my mind.

When I'm done, I usually tuck it away and never look at it again. But the locations of the structures I drew are etched in my memory. They will be referenced, and provide a powerful thought organizing effect each time I think about the problem in the future.

Rather than tucking the sketch away I could share it with a group. Maybe spruce it up a little, but not much. Then we can all work from the same mental image.

The point I want to make is that using the power of pictures not only enhances communication, it leverages the very special ability of our brains to visualize. Applying the power of visualization propels our designs to a higher dimension.

Some documentation guidelines

If documents containing pictures could appear out of nowhere, without the cost of time and money it takes to produce them, I hope we can agree that they would add value. So why would the author exult in the freedom from documents, preferring instead "word of mouth and shared experience"? Because documents don't appear out of nowhere, and the process of creating them is often misguided.

There are things we can do to make documentation more useable and cost effective. The first is to correctly identify, and always work towards, a purpose. The purpose will determine what documents are created and what the content of each document will be.

The purpose of documentation

The actual source code for a project, and, to a certain extent, the design of the code, is usually unseen by the managers who evaluate and promote. Documents, on the other hand, are visible and seen by managers. It's not unusual to hear someone in a status meeting report on their progress by telling how many pages their document is up to.

In many groups, there's a career path in producing large, formal documents and presenting them to the managers that value them. When that happens, the purpose of documentation becomes to impress. I believe it's this shift in purpose that gives documentation its bad name, making it something that's rebelled against.

To realize the value of documentation, we must always work towards a purpose. I would say the purpose of documentation is to promote visualization during the design process and to share the design visions we create with others.

When should you document?

Completion of the "design" (meaning documents) is often a milestone at the beginning of a project plan. For Agile development, we should reject this, but when should documentation take place? When we know something that would be worth the effort to document.

At the beginning of a large software project, I'm not smart enough to see the design through to the end, and neither is anyone else. Good designs emerge from repeated trials, research, and lots of thought. At the start of a project, we can document what we know: very little.

Often in the middle of a project, sometimes in the middle of the night, a better way to design something becomes clear. In an intense effort, usually lasting a few hours, I'll perform major code surgery, tossing code all around the room (figuratively) until the old code conforms to the new design, and the new design benefits from an understanding I didn't have in the beginning. This ability to change the design mid-stream results in a better design that will be easier to implement and maintain.

Unfortunately, documentation effort equals commitment. When we've spent a week or two writing and polishing a design document, we don't want to think about how the design could be improved, because changes would invalidate our documents. We lose the state of "flux" that's required to produce good designs, and go forward with a flawed design that will cost us at every turn.

We should never let the existence of a document keep us from thinking about and making design improvements. Of course, there are reasons not to implement every improvement we can think of (like a release date), but a design should not be bound by our commitment to a document.

At the end of a project is also a bad time to document because:

1. The documents are not available for reference during development.
2. Our memories of details will not be fresh.
3. Because of the lack of urgency and need to move on, it may not get done at all.

The best time to produce a document is during the design. Because an Agile design will emerge over time, your documentation should emerge over time also.

What goes into a document?

Sometimes a group will adopt formal rules about what goes into each type of document. Absent a legal requirement to do so, this is a bad practice, because the effort to fulfill standardized requirements subtracts from the effort towards the real purpose. I've seen large documents that consisted mostly of repeated boilerplate, and their size hid that fact that they contained little useful information.

We cannot come up with a set of rules that can make our software development work automatic. Like the rigid rules that expel first graders from our schools for accidentally carrying butter knives in their knapsacks, rigid rules that regulate document content are never a substitute for reason.

We must be actively thinking and making decisions throughout the process, and decide what goes into each document based on its purpose and the audience it's intended for.

What documents are required?

Software projects vary fantastically in size and nature and the quantity and type of documents that are produced should vary also. In my experience, a state transition diagram, for example, is almost essential for certain types of problems, but totally useless for most others. I worked at a place that mandated a state diagram be produced, even though it made no sense at all.

The documents that get created should be determined by needs that are identified throughout the development process.

Where should the documents reside?

In one, well known place. You would think this would be obvious, but sometimes just getting an out-of-date version of the database schema can take weeks of negotiation.

LumiCode

Earlier I suggested that "if diagrams could appear out of nowhere" we could agree they would be valuable. Even following the guidelines I've listed above, creating documents and diagrams will take up valuable resources. The cost of producing a document must be weighed against its anticipated value.

I would like to present a tool that I believe changes the cost/value equation by making the cost of producing some of the most valuable diagrams very low. The tool uses reflection to get information from .NET assemblies and the user interface is design to make it easy to pick what you want to see.

Here's a six minute movie that demonstrates the tool: Watch Movie. You can find more movies and information at: www.lumikon.com

When you've seen the movie, consider just how much was accomplished in only six minutes!

Conclusion

Agile succeeds because its prescriptions flow from the collective common sense of experienced developers. As an experienced developer, I've seen how visualization aids the design process. I've also seen how the communication of designs, through documents, can actually improve the efficiency of a development team.

Before we take a stand against documentation, we should be sure we understand what it is we object to. I believe it's not the documents, but the up-front, overblown, please the bureaucracy way they are produced in many development groups.

We need to be open to the idea that, with the right processes and tools, documentation can improve our Agile development process.