Software isn't hard, thinking is hard!
"The essence of a software entity is a construct of interlocking concepts ... I believe the hard part of building software to be the specification, design, and testing of this conceptual construct ..."
Frederick P. Brooks Jr.
Brooks suggests that the creation of a conceptual construct is the "irreducible essence" of software. Four properties contribute to the difficulty of creating such a construct: complexity, conformity, changeability, and invisibility.
Presenting, and dismissing, a number of innovations and practices that address the merely "accidental difficulties" of software; Brooks explores three approaches that he feels have potential to address essence.
Only two approaches actually deal with the problem of conceptualization, of thinking: "requirements refinement/rapid prototyping" and "great designers."
Requirements refinement and rapid prototyping helps people think about what the software is to do. It also allows developers to extend their conceptual knowledge from what they fully understand to incorporate what they understand only in part or not at all. Feedback is the key element enabling this to work.
When Brooks talks about this approach to the conceptual essence, his ideas and descriptions sound a whole lot like agile. This gives us at least a glimmer of how agility might make conceptual model development, and hence software development, easier. A silver-coated bullet perhaps?
"Great designers" are essentially people that know how to think, how to create.
Unfortunately Brooks does not provide any insights on how to help people think better. In fact, the one idea that might actually help is dismissed as a minor improvement to an accidental difficulty. I am speaking, of course, of objects.
While it is probably true that object-oriented programming (abstract data typing, inheritance) cannot make a dent in the complexity of a design - thinking with objects can!
The greatest value of the object metaphor - decomposition based on responsibilities and not functions or data entities - is precisely the kind of tool that leads to better thinking. Objects have the potential to reduce complexity by reducing the number of "parts" and dramatically reducing the number of interactions among those parts.
Great designers, even if you had them, are still not enough. You need great teams. You need teams that are able to develop a collective shared conceptualization with the further caveat that the shared concept must be a great design.
Brooks provides no insights here, but perhaps Peter Naur can help.
Get your mind out of the gutter.
Can thinkers, even great thinkers, think about the wrong thing? Absolutely.
Peter Naur makes the claim that most if not all of the thinking we have done about software and about software development has been misdirected. We have been thinking about the wrong problem.
According to Naur we have been thinking, for fifty plus years now, about artifacts and the "production" process of constructing those artifacts. We have done this in the face of overwhelming evidence that this kind of thinking "leads not to edification."
Moreover, according to Naur, this kind of thinking is inconsistent with empirical evidence of what people actually do when they successfully develop software.
Thinking about "production of software and certain other texts" does not and cannot tackle Brooks' conceptual essence. We are wasting our time thinking about accidental difficulties. We need to escape this rut (gutter).
What we need to think about is theory: ala the philosophical ideas of Gilbert Ryle. A theory of "how certain affairs of the world will be handled by, or supported by, a computer program."
Naur's idea of Theory is precisely the conceptual construct that is THE essential difficulty of developing software.
How developers gain skill in theory building is not addressed by Naur. Nor is the problem of how a group of developers obtain a common theory.
Most of Naur's paper is devoted to illustrating how the lack of theory impedes software development (especially modifications) and how the presence of a theory allows developers to accomplish tasks with relative ease. (He also points out that developers who are theory builders should have higher status and pay: desirable always.)
There is a second way that our thinking has been misdirected. Instead of seeking a concept of affairs of the world and how they would be supported by software, we spend most of our time thinking and building mental models of how the computer is going to execute our program. Our conceptual model is filled with algorithms and data structures - neither of which have anything to do with affairs of the world.
The object metaphor was supposed to change this orientation; so too the agile story (as we will discuss a bit later).
Naur reinforces the assertion that the conceptual model is the essential requirement for successful software development. Naur also demonstrates that this conceptual model is indeed conceptual - it exists only in the minds of the developers and cannot be reduced to documentation.
Theory exists in the heads (plural) of developers (plural). It exists in the same heads that are subject to Millers Magic Number 7 +/- 2, the same heads that Brooks claims are incapable of holding such a large and complicated model - at least consciously. So how is this apparent paradox resolved?
Group Think - Hive Mind
"... thinking doe not take place inside our heads but is an activity that we perform with our heads. Most of our mental activities need external resources, and very often thinking is merely a grouping or regrouping of objects in our environment. ... humans basically use artifacts to acquire knowledge and create meaning rather than to represent it."
Reinhard Keil-Slawik
Keil-Slawik is also concerned with theory (conceptual models) as a pre-requisite for successful software development. He makes many of the same arguments as Naur to explain why representational documentation[1] cannot contain theory. He also grounds theory firmly in a physical context.
"A gestalt [synonymous with theory for our current purposes] emerges when certain objects or phenomena in the environment are related to each other in a meaningful way. ... the perceived gestalt is a construction of the observer. ... we perceive the world by constructing meaningful relations (gestalten). Consequently, we can only perceive what we construct."
Reinhard Keil-Slawik
These insights open the door to an understanding of how groups of people - sharing the same physical environment - can establish a collective gestalt (conceptual construct) - a theory - that can then be used as a foundation for software development.
A collective understanding - utilizing an entire context of external resources, organized into what Keil-Slawik calls an "ecological model of iterative learning - is more than sufficient to deal with the scope required by Brooks' conceptual model.
We now have all the pieces to ask what agile practices can contribute.
Agility is a Silver Bullet
Most of the agile practices, when used in the right way for the right purpose, directly attack the essential difficulty of software. They can be a silver bullet!
Let's examine some of the agile practices from the perspective of how they contribute to theory building.
Consider the agile environment: big visible charts (information radiators), walls with story cards, whiteboards full of drawings and notes, a build-lava-lamp, food, rolling-chairs on hard floors, and computers of course. The agile environment provides exactly the kind of context full of "artifacts to think with" that Keil-Slawik argues is necessary for a group to successfully construct a shared gestalten.
Think that rolling chairs on hard floors are an irrelevant detail? People are tangible artifacts too, and rearranging them along with the artifacts does enhance theory building. Context matters, gathering people around the story progress tracking wall matters just as much as physically handling the story card when doing a stand-up meeting. Meaning arises from the arrangement of people and things in the environment.
Compare this with the sterility of Cube World. Although an individual might have a sufficiently rich environment for her to come up with a theory, it is almost certain that the theory will be about the computer and not an affair of the world. There definitely is insufficient context for a team to build a consensus gestalt.
How about electronic artifacts - using RallyDev and Rose and similar tools to create a shared online environment? Maybe, but only if everyone has a 50" monitor and can have lots (lots!) of windows open at the same time along with the ability to rearrange the representations in those windows with the same flexibility afforded by a physical environment. You also will need to mix in video and audio feeds to get the people into the virtual world. Maybe, but not likely.
A story is not a requirement. A backlog is not an inventory.
A story card is an evocative device[2] that recalls to mind past conversations and context that are far too rich to capture in any representational model. [A story card can also be a reminder of the need for future conversations.] A story card is an exemplar of the kind of external artifact necessary for gestalt formation.
A story is a fragment of a theory. A backlog of stories is a model of the current understanding of a theory - or a subset of that theory. The product backlog, captured and displayed as a set of story cards is a visual and tactile artifact of the group's current understanding of a theory. A sprint backlog is the same with added focus.
Stories are written by customers/users because the point of theory is an understanding of an affair of the world. Software developers, traditionally, have very little understanding of the world; and, stereotypically, have very little understanding of affairs.
Stories are written and told to develop and expand a whole-team gestalt. Stories are not created to be units of work that enable Scrum project management.
Pair programming is micro-theory building using tests and code as the external artifacts.
Each pair extends the shared theory with character development (objects) and dialog (messages, object interactions) in service of the overall plot (the affair of the world) being advanced by the code. Judicious mixing of pairs ensures that the entire group shares in the same gestalt. Collective code ownership accomplishes much the same goal - ensuring that everyone shares the same theory.
Even something as prosaic as a burn-down chart can contribute to theory. If we truly have a theory we know things about the world and about the software - including how difficult it will be to translate a world thing into a software thing. Burn down charts, along with similar charts for estimates, provide the feedback that confirms - or denies - our current theory.
Theory is dynamic in at least two dimensions. First, the theory is about an affair of the world and hence is dynamic to the same extent that the world is. Second, theory is emergent over time, as it expands and morphs. Embracing change, iterative, explorative, and incremental development - the core of agility - is the only way that theory can be constructed.
Coulda, Woulda, Shoulda
Agility and agile practices, properly construed and used, are a silver bullet!
Unfortunately, most of what passes for agility is merely the application of techniques and method to solving the accidental difficulties of software and the production model of software development.
Agility and agile practices, properly construed and used, along with object thinking (not object-oriented programming) simultaneously simplify the prerequisite conceptual construct and help people to become better at formulating such constructs.
Agility yields the theory and the rest is merely software engineering.
References
Brooks, Frederick P. "No Silver Bullet: Essence and Accidents of Software Engineering. Computer, Vol 20, No 4 (April 1987).
Naur, Peter. "Programming as Theory Building", reprinted in Cockburn, Alistair, Agile Software Development. Addison-Wesley Professional. 2001.
Keil-Slawik, Reinhard. "Artifacts in Software Design" in Floyd, Christiane, ed., Software Development as Reality Construction. Springer-Verlag. 1992.
About the Author
[1] Representational documentation - Western culture, since the Age of Enlightenment, has believed it possible to construct formal written or mathematical models that represented reality. This general belief is exemplified in computer science by the belief that you can formally capture requirements, specifications, and models and that these documents are sufficiently representative of reality that they carry all the semantic and syntactic information necessary to create the software.
[2] Evocative Device - human memory seems to be highly associative with the added feature that single part of a memory can evoke - return to conscious attention - the entire memory. Smell is often considered to be the strongest evocative device with a mere wisp of cinnamon in the air causing a flood of memories about mom and baking. Religious icons are another example - recalling to mind complicated stories and myths.
