The Origins of Agile Programming

The Agile Manifesto is common knowledge now. I had always assumed that its origin story was somewhere in the early 90s or late 80s. But in the book I'm reading I now see that it started far earlier. In the Basili book I'm reading, this comes from page 30 which is a paper from 1975:

1. Any difficult in design, coding, or debugging a modification should signal the need for redesign or receding of existing components.
2. Modifications should fit easily into isolated and easy-to-find modules. If not, then some redesign is needed.
3. Modifications to tables should be especially easy to make. If any table modification is not quickly and easily done, then a redesign is indicated.
4. Modifications should become easier to make as the iterations progress. If not, then there is a basic problem such as a design flow (sic I suspect it should read flaw) or a proliferation of 'patches.'
5. 'Patches' should normally be allowed to exist for only one or two iterations. Patches should be allowed, however, in order to avoid redesigning during an implementation phase.
6. The existing implementation should be analyzed frequently to determine how well it measures up to the project goals.
7. Program analysis facilities should be used whenever available to aid in the analysis of the partial implementations.
8. User reaction should always be solicited and analyzed for indications of deficiencies in the existing implementation.

While there is much to fault in this model, it clearly emphasizes the need for frequent refactoring of code as design flaws are recognized and the iterative cycle with far shorter sprints than was common at the time it was written. Elsewhere he talks about decomposing the requirements into a list that can be implemented incrementally. The domain of this paper was the creation of a compiler. The increments were the specific language features. So each release of the compiler would be complete for that set of language features.

Clearly the years and our personal experiences have brought us to this belief. I'll be curious to find the empirical evidence that supports this belief.

Starting a New Book: Foundations of Empirical Software Engineering: The Legacy of Victor R Basili eds Boehm, Rombach, Zelkowitz

I expect this book will represent a form of manifesto for the research group I hope to work with during my PhD studies. The basic outline of the study of Empirical Software Engineering is laid out in a paper Basili wrote in 1996 and is the first one presented in the book. Here is a key point:

"First we define some terms for discussing experimentation. A hypothesis is a tentative assumption made in order to draw out and test its logical or empirical consequence. We define study broadly, as an act or operation for the purpose of discovering something unknown or of testing a hypothesis. We will include various forms of experimental, empirical and qualitative studies under this heading. We will use the term experiment to mean a study undertaken in which the researcher has control over some of the conditions in which the study takes place and control over (some aspects of the independent variables being studies. We will use the term controlled experiment to mean an experiment in which the subjects are randomly assigned to experimental conditions, the researcher manipulates an independent variable, and the subjects in different experimental conditions are treated similarly with regard to all variables except the independent variable." (p4)

Of course he is describing basic empirical study that is no different than any other scientific discipline. What is remarkable is how remarkable that it is directed at software engineering, a field that traditionally was led by ideas that sounded reasonable but were never subjected to any form of empirical study.

On Non-Functional Requirements, Martin Glinz, 15th IEEE Intl Req Eng Conf

This small paper tries to tease apart a good definition for non-functional requirements. He makes a good point in the dependence of classification on how the requirement is represented. For example, there is a difference between stating "The data shall prevent any unauthorized access to customer data" from "The database shall grant access to the customer data only to those who have been authorized by their user name and password". But note that the second requirement is more restrictive than the first and includes an implementation detail that was not present in the first; username and password. There was an implied design step between the two that introduced functional requirements to meet the original non-functional requirement. When authorized by the username and password, the system grants access. The first requirement leaves that detail unspecified allowing the designer latitude in how the requirement is met. The first requirement could be satisfied with retinal scans or other bioinformatic data that could no longer become part of the system specified by the second requirement. It seems that all non-functional requirements ultimately reduce to functional requirements and this makes sense since they must be implemented using the formalism of functional design. What makes the study interesting is doing this decomposition of non-functional requirements into functional specification in such a way as to aid in the construction of test cases that determine whether or not the requirement was met.

Where I seem to be going in my interest is being rigorous in the input to the design process so as to provide a solid base for the more rigorous look at the design process itself. The outcome of the design process should ultimately be fixed by the validation of this product against the specification and the verification that it behaves as intended.

I had a conversation with Earl in which he agreed that functional versus non-functional seems to make little sense in most usage. But he resisted my assertion that there is any relationship to any notion of a mathematical function at its core. His resistance is fueling my desire to get to the bottom of this so I can turn back to a more rigorous understanding of what is meant by software "quality."

In section 4.2 this paper offers a definition that embodies the point I was trying to make with Earl.  "A concern is a matter of interest in a system. A concern is a functional or behaviorial concern if its matter of interest is primarily the expected behavior of a system or system component in terms of its reaction to a given stimuli and the functions and data required for processing the stimuli and producing the reaction." Where I think this moves the conversation forward is the way it brings attention to the stimulus/response of the system as the primary differential. After hearing Earl's comments I feel I need to gain a better grasp of the formal definitions but informally in the undergrad curriculum functional is always defined in terms of a component that accepts multiple input variables and returns a single result. It is defined by its signature and the unit of abstraction for design. If I'm allowed this definition of function for the moment, a functional requirement is one that specifies the response given the state and input to the system. Simply stated then, the functional measure of conformance to the functional specification is whether or not when the system is in the specified state and presented with the specified input, if the system produces the specified response.

Of course this type of functional specification is one that the paper refers to as a "hard" requirement (I believe) and one that is either always met or it is judged to not meet the specification. I infer that what the paper means by "soft" requirement is one where some probability function is specified that defines the tolerance allowed for the behavior.

This leads me to two points that I believe need to be explored. The first point is the temporal aspect of behavior and the second is how systems and requirements are decomposed or aggregated in a complex system.

There are advanced logics that address the temporal component and I had briefly studied one of the temporal specification languages that addressed these temporal concerns. However it is clearly understood in the literature that these languages stand apart from the more common ones which do not. To be able to distinguish between a deadlock and an underpowered processor is vital in analysis of design. But at the level of specification or requirements it is expressed as a constraint on the functionality at some level. Perhaps an interrupt is only presented for a fraction of a second. The system must recognize and remember the interrupt within that time parameter or the system will not function as intended. OCL and other specification languages allow this additional information to be captured in the specification model. But I think it is important that it supplements, not replaces, the stimulus response aspect of the system.

The other aspect of the specification is how it is expressed at different levels of decomposition. Something like the overall usability of the system has a cascade of specifications as it is decomposed into the design. When the requirement includes the requirement to allow an arbitrary number of "undo" steps in the system, it begins to constrain various sub-specifications to the parts of the design. This includes both behavior and information representation choices and even has business implications that the user must be made aware of as the ability to backout a transaction may entail the "undoing" of later transactions that depended upon this state of the system.

I think the levels of specification and behavior that I see can most easily be seen in the difference between an requirement-in-use like the usability requirement from above, and a requirement-of-ownership like maintainability.

The system that is being specified for a requirement-in-use is not just the software but also the software plus the hardware and human environment in which the software is deployed. There is a continual surprise when testing fails to uncover an error from a common business transaction due to the lack of sufficient understanding of the domain compounded with the lack of end-user involvement in the specification and test preparation for the system. To think of this as only a software requirement will perpetuate this mismatch and ensure we do not find a solid foundation. But most important is to note how this differs from the requirement-of-ownership like maintainability.

In use, the system of interest is the production software, hardware and business environment. This environment either does not include the human organizations concerned with the future versions or the planning for that change. It does not include the development environment. Even more dramatically for a compiled language the system-in-use does not even require the source code. These are two distinct systems that overlap in the production software code and some interfacing organizational units. At the macro level the metric for a software change includes the variables for the size or complexity of the change, the skill set of the resources to be allocated to the change, the development environment deployed to support the maintenance and the required time and cost for that change. The response would be quantified by the achievement of some level of MTTF or defects detected against some standardized test bed.

These definitions are not just academic exercises. They become the foundation which supports the engineering of the system and the basis for an unbiased and objective assessment of whether the system met its specifications. It is these specifications, and the attributes of the product produced from them, that provides the basis for the assessment of the product composite quality.

Towards an Improved Definition of Software Quality

This is just a short rant to mark what I intend to become a full essay on the definition of software quality. First there is this website (http://www.improvingsoftwarequality.com/2011/10/qtp-tfs-generic-test-integration.html) that I found in a Facebook post from http://www.facebook.com/pages/Software-Engineering-Microsoft/174748642567856. The page is crowing about the integration of QTP into MS VS. What raises my blood pressure is the casual way in which this page and many other references to software quality seem to raise it as some totem, that if they just say it improves software quality that everyone will believe them. That is just such an overblown and fallacious suggestion that I feel it borders on fraud. So please let me explain why I think this does the software engineering profession no good.

First, software quality is not a single property of software but at best some weighted sum of many constituent properties. The Japanese automakers developed this very nicely decades ago and it has been well developed for software by Carnegie Mellon's SEI. I really don't understand why I continue to hear people refer to it in the singular when it is more appropriate to speak of qualities rather than the singular quality. To use it in the singular with no qualification is to signal a disregard for any form of empiricism since this has as much weight as hand waving.

Second, quality is often meant to be something that is not purely of the software artifact but the software artifact embedded in a given or assumed context. Take performance as an example. Clearly a piece of software that does not perform acceptably on a weak platform may perform adequately on another. It would be wrong to speak of the software as possessing the quality of performance when that attribute only exists for the software/hardware combination.

It might be argued that a property of modifiability is purely a software property since no hardware is required to change the pure software absent its embodiment on a hardware platform. But even here there is an implied context. Modifiability by whom? in what way? against what standard? What is intended in most cases is that the software be successfully changed by an assumed skill level in an understood organization within some unstated unit of time with some unknown defect level as measured over some indeterminate period of time following its installation. That's a lot of unknown information and it is clearly context dependent since the same piece of software could be judged modifiable in one environment and unmodifiable in another. So when it is alleged that the integration of a software testing tool into an IDE will improve quality simply because it makes it more efficient to test the product, I am not convinced.

The ultimate irritant in this fluff piece is the clearly fallacious argument is that the improved environment will have any effect on the program quality at all. The quality improvement comes from finding the ways in which the behavior of the product does not match the specification for that product. Making the testing process more efficient certainly improves the software economics. But that is more an attribute of the software development process and not of the artifact itself. One must assume that to improve the quality of the artifact that the engineer will improve the testing. To assume that testing quality improves because of the time saved due to better integration of the tool set is clearly wrong.

Software quality is clearly a term that must be more adequately defined if it is to be used in any context within ESE. I will begin research finding who has attempted to define this besides SEI and consider expanding this rant into an essay.

Factors that Affect Software Systems Development Project Outcomes: A Survey of Research

(contd)
This paper touches on a vital subject on page 8 under project outcomes. In the most understated sentence of the paper the authors say "In general, there remains a lack of consensus on how to define success, lack of success, and failure." In my commercial work, this was rarely done well and often remarkably badly. I remember being the junior member of team from a subcontracter under one of the most respected management consultants in the country discussing the initiating phase of a major project for a F100 company. I naively asked what benchmark they had established against which to measure the success of the project. You could have heard a pin drop in that room. There had been no thought to gathering any baseline of the existing performance of the system despite the clear need to demonstrate a hard financial benefit from the project. There are many times that metrics matter and in this case they mattered a lot. We needed to discuss the various ways that success for that project was going to be measured and take the time to gather the basis for comparison. An this was one of the successes.

Too often a management team will not be interested in doing the hard work of reifying their needs to the point of any metrics at all, no less a set of metrics that will withstand the intense scrutiny that is shone on a failed and expensive project. Yet for corporations that invest great amounts of time an effort in management information systems and detailed cost accounting it is astounding how often major capital projects are undertaken without any regard for how success or failure will be measured. Is it any wonder that when the way the success will be judged that success is sometimes a reconstructed reality post hoc?

In summarizing their survey, these authors offer several important dimensions: technical, economic, behavioral, psychological and political. Clear above I was focusing on the economic which can lend great clarity to decision making when it is the driver behind the project. With some reflection I can think of instances of the others. For example, the installation of some form of middleware or infrastructure is clearly a case of the technical dimension. While the project may run over budget or time, those are largely secondary since the need is to establish that technical capability for many other projects to leverage. While it might be desirable to quantify the payback on this kind of investment, it is rarely possible to do that easily and is often not worth the bother. For these other dimensions, I'll be interested to read the cited papers to get a better idea of what they have in mind.

I naively believe that the central point of success or failure can only be measured against the motivation for starting the project. If that objective has not changed over the course of the project, a well thought out and reified objective should provide many points to argue for the success or failure of the project. Where there has been failure, they should also provide some trace-ability back to the decisions that caused the outcome and objective to diverge.

**************************
I am rethinking my plan of working through this paper at this depth. This lays out a very big forest to me and while I am looking for a few good trees to prune, I can't complete the survey in one or two sessions. I intend to refer back to this paper throughout the year as I try to survey the various areas within the scope of ESE.

This raises an issue I have been struggling with to understand where ESE fits in. Where is the line to be drawn between Economics and Software Engineering? Clearly Boehm made a significant name for himself at this intersection. Since SE is about purpose built products it is hard to imagine that economics are ever too distant. Yet how risky will it be to invest myself in looking at the utility curves of the various choices that can be made in development?

Factors that Affect Software Systems Development Project Outcomes: A Survey of Research

by McLeod & MacDonell from ACM Computing Surveys, Vol 43, No 4, Article 24, Publication date: October 2011

This is the next paper we will discuss and the one that I most viscerally connect to. I'm not sure how we will discuss it since it's scope is so wide. Yet out of this scope I see so many opportunities for further research that I'm overwhelmed. The biggest challenge I'll face will be finding a small enough topic to support my research without it blowing out of control.

Since I don't have enough experience with this group to know how Prem will want to parse this paper I'm at a loss to know how to prepare myself for the meeting when we get together in two weeks. So I'll first pass over this paper contemplating my interest in each of the areas the authors identify. I'll go back afterwards and consider their methodology for finding these papers.

The novel contribution of this paper is the inclusion of a category the authors call People and Action in addition to the categories of Development Processes, Project Content, and Institutional Context. While I am personally in touch with these factors as vectors to be studied in the analysis of project success predictors, I was surprised to see them included in a paper that we are studying since I had thought these might stray too far outside of ESE. Perhaps they do. That is a question I will ask when we meet. Much of this overlaps with management, especially project management (Drucker) or social systems, group dynamics, etc. There is no doubt that all of these can impact project outcome but where will we draw a line between these other disciplines and ESE? What is the tolerance for the journals and conferences for a paper on the impact on project budget due to cultural collisions? That sounds so completely social science to me that I find it hard to see how some of the major venues will accept it. But I have a lot to learn.

The authors call their first category Project Content and consider these properties of the project itself. So the charter, mission, scope, and goals to be Project Content factors. They also include the resources of the project (why do they phrase it as "the resources it attracts"?) To them, this includes the software and hardware used (to me, the tools). To me this would include the IDE, any CASE tools, the languages adopted, the processing power dedicated to this project. For this survey they exclude people since they are included in the new category.

Their second category is Development Processes. It is surprising to me that they choose to place requirements determination in this category while including the goals, and scope in the prior category. I have always understood requirements to be an elaboration of the mission and goals of the project extended into more concrete terms. However I do clearly understand their placing things like the method to be used (methodology) which comes from outside the project. There is clearly some overlap here between processes and external constraints. A methodology may be imposed as an institutional constraint or it could be something that is chosen by the project manager since there is no imposed requirement for a given methodology by the organization. It isn't clear to me how the authors chose to deal with papers that may arguably be within these overlaping areas. But if their intention is simply one of making sure they included all the papers perhaps the categorization is not important.

From a system's perspective I can understand how this is a workable categorization for the authors' purpose but there are some nuances that I think are important for the very large organizations. First, there is not a single layer of projects within organizational contexts. A project may be part of a program which itself may be within the context of a long-term architectural plan for the organization. Also there are various levels of organization which are often involved. The users may belong to a department in a business unit which is part of a subsidiary that is owned by some conglomerate. It is not uncommon for a project manager to strategically chose to ignore some organizational constraint. He may know that there is no enforcement of that requirement, have more power and influence in the organization than the area that is trying to impose that requirement or perhaps by making an appeal to the inapplicability of that constraint on this project. This is analogous to Daryl's paper on ecology where the proper level for the analysis must be found before it becomes meaningful.

One aspect of this hierarchical embedding intrigues me. We are spending more time talking about software systems. I always take this to really mean software-intensive systems since I am hard pressed to think of any software system that provides value in the total absence of hardware. (The very fact that we can now study software divorced from the hardware that it runs on says volumes to me about how far we have come). So there is a software embedded in hardware and it is this combined system that generates some value. However from a change management perspective it was never sufficient to allow this to be the only scope of study and analysis. Any system is contained in some larger context and it is often the qualities in use that are important for analysis and measure. This necessitates including at least the people who directly interface with the system and possibly others. The focus becomes tasks that are done with the assistance of the system. How else can usability be understood if not within this context?

While the organizational scopes add an interesting way of parsing out utility from the system, some qualities are only understood when viewed longitudinally. Modifiability, to name one, can only be measured over a period of time that includes enough transformations of the system to make some observation about how well this system meets the quality requirement it was given.

(more later)

One sentence sticks out to me on page 5 at the bottom. "While generic influences are common to a range of systems development contexts, they manifest themselves differently in specific situations." I suspect I'll need to read the cited paper to understand their point.

Reflexive Design Thinking: Putting More Human in Human-Centered Practices

This is the title to a piece a friend wrote that appeared in the mar/apr 2011 Interactions mag from ACM. I'm fond of this piece because it calls out the centrality of human group processes as an important factor in project success and the danger of an approach that is too narrowly linear and deconstructionist.

In this piece Friedland/Yamauchi also give voice to another process truth I had also read about and feel must be true. While a waterfall approach assumes we explore the problem space and come up with a statement of need that is then fixed and forms the basis for the exploration of the solution space, it is clearly not the way it happens in practice. If it were so, we would not have chestnuts like "to a man with a hammer, everything looks like a nail." Our tools (using this in the broadest way possible) shape how we at least conceptualize, if not see, the problem at hand. The exploration of the problem space is done relative to the solutions we know about. This allows us to quickly dismiss solutions that appear completely unworkable and focuses our attention on the gaps between the prior successes and the current challenge. It becomes more of a survey to continue exploring the contours of the solution we know we can build to see where the problem fits or does not fit that contour.

This parallel activity of refining the problem statement while holding onto prior successful solutions helps me come to grips with a dysfunction I have seen far too often in practice. Very large products often have sufficient plasticity that they can solve problems outside the domain in which they were designed. A case in point is an insurance policy administration system I worked with that was sold to senior management as a solution for both a travel product, a membership product and as an enterprise billing and payment solution. My immediate reaction was, yes, it CAN do that. But SHOULD it? The argument that ultimately won the contract was the simplification of the relationships that needed to be managed to complete the project, the operational simplicity of the finished product and the cost savings that would accrue to the organization by that decision. The vendor had developed enough good will that they were judged to be someone who could be trusted to deliver. The project was generally successful in those high risk areas but I have never found that solution optimal.

What I take away from this is how senior management must deal with exceptional levels of complexity and they are as inclined to abstract their cognitive model of the organization as a programmer is to abstract large collections of code. Clearly concerns like the cost savings of consolidated operations and products can more than compensate for minor inefficiencies of using products that are not necessarily the best of breed. However the trade-off point is hardly an easy one to analyze and argue with hard facts. These decisions either come down to management fiat or they happen because of persuasion from a committed staff member. Sometimes these arguments are won by pure salesmanship.

A major insurance company I worked for had a large installed base of Sybase databases and had negotiated a very favorable contract with Sybase for new licenses. This made the financial justification for new projects very easy when there was a database component to the solution. As far as I can tell, Oracle was able to convince the most senior management that Oracle databases were their future despite the lack of any clear functional superiority to the product. Even worse, the favorable pricing for marginal licenses was lost and many projects that could be economically justified under the old schedule became economically unfeasible with the new schedule of pricing. Even worse was the cost to the organization to convert the existing databases, a process that would take them many years to complete and at a cost that would exceed any immediate short-term savings. Perhaps it was someone's bonus for the year that was saved but it was not the right long-term solution for that company.

So to get back to the article, the conclusion emphasizes the importance of imbuing the organization with a "designerly way of knowing." I don't disagree but I don't find peace in this statement of the conclusion. What I feel happened is the release of energy that became possible when the staff was empowered to build the environment they wanted instead of the one that was designed for them. Most often, a designed environment is constraining and not empowering. A particularly stark example of this is how Frank Lloyd Wright approached his buildings. He attempted in some cases to not only furnish a structure but the furniture that would go into them with floor plans that virtually dictated where everything would be. This can be acceptable design for some people and some homes. But to inspire maximum creativity for a staff this is anathema. You would never put children into an environment that gave them no options for play. The mind must be free to assemble the pieces in experimental ways, ways that can fail as much as succeed. I see the lesson learned here as one that creates the right environment and empowers the staff to modify it as they need. But I know I am only playing with the semantics here.