Teaching the uses and limits of philosophical rationalism for large scale commercial software engineering

ABSTRACT
By the nature of computer science as an applied math and its use to train our legions of software engineers, software engineering is deeply wedded to rationalism as the primary mode of thought. But that mode of thinking has limits that are not often realized and if taught at all are poorly taught. This short essay looks at the limits of rationalism as it applies to the development of large scale commercial information systems by software engineers steeped in this tradition. It concludes with some recommendations on the adjustment of an undergraduate program that would more directly address this shortcoming and give the new software engineer a larger mode of reasoning about how to approach the task. It also has implications for the graduate school programs and the directions that future PhD students could pursue.


INTRODUCTION
To keep this essay short, it is limited in scope. The focus is on the engineering of large scale information systems within a commercial context. This is for several reasons. First large scale systems cannot be created by a sole contributor. The inability for one person to satisfy the client need without coordination, communication and collaboration with peers creates a change in the techniques that can be applied. What was internal thought must now be shared and what was vague must be made explicit. Since language is the way this is done, the work cannot be done without artifacts and process.

We currently use software dominated computer systems for many purposes including entertainment and machine control to say nothing of research and artificial intelligence. But the historical domination of information systems gives us a rich history to rely on and a shared foundation from which to begin. This paper is largely dominated by that history of information systems that could be specified in formal ways.

An immediate application of any contribution toward the tasks of software engineering is the education of new engineers. This piece will not stray from undergraduate material that one might hope is universal in every university computer science program. Instead it will focus on the interdisciplinary application of that material.

We very briefly remind the reader of the philosophical concept of rationalism and its contrasting concept of empiricism. The two most common process models of software methodology are mentioned, waterfall and Agile-oriented. Then the remainder looks at specific ways in which we try to apply rationalism yet find it wanting in practice. The conclusion looks at these shortcomings and possible ways of restating the issue to offer the most promising alternative to a rationalist approach.


BACKGROUND
Rationalism has rich roots in philosophy. While it includes the philosophy of mind in this essay we look at the question of what can be taken a priori versus what must be learned. Constrained in this way we can apply that concept to many important software engineering areas including requirements engineering, specification, design and testing. While rationalism is seen in the rationalization of human process (methodology) and the intrinsic nature of formal languages and machines, it is also attempted in specific tasks such as testing and requirements engineering.

Large scale development without some degree of methodology, whether explicit or implicit, is prone to failure and not a good model for engineering. There will always be exceptions but in an educational environment we teach to what can be learned, not what depends upon an exceptional practitioner.

The discussion follows the most canonical progression of software engineering tasks as often taught in the waterfall methodology.

RATIONALISM IN ORGANIZATIONAL CHANGE
A common characterization of large organizations which consume the majority of commercial software engineering services is that the need or opportunity to improve the organization through some project is realized. While this alone may be interesting it is before the commencement of any recognized software engineering methodology. However we want to note that it is rare for a project to to commence with a completely blank slate either in environment or in the pre-formed attitudes of the people who will make the decisions. There is an a priori set of beliefs, processes, constraints and other limiting factors which will shape the project. Insofar as it is a purely rational process, some idealized process of project management might find the result predictable. The fact that it is not is the very truth we wish to examine.


RATIONALISM IN REQUIREMENTS ENGINEERING
We lie to undergraduates in the computer science program. We lead them to believe that the clients give clear requirements, (ok, perhaps we help a bit) and from those we create some high level design which is reified until we have the specifications for all the modules to be coded. If we are good at our jobs as software engineers, we take those specifications and implement some algorithm which satisfies the specification. In the best of all possible worlds, that satisfaction can be proven using static analysis giving ultimate certainty that as long as the hardware properly carries out the logic that the valid input will result in the desired output.

The failure of rationalism here is rather clear, clients are rarely capable of thinking in the extremely abstract manner that is needed to reduce a human need to a formal language. Yet they alone possess the domain knowledge and judgement to decide that the completed project is acceptable. They must be satisfied and a failure to properly interpret what is needed cannot be the fault of the client alone. The software engineer is responsible for the extraction of the necessary information.

The primary tool that allows clients and software engineers to communicate is language. This is not just natural language but also the various semi-formal languages of diagrams and models. Large scale enterprises often have extensive documentation of their processes which can form at least part of the corpus of support for a requirements document in a waterfall methodology or a reference for an Agile-oriented process. But these are not formal languages and they all suffer from the common fault of all non-formal languages of potential ambiguity. There is no way to teach how one proceeds with the reductionism of natural


RATIONALISM IN PROJECT MANAGEMENT
Agile methodologies have successfully challenged the command and control with linear process that is the waterfall methodology. One reason the waterfall is still taught is because it is rational and easy to understand. But the reality is that the pure method was more unsuccessful than successful in the industry. Rather than believe that the business analysts have completely captured all the success criteria for the system under development an Agile process assumes that the requirements are not complete and that only a working prototype will elicit the knowledge of what is lacking in the system. This is a far cry from the methodology of formal methods that depends upon formal predicates that form the needed post conditions the system must satisfy.

RATIONALISM IN DESIGN
"In Nicomachean Ethics, Aristotle claims that to discover the human good we must identify the function of a human being. He argues that the human function is rational activity. Our good is therefore rational activity performed well, which Aristotle takes to mean in accordance with virtue. " (Aritstotle's Function Argument from The constitution of Agency, Christine M Korsgaard, Oxford Univ Press). Many attempts have been made to rationalize corporate governance and process. But the reality of management is that it is often necessary to make decisions in the absence of perfect knowledge and this defines a limit to this attempt at complete rationalization. People will make decisions that if complete knowledge were available might be different. Such is the nature of being human. 

While systems are not humans, a similar argument applies, a good system is one which executes its function well. Function takes on a distinct mathematical definition in computer science and it is important to note that the word is qualified by "well." What does this mean?

In an earlier age we talked about functional and non-functional requirements. Those non-functional requirements are better called qualia in keeping with the work of CMU's Software Engineering Institute. So their argument goes the structure of function in a large system is not unique. There are a large number of ways to organize the ensemble yet still achieve the strictly functional specification in this narrow mathematical sense. But what can change is the other qualia of the system which can be vital to the ultimate satisfaction of the human need which motivated the project.

For large systems, the client becomes more difficult to define in the traditional engineering sense. It can be the person who pays the invoices, the person who casts the tie breaking vote when there is a disagreement over requirements, it can be the person whose business unit will benefit from the new system. Project management texts tend to call all of these stake holders in the project. There is some function of individual satisfaction that will yield a result that will "satisfice" for this collective client. But that is virtually never a well defined function. Again, there is no rationalism in the final decision regarding the intended design of the system, merely a large number of possibly contradictory or overstated requirements from many people who may disagree with each other about the true mission of the project and the system to be built. As in many other human affairs, rationalism often fails to bring consensus to a group.

Even when there is consensus on the various qualia to be achieved by the system, there is no reductionism that gives a linear path to that final design. There are heuristics, expert advise, patterns to follow and other guides but none of these represent the methodical approach that is implied by rationalism. The objective is to balance the many controllable factors to create a whole that again can "satisfice" for the problem to be solved. Much of the work of design methodology has been inspired by the auto industry which deals with a very analogous problem.

Cars must run, steer, stop, etc. These functional qualities are augmented by things like acceleration, comfort, and appeal. Metrics are easily acquired for functional properties but become progressively harder with many other qualities. How does one measure comfort? or appeal? But coming up with the proper collection of properties while remaining within the economic zone of cost and time has always been a challenge and more of an art than engineering science. This too is an area that does not yield to rationalism.

If we had rational specifications for the system to be built, we would also have what is needed to demonstrate at least partial compliance to those specifications. But design is often recursive and exploratory, not some linear process that begins with a complete specification. If a specification for the complete system is ever created, it grows with the design of the solution.


RATIONALISM IN HUMAN COMPUTER INTERFACE
Individuals are not rational creatures. Predicting how attractive any individual will find a particular automobile is difficult although using statistical techniques it may be possible to predict some statistics for that question. Since we left the glass CRT behind and embraced bit mapped graphics we have created a progressive series of interface devices that become ever richer in their visual communication ability along with the other senses. It is not yet possible to see how far the engineering of human computer interface design will go. But one thing has become clear, the average computer science student is poorly prepared to create a usable interface without training. And the challenges of creating a linear method of going from the system design to the human interface are far greater. It requires a knowledge of the human perceptual system, principles of good visual communication as well as an ability to understand the human context in which the system will be used.

RATIONALISM IN DESIGN REALIZATION
We want to believe that a software engineer is so familiar with the language and the idioms of coding that once the specification is given they proceed in some linear way to write the lines of code that will result in that system. We proceed to refute that with analogy to several other similar tasks, non-fiction writing and theorem proving.

Some writing instructors say that writing is thinking. It is not desirable for a program specification to be so reduced as to leave no room for variation. Such a specification is often difficult to work with since it hides some of the higher level knowledge to inform the software engineer how this module functions within the larger system, information that is needed to guide possible decisions to be made while coding. The coding proceeds until there is some question of design that arises that the software engineer does not have enough information to answer. The engineer must research and consider the choices and then proceed. This breaks the linear rationalistic conceit.

RATIONALISM IN THE VERIFICATION AND VALIDATION OF THE SYSTEM
As we observed it is rare to have a specification for the complete system. This is the ideal state of a project that applies formal methods and when done, it is possible to methodically prove compliance to some or all of the design even before it is complete. But without that specification the best that can be achieved is a methodical process of module/system/acceptance testing which attempts to present evidence that the system under design/being delivered will perform acceptably. This also recognizes the reality of clients who will often accept less than total perfection, something economists call "satisficing".

Leibniz, Locke and Wittgenstein: Rationalism versus Empiricism and the Language Game Synthesis

FSE2016 Panel: The State of Software Engineering Research
Prem 3 Lines of Research - Leibniz, Locke, Wittgenstein

https://www.youtube.com/watch?v=sE_jX92jJr8&feature=youtu.be&t=4
28:00-34:00
Rationalism epitomized by Gottfried Leibniz
We locate defects by proving a program can fail

Empiricism epitomized by John Locke
We locate defects by finding patterns of human error

Synthesis
Empirical evaluation of static versus dynamic typing
Cost-effectiveness of dynamic analysis (fault localization)
Ordering static analysis warnings by Naturalness
Exploring statistical language models for type inference

Wittgenstein might question semantics. What is a defect? A server? etc. Raises question of the language game. A language game recognizes zones of agreement.

Prior mentions include Goguen, Zave, Osterweil, Shaw, Garlan, Fielding, Taylor, Medvidovic, Qualitative work

Daniel Jackson, The "broken" concepts & rules of Git, Daniel Jackson 2016
The conceptual blending of "developer" and "operator" in DevOps
The architecture of IoT
Formulations of Security Agents, Policies, Principles

Are We Neglecting our Concepts?
Empirical and Formal are OK
"Language Game" areas: Architecture, Requirements, Process? Not so much.
Two risks: Fixations on tired old ideas, and missing out on radical new innovations

References:
Joseph Goguen
The Denial of Error, Softrware Development and Reality Construction, pp 193-202
Four Pieces on Error, Truth and Reality, 1990  accessible at https://www.cs.ox.ac.uk/files/3412/PRG89.pdf
Value-Driven Design with Algebraic Semotics, draft book with Fox Harrell, 2006, accessible at https://cseweb.ucsd.edu/~goguen/courses/271/book/uibko.pdf

Pamela Zave,
The operational versus the conventional approach to software development, CACM, Feb 1984, Vol 27 Iss 2, pp 104-118
Four dark corners of requirements engineering, Jan 1997,  ACM TOSEM vol 6 iss 1, pp 1-30

Leon Osterweil
Software Processes are Software Too, from Engineering of Software: The continuing contributions of Leon J Osterweil, ed Tarr, Wolf, pp 323-344, ICSE9, ICSE97 Proceedings of the 19th pp 540-548

The Semantics of "Bug" in Empirical Software Engineering

In my graduate studies I regularly read papers and hear people talk about software bugs. But I often find that what is meant by the word bug has a very specific meaning in their work. This short essay voices some of my concerns about the implications of this for serious work on software engineering research.

Perhaps the best comment about the meaning of the word bug comes from Dijkstra who famously railed against its use at all. The term apocryphally only came into use because of the physical nature of the post-war computers which used relays that were fouled by insects. It stuck as a way of talking about something that causes an algorithm to behave in an unintended way. But it has the unfortunate connotation of a logic error that is deus in machina, something that was not an error in thought but something that could not be foreseen. It provides far too large a loophole for algorithm designers, who design the algorithms and distances them from their lack of rigor.

In contemporary research the most common definition of bug is a reported defect or error in some bug tracking system. This can be most anything depending upon the context of the system and the human process by which those reports are created. This can include such obvious failures as an abnormal end or crash which causes the uncontrolled termination by the operating system or potentially even a crash of the operating system itself. It could be some unquantified quality that is missing such as too low a MTTF because it is difficult to find the source of the errors. It could be some expected behavior of the system which was not met but on reflection was always an expectation by the client that commissioned the system's creation. But this can also include defects that only exist because a change in the environment or requirements not anticipated at the time of the original creation. These should never be judged as all being bugs of the same category but the practicality of empirical software engineer forces them to become equal. I personally find it difficult to do any meta analysis of research because of this and doubt the results of individual papers that do not properly consider the heterogeneity of the data sources used. Can we find some agreement on the semantics of the word rather than allow it to become any observed deviation from what the end user thinks? I think this is clearly yes. But it opens a Pandora's box of semantics as it forces us to deal with design and specification, not merely the implementation.

A formal definition of bug would probably be some observable deviation of behavior from what was specified. But outside the realm of formal methods there are almost never any well formed specifications that can support this metric. For functional specifications these metrics are infrequent enough but for the previously "non-functional requirements" (software qualities) these are even rarer outside a couple domains such as military or aerospace. Even when they exist I believe the will to keep records to relate a particular code change to the one or more quality specification within its scope is lacking in most commercial products.

One technique that offers a different approach is Test Driven Development (TDD) which packages a code unit with a series of tests that become part of the automated recursion testing. This fixes the shortcoming of having an easily identified specification to match to the code unit. But I have never heard of a recursion testing system that includes the many more difficult tests such as capacity testing or performance testing. An of course testing for usability or maintainability are so difficult that I doubt anyone tries to include them in recursion testing. So while TDD offers some promise it is far from providing a framework for the kind of empirical study that the industry could benefit from.

In my conversations with Silicon Valley software engineers I have noticed a distinct trend away from anything that even bears a resemblance to the old waterfall methodologies. TDD may be the last vestige of a process that requires thought into how to articulate the behavior that is desired. This has been spurred by the strong reaction that consulting practitioners brought to the industry against waterfall methodologies. I have not yet read anything that uses the words I do to contrast the two approaches to system creation so let me explain the contrast I see.

Waterfall was a big-design-up-front. It required deep analysis and frequently caused paralysis through analysis as everyone became afraid of making an error. Agile broke that logjam by insisting on fast turnaround and the delivery of quantitized function.  It encouraged an organic approach where some prototype of the function was continually reworked and accreted function until the gap between the desired system and the delivered prototype "satisficed" and was accepted into production.

But the Agile process has an inherent defect introduced by that process, it discourages a great deal of thought into global issues of system design. For small systems or systems that do not require integration later, this is perfectly acceptable. However many large systems that will accrete their functionality over a long period of time in a partly unknown domain will not lend itself to the kind of immediate insight that allows for prescient decision making from the first prototype. There are some qualities of a system that are only evident in the whole and cannot be found in the constituent parts. These can be emergent or are qualities that can be impaired from the failure of one component. We use the term "software architecture" when we try to discuss design issues of a large ensemble of components that comprise a system. And a failure to properly appreciate the interconnectedness of these components in the beginning can lead to some very painful refactoring later in the project. It is this trend toward needed refactoring coupled with management's reluctance to acknowledge and fund the refactoring by denying the technical debt the system has accumulated. Software engineers can call this a management failure but management will call it an engineering failure as the relationship between the incremental functionality and that cost of implementing that functionality diverge because of that refactoring. At its most dysfunctional an existing system will be abandoned and rebuilt while a project not yet implemented may be canceled.

So I argue that the term "bug" can be indicative of a software engineer who is lacking in the maturity of their profession that comes after many years of watching these forces play out on real projects. I have a tendency to extrapolate from my experience to see this in the current software engineering practice. But I hear enough stories to convince myself that no one has found a magic bullet which helps a software engineer use a process that is in any way analogous to that used by engineers in other engineering fields. The reasons for this alone are interesting but beside the point; we must accept the fact that real software quality cannot be attained when the only focus on the delivered software product is limited to the most easily or badly quantified measures that sweep the subtlety of software defects away in an effort to use an existing dataset and avoid the time and expense of data gathering.

One of the last posts was about engineering design. Software Engineers are not taught how to design and it is interesting to find so many other engineering disciplines who have devoted time to the process of design. This needs to be addressed in the software engineering education programs. But before we can talk about how to teach it, we must understand it and the good books on the subject are sparse. I am going to devote some posts to trying to organize my ideas and they will be influenced by the books I cited in the prior post, information I learned from Carnegie Mellon's Software Engineering Institute plus thoughts inspired by the architect Christopher Alexander.

Christopher Alexander, for those who do not already know him, is an architect who taught at Berkeley for many years and devoted a great deal of his career to the process of understanding how buildings are created. His PhD thesis was published as a book called Notes on the Synthesis of Form. This was a profound and abstract work that is more inspiring than informing. But it does lay out an argument that design goals create a network of connections between materials and constraints and leads to some form.

Later in Software Architecture in Practice by Bass et al they laid out a similar argument that the functional requirements in a software system do not give it the structure but instead the "qualities", what were formerly called non-functional requirements, that gave it shape. For after all, a functional system can be refactored in any number of ways without changing its functional properties. If the factoring, which is the structure of the system, does not change function, what is changed? This gets explored in their work and others.

As I see it, a key and fundamental difference between computer science and software engineering is that an engineer is generally hired to provide a service, in this case the creation of a software system that will satisfy some need. It is not done for the purpose of scientific exploration but to satisfy a client. Since economic concerns are directly brought forward as constraints, and since the human element plays a pivotal element in the use of any such work for hire, and since such work is more often than not a team effort and not the work of a single software engineer, the very nature of the task is qualitatively different. The ultimate driving force is to satisfy a client's need with economy.

Any problem solving requires a person to see what is not there. To use existing tools and processes requires an understanding of what is there and the creative possibilities of those available resources. Therefore there is always a kernel of creativity in any software engineering project. This can be as minimal as installing a commercial product or as difficult as a green field development for a completely novel and previously unsolved problem. Still the engineer must be able to both communicate with their client and have sufficient mastery of the technology to deliver on the vision that was discussed.

(to be continued)

How Much Does Wisdom Weigh?

The movie 21 Grams riffed on a 1905 study that claimed that the soul weighed 21 grams. But when someone dies we know we lose information. Secrets are taken, wisdom is lost. Where does it go?

From a purely information theoretic position we know that brain can maintain complex states and that those states encode information. Those secrets, that knowledge of the world and other people disappears once the brain can no longer maintain the complex state it once did. Entropy wins and what we once had as a society is lost.

This fact has been riffed upon countless times in sci-fi with Altered Carbon's sleeves which are merely the meat that houses the information that can be stored and transferred. Putting aside the technical details and challenges of such a technology, it is interesting to me to ponder this invisible world of representation, of things unseen that are only obliquely represented in some state within the brain, the ghosts within the machine.

Within each person lies a universe accessible only to them. Anyone else can infer what is contained within. And our inherent ability to empathize is enabled by sharing a "sleeve" very similar to the other person. As the same species we can imagine what the other person feels. But before humans had the kind of language we had today that ability was severely limited.

Language is the glue that binds people together. Language is the magic that enabled a great leap forward for the species, not necessarily the individual. For with communal behavior our species increased its chances of survival and in fact eventually prospered far beyond what any prior species had. For all of the talk of individualism, human being are very social animals.

The spoken language enabled one very important skill, the ability to pass vital information from one generation to another in a way far more malleable and effective than genetics could. These nuggets of wisdom are called "memes", a label that suggests that they operate like genes.  Putting aside the details of Dawkin's theory, it is clear what is passed is pure information that is transmitted via nurturing in a given society. This was done long before we had a written or perhaps even spoken language.

This universe in our heads is filled with objects. Many of them are representations of objects in the world of atoms. In our heads they inhabit a world of bits that mirror that reality. As human beings who must remain alive, the accuracy of that internal representation is vital and we are gifted with a brain that can do that.

The cultural memes that support the complex societies we form also are found as representations in our minds. The leaders, the people to be feared or pleased, the ideas of what constitutes justice all swirl in that complex dynamical system which encodes those abstract objects in our brain and allow us to function within the society. But they have no weight. A brain that has ceased to function weighs the same as the wisest person they were before their death. The creation of brain state does not require additional mass, only a functional brain. It is pure information.

Language in all its forms is likewise pure information. We need some medium with which to exchange that information, sound waves through air, marks on a clay tablet or paper, magnetic encodings or other solid state devices. But the medium is not the information, only the messenger. And the communications theory of Claude Shannon as adopted by communications theorists is as good as any for capturing how that is done. Taken collectively these waves of symbols are states of matter that have no meaning to the world of atoms but only the world of sentient beings. They are a shadow world.

The Enlightenment saw an explosion of knowledge and an ability to gain significant advantage over our physical world, that world of atoms, for the great benefit of our species. We could create things to extend our power both over the inanimate world but also the world of other living creatures including other men. But the extension of our power and will was largely limited by the reach of our language and life. A dead king quickly loses sway over the people as the power vacuum is filled.

With each new technological innovation we seem to be able to extend a person's power and project it farther in time and space. Whether it is propaganda or wisdom, our language has created new empires and enabled new entities that transcend individuals. We have corporations and political parties that take on a life of their own independent of the people who formed them. They exert great power over the people who interact with them. They are powered and maintained by information as much as by any physical manifestation. And the modern world of information science is morphing into a post-modern world of overt embrace of control not through the manipulation of atoms but of bits.

Public relations specialists, advertising executives, salespeople and so many others in our modern world prosper not by their brawn but by their brain. Our society is now far more driven by finance and sales than it is by manufacturing. What was once the world of factory workers is becoming the domain of automation. And what controls the automation but the world of bits that permeates our society. And those who understand how to master that world of bits are the latest class of people who are seeing their stars rise in the economic battle.

I happen to believe that the modern world that started with the Enlightenment is morphing into a post modern world by the ascendency of the masters of bits. Bloggers, politicians and yes, information specialists now both directly and through their services make an outsize difference in our world. And more often than not, there is no rearrangement of atoms in the world to mirror these tidal waves of social change. Ignoring the mass of pulp that was printed because of the sexual revolution of the 60s, there was little that overtly changed in the world. But the social changes were enormous. We now live in a world of social revolutions that allow most to keep their heads.

Of particular concern to me is the role of the software engineer in this new world order. The market for those who can create machines that automate manufacturing processes is obvious. The ability of those same engineers to create bots which impersonate humans to influence buying behavior or political opinion is just being seen. The ability to project a corporate presence into the rich and interconnected world of human communication is only starting. People are afraid of robots. But it isn't the robots that should be feared by the ability of humans to create social controls and influence that they are not even aware they are creating. For complex systems do not always have predictable behaviors. Like climate change, any complex adaptive system will find an equilibrium but predicting the steady state of that system is beyond our ability. It turns out that any machine that can predict it is unlikely to do so faster than the system itself finds its equilibrium. This in itself can be an alarming realization.

We seem to be largely unaware of the vast networks of bits in which we live. Convergence is quickly occurring yet we still see distinct media and believe that computer systems are merely tools of commerce or communication. We have only the dimmest realization that this shadow world of bits that represents not only the shards of our minds as captured by this vast web but also the agents of others and potentially even agents that become detached from any human that have the ability to live within our information networks to continue doing the will we originally programmed into them regardless of the continuing desire to have that will propagated.

We not only need the tools to actually see this shadow world of information, we need a philosophy to understand how it serves or harms our species and its environment. We need thinkers who can move beyond the modern world of markets and commerce into a new world that is a battleground for hearts and minds, and yes, control over human behavior. I am calling this a post-modern attitude toward software engineering until I happen upon a better phrase.

Arotars in Cyberspace

The basic problem of software engineering is the creation of a computer system which solves some human problem. But the satisfaction of a human problem cannot be reduced in a straightforward way but instead forces a designer to solve many problems in parallel which ofter are in contradiction or interaction with each other. This is a design challenge and has not yet given us a linear way to perform software engineering that at once ensures logical perfection (is functionally correct) while also satisfying what are called emergent properties. This essay begins to layout the structure by which I am approaching this engineering challenge to make it tractable through a series of essays that I hope will coalesce into a book which puts a human face on the role of software engineering in human society.

The Role of Functional Engineering
We have not yet achieved technical perfection in the realm of functional engineering but we have made huge strides in the past few decades in engineering and the education of software engineers. A banking system ensures balanced books and accurate balances for the customers. A hydro-electric plant operating system will ensure safe and efficient operation of the many complex components of the system. But as computer software is the manipulation of abstractions of the real world which only later are transduced into reality, the primary task is one of model building.

The marvel of the modern computer is that of the blazing fast logic circuits that reduce even complex computation into tractable solutions. The process of programming is the reduction of the physical problems into abstract mathematical models, which after all are nothing but logical models of the real world. If the model represents a useful homomorphism to the real world, the manipulation of those abstractions through logical operations will give us control over the physical universe that solves the human problem. In the vocabulary of object oriented programming and design, we create objects for computation that model the real world. We are creating cyber models for the real world.

Software Qualities


The Linguistics of Computer Communication


The Differences Between Human Processes and Machine Processes



Merriam-Webster define avatar as the incarnation of a Hindu deity in human form. This basic definition is extended to mean any concept embodied in human form such as the avatar of charity and concern for the poor, for example. The cyber models we construct are similar to yet the inverse of this definition of avatar since those avatars are physical manifestations of abstract beings. The word seems to have no prior and accepted antonym.

Word origin and history: 1784, "descent of a Hindu deity," from Sanskrit avatarana "descent" (of a deity to the earth in incarnate form), from ava-"down" + base of tarati "(he) crosses over," from PIE root *tere- "to cross over" (see through). In computer use, it seems to trace to the novel "Snowcrash" (1992) by Neal Stephenson. If my understanding of Sanscrit is correct, the opposite of ava- would be aro-. So I backform a noun, arotar to be something from the physical world lifted into the otherworld, in this case cyberspace.

I suggest that the process of software engineering is the construction of these arotars and the success or failure of the complete computer system depends upon the accuracy of this transduction between the real and cyber world.

I want to also suggest that there is reason to go back to a philosophical debate raised by Descartes in his conception of dualism. For sometime we have comfortably accepted the mind as a physical device no different from any other device in our physical universe. Yet the dominance of social constructs in our lives gives us as many objects of an abstract nature as we have in the physical universe, allowing for the abstractions we subject our universe to in making it comprehensible. After all, by the very nature of our brain we cannot comprehend all the grains of sand but subject them to collective nouns which are abstractions of the multitude of individual grains. These social constructs result in some avatars which inhabit our physical universe such as currency which is an avatar for money. But in computer systems these objects are mere informational objects whose representation is almost infinitely mutable so long as we have an agreement about that translation from one form to another. Just as we can exchange currency, we can pour cyber objects from one object into another.

The way we build software models is with the use of specialized languages which we hope are unambiguous and useful for the task of building these models. They must allow us to build structures (nouns) and processes (verbs) and then weave these together into ensembles that will collectively solve a problem in cyberspace that hopefully maintains its correspondence to the human problem in the real world. This parallel operation has such a strong similarity to the substance of mind that Descartes spoke of I question whether we must completely reject dualism as a practical matter and instead admit of a cyberspace that is a stand in for what Descartes may have meant were he familiar with modernity. So I use this as a starting point to explore an alternate way of conceptualizing the software engineering process, if only as a thought experiment. Let's make a leap and think of cyberspace as some alternative universe that is in parallel with our own and intersects through a collection of avatars and arotars that come into existence, cross over that barrier and cease to exist. Does this buy us anything in our understanding of the place and use of computer technology as it is experienced by human beings?

You can take most any major corporation today that does business with customers and you see arotars. They interact with us not as individuals but as cyber entities. It is not our actual attributes they see but the attributes that have been set in those cyber objects in their databases. Anyone who has had problems with identify theft fully understands the implications this has in our lives. It is not the reality of our lives that controls their interaction with us but their belief about us that is formed by their databases. Once those databases have been corrupted, the responsibility falls to us to struggle to bring correspondence back between the arotar and our reality, often at great expense to ourselves. For the corporation, the arotar IS the human being.

This concept has been fairly well explored in science fiction as was suggested by the definition above which used avatar to represent the "embodiment" of the real in the cyber. This was also repeated in the movie of that name. But I am choosing to make a distinction between the physical brought into the cyberspace, what I am calling arotar, with the more traditional understanding of avatar which would be the physical embodiment of some abstract cyber object into our physical and social lives. But with that small adjustment in language the science fiction works have nicely explored the way the mind grapples with this interaction between the physical and cyber worlds.

Human Society, Its Construction and Maintenance
memetics

The Creation and Maintenance of the Social Meme Structure
speech act theory
The construction of a social reality that uses abstractions that are not directly found in nature long precedes cyberspace. Those who study linguistic pragmatics recognize that humans use language for that very purpose. We construct new concepts and then use language to communicate those concepts to others. Some of those concepts construct, constrain and mould our society. Human relations, hierarchies , and organizations are formed using these words and people have a predisposition to accept many social conventions without thought. Richard Dawkins offers a compelling argument that these concepts are in fact an evolutionary step for social creatures to break the shackles of our genetic adaptation and instead have bundles of information that are passed from one generation to another which gives the society some evolutionary advantage over other societies in the competition for scarce resources. Whether or not you accept Dawkins memetic argument may not be important to my thesis but it at least offers one way in which we discuss abstract concepts which are communicated with language and have profound influence over a society and its members. This is not possible without language.

The specific ways in which language gave primitive societies an advantage is a subject of speculation. We can assume it enabled more complex forms of coordination among the hunters and gatherers. But even the earliest recorded inferences we make about language suggests it was used for means other than mere survival. There was an oral tradition of stories that served some purpose in their societies. One use was in the telling of stories. Some of these stories purported to be retellings of history. But the standards of accuracy were not strict by modern standards and the telling of an entertaining yarn were as important as staying true to historical facts, even if the memory was accurate. In time this gave way to stories that were complete fabrications and fiction was born.


The Counterfactual, Fiction and the Communal Construction of the Intended World
Even linguists stumble when discussing fiction. When someone is acting a role in a play they do not become the character despite their best attempts. And no audience member believes the player becomes the character, no matter how convincing the performance. There is an odd social transaction that the player(s) and the audience enter into a mode of suspension of belief so as to inhabit this constructed universe of the playwright. The playwright has license to create an alternate universe and populate it with anything that can be conceived by the human mind. I do not think it is too much of a stretch to suggest that from its earliest manifestations that these alternative universes were an early cyberspace.

Of course from story telling and plays passed through an oral tradition humans found a way to fix their language using physical devices like clay tablets and other forms of writing. This innovation in fixing a story helped to constrain story telling making a distinction between that which was meant to be historical from that which was allowed to be fanciful. A written history would belie the fable teller's altered version but still allowed for embellishment where the written account was silent.

While the volume of alternative universes has grown at an every increasing pace, things did not change in any fundamental way until the creation of new technologies for the capturing of visual images. Photography, cinema, sound recording, television and radio have increased the amount of information that can either be recorded or fabricated to inform and entertain. These new media extended the evocative power of story telling to allow an even greater suspension of disbelief and engagement in the alternative universe.

The realization that language and all media can be reduced to flows of information with the most minimal and effervescent of physical trace allows for the convergence that has been long anticipated. Cinema can become virtualized with constructed actors and scenes. Music need never involve physical instruments that manipulate air. While composers may be able to construct music without the realization of the sound, most people can appreciate the realization but not the idea alone. One is the design, the other the construction and implementation of that design.


Software As Memes Trying to Live
Software is in the realm of ideas that at best are realized statically as text or binary files on a disk. However they are mutated into something else once transcribed or translated into the memory of a physical machine. Once there and once given agency, they command the machine to do its will.  The machine is in the realm of the atoms while the software reaches through the machine to create an interaction between the world of ideas and the world of atoms.

One aspect of software engineer that departs significantly from computer science is the realization of the idea using physical machines. Unlike idealized logic machines, a computing device may not behave as predicted. For example all modern machines have the operating system as an intermediary. Various conditions may result in unexpected and illogical results, such as a read failure from storage because of a software failure in the OS. To the application program this cannot be anticipated and makes no sense yet it is a condition that must be programmed for and handled. The contingency of many software instructions must similarly be allowed for and has created a great deal of art. Unless and until we have physical machines that achieve their logical perfection, we must program contingently and always include a fair dose of risk management in those systems.


Software Engineering Through the Lens of Dualism
I believe it is time to revisit and old philosophical position, Cartesian dualism. No modern is willing to suggest that the abstract thoughts of humans is metaphysically separate from the physical universe of atoms. But there can be little doubt that abstract thoughts pass from human to human using language and have little or no physical presence other than their representations in human minds and the various representations they take in the physical universe such as spoken word, written symbols or the various representations they take in computer systems. By themselves they do not interact with the universe of atoms unless mediated by human beings and the effect those utterances have on us.

These abstract ideas can  of course of profound consequence on the world of atoms. One only need consider the choices societies have made since the invention of agriculture. We are now seeing the effect of anthropomorphic climate change based on the behaviors of civilizations over eons of using fossil fuels. This change only happened because of human behavior. And that human behavior was shaped by the accumulated knowledge of how to exploit the natural resources to propagate the species.

One cannot place humans in some category that is completely different from any other animal. There are after all many species which can create profound changes to their environment as well. Populations may grow beyond the limits of their ecosystem and starve. Changes in weather patterns or meteor strikes may present challenges that a species may not survive. But humans are different in their adaptability and their ability to grow in number beyond most any other species. It is this unique ability of humans that must be examined to understand how software engineering plays a pivotal role in our society today. And this requires an examination of the work of Richard Dawkins who coined the word memetics.

Computers as Machines to Propagate Memes
Unlike other animals, humans are far less bound by their genetic destiny than most any other animal. The adaptation of civilizations to the various environments in which humans moved conferred a benefit that could not be possible without some genetic pre-determinism. And that benefit is clearly bound to the ability of humans to form civilizations and work cooperatively. But many other species are social. What makes humans different?

Insect are simple creatures. It is possible to completely decipher their DNA and given time we can create models of their behavior and mimic the relatively simple behavior they exhibit given the stimuli of their environment. Yet despite their simplicity they can become a formidable species. Such is the power of social behavior. Mammalian behavior, especially the ability for complex speech, dramatically improved the ability to communicate and coordinate. Even more important is the ability to transmit information from one generation to the next. Now adaptive behavior discovered by one generation could be passed forward to the next. Now one generation could benefit from the mistakes of the past. How to grow, hunt, build, organize society, etc, was not lost. Civilizations grew and became more powerful. Dawkins called this transmitted information memetics borrowing the analog of the gene to suggest that somehow there are bits of information that the human mind could receive from parents and elders yet carry forward and pass on to their progeny. While the science of memetics is perhaps still a bit controversial and ill-formed, the basic premise that information in a society influences that society and has a remarkable ability to resist change seems hard to refute.

The Dance between the Static and the Dynamic

Communications Protocols through Linguistic Eyes

Engineering and the Mind's Eye: an annotated bibliography

Engineering and the Mind's Eye by Eugene S Ferguson (1972, MIT Press) is a discussion of a view on engineering that differs from the mainline education of contemporary engineers in most disciplines. It emphasizes the history of engineering in design and the inherent way in which engineering uses visual or even artistic representation to communicate and develop solutions for patrons. While focused on the physical engineering disciplines, I believe it informs an approach to software engineering that is not taught today and which is poorly understood or appreciated.

"The Secret of Design
The design of Italian fortresses illustrates a fundamental tenet of engineering--one so right and proper, so self-evident (once it has been pointed out) and now so ingrained that it has become axiomatic: Whatever an engineer may be called upon to design, he know that in order for the plans to be effective the system being planned must be predictable and controllable. The designer first defines the boundaries of the system (often involving highly arbitrary judgements), just as the fortress designer set his boundaries at the limits of cannon range. Then the permissible inputs to the system and the permissible outputs from the system are carefully determined. Nothing may cross the boundaries unobserved or unaccounted for. There is no place an ideal engineering system for unpredictable actions, either by machines or by people. Thus the assumptions and essential procedures of the fortress designer--the fortress mentality--fit naturally the needs of the modern engineering design."  p 70

This passage is pregnant with implications for application in software engineering and is just one of many passages which discuss civil engineering or other physical engineering traditions but can be mined to gain insight into how software engineering is done by contrast with how we do not follow in this tradition.

The first obvious point is how systems engineering is about the identification of system boundaries with focus on that which crosses the boundaries. This is always taught to one degree or another in a standard CS curriculum but often given only an hour, at most, in the classroom. Yet there is a great deal of subtlety in where these lines get drawn and the very central issue of figure and ground that is assumed. In the 90s a popular mantra in systems consulting was the risk/reward ratio of scoping a re-engineering effort. A large percentage of system's projects are not greenfield development but instead a re-engineering of existing function. There is benefit to the patron in the re-engineering but that benefit is tied to the scope of that effort. A small scope can have only a small benefit while a larger scope may have a larger benefit. If the relationship of cost/benefit and the risk is linear, this is an easy decision to make. But it is rarely linear and it takes a measure of educated judgement to scope the project to maximize the benefit to the final cost.

This kind of management decision crosses the boundary between the technical engineering and the project management and the organization culture and upper management.

Another insightful comment has to do with the predictability of the system. Earlier in the book the point was made that much of modern engineering has to do with the manipulation of mathematical models of the system under design. The ability to accurately model a complex physical system with math varies depending upon the system. And the arc of engineering in any discipline has been the refinement of those models, often with the insight of prior failures. Engineers have always depended upon the knowledge gained from building the scale models or even complete project based on novel ideas and then refining based on experience over time. (Tacoma Narrows is only one example). Complex physical systems are often not completely modeled by the mathematical models. Insofar as we have a solid logical model of a computer system, this would not be a problem. But at some point every computer system will interface with a human being. And our understanding of HCI is in no way modeled using a mathematical model. We simply don't have the tools to do that nor do we even know whether that would be possible.

***

We have become so accustomed to this subordination of technology to science that it is difficult to realize that the Renaissance engineer, trained as an artist and retaining the artist's use of intuition and nonverbal thought, had significant counterparts in the United States as recently as the nineteenth century. For example, Robert Fulton of steamboat fame and Samuel Morse, inventor of the American telegraph, wee both artists before they turned to careers in technology.   (p 155)

***

The Crisis in Design
In the late 1950s, the declining ability of engineering graduates to carry ut design projects was becoming painfully evident to employers and to older teachers and administrators in engineering schools.

In 1961, Pay Chenea, head of the mechanical engineering department at Purdue, observed that while the engineering sciences had developed rapidly, design had "slipped back." Perhaps that was because the "faculty of several institutions have concluded that this may be an area of knowledge that cannot be successfully taught and hence have taken steps to eliminate an experience in design from the students' program." Perhaps a better explanation of the slipping back was that given by B. R. Teare, Jr.,m dean of engineering at Carnegie-Melon University:  "Design must compete with the engineering sciences for a place in the curriculum and for academic respectability and prestige."

A 1961 report issued by a faculty committee at MIT fully justified the fears of those who saw a crisis in design. Recent graduates were criticized for "unwillingness an inability" to consider a complete design problem and for preferring to tack only the fully specified problems that could be solved by analytical methods. This tendency was more marked at the M.S and Ph. D. levels, where young engineers "feel at home in solving problems which have numerical answers--the kind of problems used in school for teaching analytical techniques." Also, these "young engineers tend to consider problems which do not involve mathematics at least at the level of the calculus as beneath their dignity--something to be turned over to a technician who [is] without the benefit of a higher education."  (pp 161-162)

***

Semiology of Graphics by Jacques Bertin: An Annotated Bibliography

Semiology of Graphics: Diagrams, Networks, Maps
Jacques Bertin
Translated by William J Berg
Esri Press, Redlands, California, 2011
from the library of Carollina Song

Originally published in English by the Univ Wisconsin Press
1983

Originally published in French as Sémiologie grahique by Jacques Bertin copyright 1967 Editions Gauthier-Villars, Paris

LOC QA90.B47513 2010

monosemic system: the meaning of each sign is known prior to use
polysemic system: the meaning of the individual signs follows and is deduced from consideration of the collection of signs. "Signification becomes subjective and thus debatable."
"...and perception consists of decoding the image. The reading operaton takes place between the sign and its meaning. The abstract painting represents an extreme form of polysemy. In its attempt to signify "everything" it no longer signifies anything precise and so becomes "pansemic"."
p 2

"On the other hand, in graphics...each element is defined beforehand.", "The reading operation takes place among the given meanings."

"But graphics and mathematics differ in the perceptual structure which characterizes each of them. It would take at least 20,000 successive instants of perception to compare to tables of 100 by 100. If the data are transcribed graphically, comparison becomes easy; it can even be instantaneous."

"Monosemy is the fundamental condition of logic, but it also defines its limits. In effect, monosemy can only exist within a finite domain of objects and relationships. Logical reasoning, therefore, can only be a moment of reflection, since there is an infinite number of finite domains, however large they may be. Logic appears, then, as a sequence of rational moments immersed in the infinite continuum of the irrational." (footnote, p 3)

"Thought can only be expressed within a system of signs." verbal and written language are codings as are digital representations. "Any transcription leads necessarily to a separation of content from form. The 'content,' those elements of the thought which can remain constant, regardless of the sign-system into which they are translated, must be distinguished from the 'container.' that is the means available in a given system and the laws which govern their use. These elements are constant, whatever the thought to be transcribed." (p 4)

Software Art Thou: A summary of the lecture by Glenn Vanderburg from Real Software Engineering

Software Art Thou: Real Software Engineering
A summary of the lecture by Glenn Vanderburg currently VP Engineering at First.io

https://youtu.be/RhdlBHHimeM

Zendesk published a Youtube in January 2018 in which Glenn Vanderburg talks about the nature of Software Engineering and offers some insightful comments. Since many of his points are important to the research I am doing I am creating a summary of this Youtube to assist in citation. Parenthetic remarks are time m

Where SE went wrong and how to fix.
(0:27) Programmers: Stop Calling Yourselves Engineers, The Atlantic. Are SEs engineers?
Software Engineering: An Idea Whose Time Has Come and Gone?, Tom DeMarco, Computer Magazine, ACM? IEEE?
"Calling what we do 'engineering' doesn't make us engineers. It makes us liars, Dave (Pragmatic) Thomas, Extreme Programming mailing list at 2003
(2:04) Arguments against SE as engineering reduce to charge that it is fundamentally different from other engineering or that it is more artistic than engineering which essentially is the same argument.

The term Software Engineering can be traced to a NATO publication from a conference in Garmisch Germany in Oct 1968. And then Rome 1969.

He calls the 1969 the launch of Academic Software Engineering and was analytical, formal and aspirational toward engineering.

All academic software engineering (ASE) modeled process models of large scale engineering projects. Requ

Allister Colbert, was proj successful? how strictly to chosen process? PM differed from coders. Would you use this process again? (mostly no)

good people who cared were motivated to subvert process

in mid 90s, attempt to find new process, Agile
less iterative, experimental, less formal, correctness by testing, rapid adoption.
Now two views: software development should be more like engineering and software development is NOT like engineering.

Both ignore what engineering process really is.

Herbert Simon, The Sciences of the Artificial,
A Rational Model of Software , Parnass

(12:50) Glenn's view:
Software development already is engineering
it always was
the rational model (Parnass) is a caricature of engineering since no engineering is really done that ways and some look nothing like that
We don't need to become engineers but refine the software engineering process which does not have to adhere to the academic software engineering processes that were first suggested.

Remainder of talk:
the errors of ASE, what is engineering?, what is SE?, what's next?

(15:00)
The Errors of Analy\tical software engineering
Brian Randell, newcastle ed 1968
history of software engieering conference 1996
At that history of SE cibnference, he said:
"Unlike the first conference, at which it was fully accepted that the term softare engineering expressed a need rather than a reality, in Rome there was already a slighgt tendency to talk as if the subject already existed. And it became clear during the conference that the organizers had a hidden agenda, namely that of persuading NATO to fund the setting up of an Internal Software Engineering Institute. However things did not go according to their plan. The discussion sessions which were meant to provide evidence of strong and extensive supooret for this proposal were instead marked by considerable scepticism, and led one of the participants, Tom Simpson of IBM, to write a splendid short satire on 'Masterpiece Engineering'. IOt was little surprise to any of the participants in the Rome conference that no attempt was made to continue the NATO conference series, but the software engineering bandwagon began to roll as many people started to use the term to describe their work, to my mind often with very little justification. ... I made a particular point for many years of res\fusing to use the term or be associated with any even that used it."   Brian Randell

Early dissents

Alan Perliss. in 1968
A softwae system can best be designed if the testing is interlaced with the designing instead of being used after the design. (TDD)
A simulation which matches the reqrements contains the cotrol which organizes the design of the system.
Througb successive repetititons of this process of interlaced testing and design the model ultimately becaomes the software system itself. ... in effect the testing and the replacement of simulations with modules that are deeper and more detailed goes on with the simulatin model cotrolling, as it were, the place and order in which these things are done.

Fred Brooks, The Design of Design

Winston Royce agreed that waterfall seems like it should make sense but then went on to criticize it in the same paper. (Managing the Development of Large Software Systems http://www-scf.usc.edu/~csci201/lectures/Lecture11/royce1970.pdf)

"John H. Van Vleck, Nobel-laureate physicist was Dean of Harvard's Division of Engineer and Applied Science when I was a graduate student there, in Aiken's lab. Van Vleck was very concerned that the practice of engineering e put on a firmer scientific basis. He led a vigorous shift of American engineering education away from design toward applied science. The pendulum swung too far; reaction set in; and the teaching of design has been contentious every since. I am grateful that three of my Harvard teachers never lost sight of the importance of design and taught it: Philippe E Le Corbeiller,  Hary R Mimno, and Howard H Aiken, my adviser."
Preface to The Design of Design by Fred Brooks

(19:47)
Devalued the role of code.
What is needed is not classical mathematics, but mathematics. Systems should be built at levels and modules, which form a mathematical structure." Friedrich Bauer

[...] our basic tools are mathematical in nature. -Edsger Dijkstra

David Parnas 2007 oopsla, "In engineering...people design through documentation,", "Code cannot fill that role."

Academic software engineering emphasized correctness over cost.

(23:27)
graph of cost of error correction versus phase of project where error is found. But, Glenn says, emphasis on careful review of all artifacts at all stages increases costs throughout since more reviews, more quality control is needed. "The cure is worse than the disease." Low cost, high impact. Cost of long feedback loops.

(25:00) What is engineering anyway?
"Unlike the extensive analysis of the scientific method, little significant research to date has sought the philosophical foundations of engineering. Library shelves groan under the weight of books by the most scholarly, most respected people analyzing the human activity called science. No equivalent body of research treats the engineering method." -Billy Vaughn Koen

8 good books on engineering process

The sciences of the Artifiicial, Herbert A Simon
The Design of Design, Gordon L Glegg
Definition of the Engineering Method, Billy Vaughn Koen
To Engineer is Human, Henry Petroski
What Engineers Know and How They Know It, Walter G Vincenti
Engineering and the Minds's Eye, Eugene S Ferguson
The Existential Pleasures of Engineer , Samuel C Florman
The Design of Design, Fred Brooks


[Programming] is not some kind of engineering where all we have to do is put something in one end and turn the crank." -Bruce Eckel

(or paraphrased, it is not algorithmic, nor can it be)

Engineering process is inherently creative, exploratory, and will always have false starts, dead ends, and learning processes.

"The Engineering Method is the use of heuristics to cause the best change in a poorly understood or uncertain situation within the available resources." -Billy Vaughn Koen

Different engineering disciplines are different. Process Control Model, defined at one end, empirical at the other.

"The defined process control model requires that every piece of work be completely understood. A defined process can be started and allowed to run until completion, with the same results every time. The empirical process control model provides and exercises control through frequent inspection and adaptation for processes that are imperfectly defined and generate unpredictable and unrepeatable outputs." -Ken Schwaber and mike Beedle, SCRUM

Engineering which has expensive construction requires are defined.

"The conversion of an idea to an artifact, which engages both the designer and the maker, is a complex and subtle process that will always be far closer to art than to science." -Eugene S Ferguson

(31:18)
"At first sight the idea of any rules or principles being superimposed on the creative mind seems more likely to hinder than to help, but this is really quite untrue in practice. Disciplined thinking focuses inspiration rather than blinkers it." -Gordon Glegg

(38:00)
Deflection theory for suspension bridges. Tacoma Narrows bridge. Deck so thin that it had aerodynamic properties that exciting resonant frequency.

Engineers adopted formal methods to save money. Predicting failure point makes it possible to reduce safety factor.

Structural analyses (indeed any engineering calculations) must be employed with caution and judgement, because mathematical models are always less complex than actual structures, processes, or machines." -Eugene S Ferguson

Engineering driven by practitioners but refined by academics.

Robert Hook, 1671
(45:51)
Sagrada Familia

(49:00)What is Software Engineering?
Structural engineering is the science and art of designing and making, with economy and elegance, ... structures so that they can safely resist the forces to which they may be subjected.
-Structural Engineers Association


In structural engineering the engineers create diagrams and specifications which are given to builders. Historically we map this onto SE by thinking of the designers as creating specifications for the coders. But this hasn't worked well. Are coders really builders?

The source code is not the solution. It is a design document. It is what the compiler that constructs the system that actually provides the solution.

For structural engineering the most expensive phase was construction. But for SE the compiling is the cheapest. With continuous integration this phase is effectively free.

Jack Reeves, what is software design? (http://www.developerdotstar.com/mag/articles/PDF/DevDotStar_Reeves_CodeAsDesign.pdf)

Dave Thomas said XP processes tightly coupled.

Kent Beck wrote first XP book

If you look at the 13 processes, they can be arranged along a continuum of the amount of feedback they give and the scale at which it is given. From the processes that provide the most immediate feedback with the smallest scope is pair programming. At the other end we have practices that give valuable feedback but at a large scale of time and effort in short releases.

pair programming
unit testing
system metaphor
continuous integration
on-site customer
collective ownership
acceptance testing
planning game
short releases.

One suggestion: don't use languages that enable mutable state! (ha)

platform/application divide.

Glenn: stop using unsafe languages for applications.

formal methods to prove validity and security of small micro-kernels. formal methods now practical for well restricted domain.

Glenn Vanderburg, First.io, glenn@vanderburg.org or @glv

What Does Software Look Like?

I am reading the book, Engineering in the Mind's Eye by Eugene S Ferguson. The book is an in-depth discussion of the design process as found in many areas of engineering. And the focus appears to be on how those forms come to exist. He frequently cites the many paper models of the project to be created. And he stresses the visual component of how the designer tries to document what the finished product and its components look like. But how do I apply that to software engineering? If I am going to specify a system, what images help? What does software look like?

The difference between the surface appeal of a product and the inner workings is not unique to software of course. Something like a motor presents the user with a largely inscrutable object which turns an armature. But we all know that the inner workings of such an engine, or pump, or even an auto are complex. At the extreme we have the computer which has a relatively trivial surface appearance with a screen and a keyboard. But we all know the enormous complexity that exist under the cover. By analogy it is not much different than having a modest cottage that has all the inner workings of a large modern skyscraper. The appearance belies the complexity.

It is not as if the surface appearance does not matter. Quite the contrary. Working out the human-computer interface is a very important part of the design process. Not even as much because of the importance of that interface for the successful operation of the product, but because it is the way that most users can most easily explain the expected behavior, and the complex inner state transitions that are expected.

Some software product however have no human-interface design at all. Embedded systems have an interface of course. But that interface has no user perceived behavior. It becomes a piece of the infrastructure like the plumbing that we never want to deal with. And as long as it works, we don't need to.

Yet these are all important applications of software technology. And we must find a way to understand how to specify the software we need and communicate that to those who can build that for us. And much is the design literature leads us to believe that this division of designer and implementor is something artificial even while scale may require some specialization by skill.

So in this book I am hoping to find the start of some answers about what software looks like. When I engage in creating a blueprint for a software system, what should that look like? What should be visual? Are UML diagrams the right way to communicate intentions? What must be tabular? or do we need new extensions to computer languages to capture the information that right now seems to imperfectly flow from designer to builder in the processes we use today?

Software Engineering and Society

The Atlantic ran a provocative article a few years ago and I take strong exception to the points it makes. Titled Programmers: Stop Calling Yourselves Engineers (https://www.theatlantic.com/technology/archive/2015/11/programmers-should-not-call-themselves-engineers/414271/) I can agree with the point that much software engineering fails to live up to the aspirational goal of being like structural engineering. But by painting with such broad strokes the article does more to set that aspirational progress back than it does to propel it forward.

First let's consider the basic metaphysics of what a software engineer must do. Advances in steel are small and incremental. Yet changes in hardware design are still following Moore's law many decades after it was first recognized. In addition operating systems have grown in size, complexity and function just as fast as the underlying hardware. How is it possible to imagine the kind of well defined processes we find in structural engineering we see today if those engineers had such dramatic changes in their raw materials? Or if the scale of the structures they built kept pace with the growth of software systems? We would not have the dependability we see today. To suggest that software engineering is failing because it is not on par with structural engineering is a destructive false equivalence.

Next the author talks about how society has regulated some engineering disciplines. Yes, some engineering areas are subject to laws, have well defined apprenticeship tracks and continuing education requirements. But to assume that the mere adoption of those superficial forms would somehow achieve the desired effect of creating fewer software engineering failures is naive at best. Those professions themselves evolved those norms after decades, or even centuries, of practice. Software engineering today bears very little resemblance to the practice of even 30 years ago. We continue to experiment with the processes by which products are created. And the needs of the clients are so disparate that it is obvious the methods we use to created embedded systems for high safety applications bears only the vaguest similarity to the free social media apps the author cites.

Another false equivalence in the article is the criticism of the data breaches that are now getting so much criticism. The way in which we are using computer systems today is far more radical than the way we are using bridges. If society and its institutions continue to use software in novel ways we must expect that we will uncover new problems. Few thinkers in 1968 could have anticipated the kind of cyber espionage and warfare we must prepare ourselves for today. To place the blame for these failures at the feet of the software engineers who were as trusting as the people who commissioned them is hardly fair.

The author's charge of "engineerwash" is not fairly placed against engineers. Like all attempts to manipulate public trust this needs to be assessed against the organizations that profit from the work of those engineers. Engineers are rarely granted the opportunity to design the products they will build without significant input from their clients. Budget, time frames and quality are almost entirely set by the client and not the engineer. As long as we have a capitalist economy, there will be tension between engineers who may be inclined to over-engineer the product and the entrepreneurs who have incentive to under-engineer them. A great deal of legislation was introduced to affect this balance and check the worst impulses of decision makers who profit from the moral hazard.

The author does do a service to the community by pointing out the most obvious way in which engineering software systems is fundamentally different than a discipline of physical rather than information forms. Yes, changes to software alone are almost trivially made compared to structures of steel. And a great deal of rapid advance was enabled by recognizing this in our processes. If we had continued with the kind of waterfall processes that dominated most other engineering processes, many of the largest corporations on the planet today would not yet have been profitable and probably would never have been built. There is clearly a tension in practice between the need to do some level of high level design and yet embrace evolutionary design.

But process is not one-size fits all. The author glibly flits from low risk products such as social media platforms through large enterprise wide systems like airline reservation systems to individual products like spreadsheets to embedded systems with human life and safety concerns. No single process is appropriate for all of these products. The most safety critical benefit from reusing proven designs on stable platforms that have been exhaustively tested. But a great deal of wealth has been created by relaxing those constraints to explore new product categories and experimental systems. The matching of the process to the product goals is something an engineer can influence but is hardly the sole decider. And this leads into some of the most disingenuous parts of this article.

Software engineers are not unique in having clients who wish to employ them to achieve their organizational goals. A necessary feature of all engineering, as opposed to pure science, is the reality of economic constraints. These constraints are mediated by the client and are subject to limited negotiation by the engineer. We have already seen some engineering failures that were predicted to some degree of certainty by engineers such as the Saturn O-ring disaster. Engineering decisions are not made solely by engineers. To blame them for all failures is bordering on scurrilous.

Let's face facts here, Facebook is nothing like the Golden Gate Bridge. A complete failure of that platform would not end lives are throw a region into chaos. To suggest that the same standard of care should be exercised equally over both is polemic and unfair. Perhaps the notion of "move fast and break things" was in retrospect a bit naive. But the entire history of innovation is really no different. If one cannot take chances with a novel and uncritical application like a social network, then when are risks worth taking? As long as the publics health and safety is not placed at risk, the suggestion that heavy handed approach to regulating the software engineering profession betrays an officious attitude toward business.

The author also charges that software engineering has become hermetically insulated from the world. As if rocket science is not?

By page 7, the author has come back to the issue of professional licensure. I am old enough to remember these attempts. But I already pointed out how quickly the professional has changed. I looked at these early attempts with scorn at the superficiality of the certifications. Even green hiring managers could more accurately divine the capability of a potential engineer than these pseudo certifications.

I'll offer one aside on this issue. It is traditional that engineers have a four year degree. But there is scant evidence that the degree, by itself, is a reliable predictor of the engineers innate ability. While it is common that corporations will impose that restriction on hiring, it is usually out of expediency in filtering resumes more than empirical evidence that they make better engineers 4 years after hire. And given the cultural filtering of who can achieve a four years degree, it becomes more of a social filter than a technical one. This argues for the prejudice that Google and Thiel promote and which the author sneers at.

I feel I have addressed the most superficial faults of this article. But there is much more we can learn from some issues raised. One is the lure of entrepreneurship. He is absolutely correct that the ability to create a prototype system by a small group of entrepreneurial software engineers and make a great deal of money by selling this promise changes the very nature of what it means to be a software engineer. These engineers have merged the role of developer and business risk taker just as many scientists have done the same when developing their contributions to basic science. But no, this avenue is NOT turning computing into a type of speculative finance simply because the number of engineers inclined, or in a position, to take those risks is a minority. But he glosses over why this avenue is so attractive. He blithely says that engineering has always been a well-paid profession. Perhaps to a journalist.

Given the explosive growth in the technology, there has always been a severe shortage of talent compared to the appetite of the market. But like so many other labor markets in the US, this market mismatch does not explain the compensation. Consider the different economic outlooks of a hedge fund manager who can act as angel investor compared to the economic outlook of the engineer who creates the product. One can consistently outperform the other by a factor of at least 10, often far more. The basic capitalistic system encourages the use, some can say exploitation, of the engineer to remain in a risk free, but limit compensation role. How is it any wonder that young and talented developers are more inclined to explore being entrepreneurs rather than engineers?

Add to this structural bias toward risk taking the fact that powerful corporations can use selective manipulation of immigration laws to grant themselves economic relief. By keeping salaries low in the US for software engineers, they control the flow of students who choose it for a profession, When the inevitable skill shortage affects them, they claim an exception to immigration laws to allow for more software engineers to enter the US. If this were done fairly by giving them green cards I would accept it as part of the inevitable globalization of labor. But that is not what is done, these corporations go along with restrictions of these foreign workers that limit their ability to respond rationally to the US labor market and go where they can be paid the market rate for their skills. The author misses a great opportunity to make an important point about the power of the engineer in society by ignoring this.

The author cites the difference between many other professional engineers and software engineers and I made some comments about the false equivalence. However I will end by saying that I think the author does touch on something that is actually a valid criticism although I doubt they understand how valid a criticism it is. ABET is the standard by which most university programs are judged. A dirty little secret is that the computer science curriculum has fundamentally changed very little since 1968. And for prestige universities computer science is the program an aspiring software engineer must navigate to have the best chance of securing a position after graduation. Yet as the knowledge in the industry has exploded, the education system has not kept up. Once someone who wanted to study computer science got a math or physics degree. We are long past the time when meaningful distinctions can be made between computer science and software engineering. Why haven't we? Why has the gulf between academia and practice grown? Why are the first two years of a software engineer spent learning things not taught in school? And by this criticism I am not talking about skills training which by necessity is the responsibility of the employer. But instead a more mature approach to the entire software lifecycle than a very superficial mastery of basic coding, if even that.

The author accurately says that software engineering titles are aspirational more than an accurate description of what goes on in the industry or taught by the schools. But on the whole the article comes across as an uninformed hit piece that is designed to get attention by offering half truths about the role of the software engineer today. It missed a great opportunity to make some very valid criticisms of the role of software and the role of software creators in today's world.

Speech Act Theory and It's Role in Software Engineering

John Searle was a disciple of J L Austin who has first defined what we now call Speech Act Theory. Searle's lectures for both the Philosophy of Language and Philosophy of Mind at UC Berkeley were both recorded and are available on Youtube. I want to make a few points about how I believe this linguistic theory informs software engineering and complex software systems.

A key point in speech act theory (SAT) is that some human realities are created with some utterances. When a person has the formal authority to marry or judge, a simple utterance can make a person guilty or married. I extend this concept (or perhaps am anticipating what others have already said) to surmise that language is not only a means of communication among people but is actually a tool by which we construct and maintain social reality. And I see no reason why this would not apply equally to how we use language and software artifacts as language to construct other realities. Here I am trying to work out what that can mean.

What is a computer program? What is software? I take a piece of code to be a blueprint for a machine. When combined with hardware and given agency it transforms that machine in which it is loaded into a realization of that intended machine. The message, utterance if you will, to run the program creates a kind of intentionality that has a world to mind direction of fit, we desire to have that machine be real and make it so with our utterance. This can be a command issued at a command line or the click of a button.

Programming can become arbitrarily complex and progressively more agency can be invested in the machines. Instead of pressing a button, we can invest a machine with the authority to press the button in response to some external stimulus (a mind to world direction of fit in reference to the agent) or on some schedule, which is after all a stimulus of passing time. As we progress with giving more agency to these machines I believe it makes sense to use a theory of mind to talk about the agents and not just humans.

There are two distinct threads in machines today, computation communications. Computation often steals the focus today with the dramatic advances in artificial intelligence and the prevalence of algorithms in social media. But the advances in communications are no less impressive as we progress toward a single pervasive and omnipresent communications network that is used for internet of things as well as for human communications. The growth of computer to computer communications is no less impressive than the ability of any person to be able to converse with any other person on the globe at will. But this intra-computer agent communication must become visible to us at our will as well, especially when it does not serve our needs and we need to understand or change it.

Does it not make sense to look at all network communications as utterances on par with human communications? Do we not preside over a cybernetic realm that is now an extension of our human and physical realm? Do we not have utterances by agents that change our social world as thoroughly as a chaplain or judge? When a municipal system issues a warrant for our arrest, even when done in error, it is a real warrant and will have real world consequences. I want to take this as a starting point and begin to review computer systems through this lens.

***

The origin story for this line of thinking came from a more modest question of how programmers document their code. It is rather obvious that a programmer is creating a unique language that becomes a kind of private language between them and compiler. If the intentionalities of the coder are to be preserved, care must be taken in the naming of the nouns and verbs. For they invariably have some connection, usually indirect, with things in the real world. And in the complexity of the design, this intended relationship is often lost. Even worse is the complexity of the mechanisms by which the surrogates of the real world are manipulated by the algorithms. Anyone trying to pick up the pieces behind a coder must enter into a unique world as envisioned by that coder with the private language they had constructed to communicate their intensions to the compiler.

The connection to speech act came about because of some earlier work I once did on trying to relate communications between a client and a lawyer and codify the conditions of satisfaction that would underlie commitments that were made. But once one understands this linguistic theory, it begins to open new ways of looking at computer languages beyond the standard syntax and semantics that are used.

The Role of Design in Engineering and Software Engineering Education.

A continuing issue I have in the teaching of computer science at university is that I believe it is either delusion or fraud. What the computer science program is used for by students is to prepare them for a job in software engineering yet few schools seem inclined to embrace that label and instead cling to the more traditional notion of computer science. I don't see how we can progress as educators if we do not look honestly at what is going on and how that is not serving the needs of the students. 

Computer science, when it was first acknowledged as a separate academic department apart from applied mathematics, was either organized with mathematics, as a branch of applied math, or in the engineering college, to align with the hardware engineering. According to a an ACM article, the first PhD in computer science was awarded to Richard Wexelblat in 1965 and the second was awarded to Andy van Dam both at the University of Pennsylvania in 1965/1966. (https://cacm.acm.org/blogs/blog-cacm/159591-who-earned-first-computer-science-ph-d/fulltext). The general trend has been to organize the department into the college of engineering but I happen to believe that is wrong and implicitly acknowledges that what is really taught as computer science is truly software engineering under the false label of computer science.

This was easily justified in decades past since there was really not sufficient material to make a distinction. But the amount of material that can be taught, and should be taught, to a budding software engineer has grown and university programs find themselves constantly squeezed to satisfy current market needs by teaching many current techniques while also unwilling to stick to the traditional programs that stressed the more abstract computer science in favor of heavy emphasis on programming. 

There is one particular failing that results from this trend to cling to the science as opposed to the engineering and that is the place of design in the curriculum. And this failure is not limited to only the computer science department but can be seen throughout the college of engineering. For engineering differs from science in one crucial respect, it is inherently tied to economy and to the satisfaction of a client. So why do I believe this is so in today's universities? I believe it is a combination of a few aspects of the culture and history of engineering in the US.
Engineers are invariably quantitatively oriented. And for software engineering in particular, the formal mathematical foundation of the discipline does not admit for much qualitative or humanistic ambiguity. The first uses of computers were for intense calculation and that did not require any design, at least as that concept was understood outside of an engineering discipline where design only meant the choices one made between some relatively well known approaches. Design, and humanities in general, were kept external to the discipline and while that works for most engineering disciplines, it is no longer tenable in software engineering. 

All engineering requires a degree of creativity although the practice is more dominated by the reuse of a few well known designs that are scaled or altered by the application. But even the most routine of engineering disciplines allows for novel design and innovation. Software engineering is exceptional in the growth and innovation that it has experienced. And its high-touch applications in human-computer interaction with its use of visual communications, it has benefited from graphic artists and others outside of the engineering college. 

The other way in which humanities was admitted to the study of software engineering was from the liberal admission of metaphors to architecture. Today major software systems are complex and enormous ensembles that dwarf any physical structures we build today. Yet the way we understood the construction of a software system was heavily influenced by the construction of buildings and other large scale physical artifacts like airplanes or rockets. This had enormous impact on both practice and education as I'll explain below. But for now the point i wish to make is that all of engineering wanted to distance themselves from a discipline that was aligned with the art department, art and architecture. 

A lecture that gives voice to many of the beliefs I hold about software engineering can be found on Youtube and was given by Glenn Vanderburg under the title, Software Art Thou. (https://youtu.be/RhdlBHHimeM)  
This was easily justified in decades past since there was really not sufficient material to make a distinction. But the amount of material that can be taught, and should be taught, to a budding software engineer has grown and university programs find themselves constantly squeezed to satisfy current market needs by teaching many current techniques while also unwilling to stick to the traditional programs that stressed the more abstract computer science in favor of heavy emphasis on programming. 

There is one particular failing that results from this trend to cling to the science as opposed to the engineering and that is the place of design in the curriculum. And this failure is not limited to only the computer science department but can be seen throughout the college of engineering. For engineering differs from science in one crucial respect, it is inherently tied to economy and to the satisfaction of a client. So why do I believe this is so in today's universities? I believe it is a combination of a few aspects of the culture and history of engineering in the US.

Engineers are invariably quantitatively oriented. And for software engineering in particular, the formal mathematical foundation of the discipline does not admit for much qualitative or humanistic ambiguity. The first uses of computers were for intense calculation and that did not require any design, at least as that concept was understood outside of an engineering discipline where design only meant the choices one made between some relatively well known approaches. Design, and humanities in general, were kept external to the discipline and while that works for most engineering disciplines, it is no longer tenable in software engineering. 

All engineering requires a degree of creativity although the practice is more dominated by the reuse of a few well known designs that are scaled or altered by the application. But even the most routine of engineering disciplines allows for novel design and innovation. Software engineering is exceptional in the growth and innovation that it has experienced. And its high-touch applications in human-computer interaction with its use of visual communications, it has benefited from graphic artists and others outside of the engineering college. 

The other way in which humanities was admitted to the study of software engineering was from the liberal admission of metaphors to architecture. Today major software systems are complex and enormous ensembles that dwarf any physical structures we build today. Yet the way we understood the construction of a software system was heavily influenced by the construction of buildings and other large scale physical artifacts like airplanes or rockets. This had enormous impact on both practice and education as I'll explain below. But for now the point i wish to make is that all of engineering wanted to distance themselves from a discipline that was aligned with the art department, art and architecture. 

A lecture that gives voice to many of the beliefs I hold about software engineering can be found on Youtube and was given by Glenn Vanderburg under the title, Software Art Thou. (https://youtu.be/RhdlBHHimeM)  A key item he reports is ideas held by the dean of the Harvard engineering school from 1951 to 1957, Doctor Van Vleck. He desired to place engineering on a more scientific basis. It should be noted that Doctor Van Vleck was a Nobel Prize winning physicist. In the talk Brooks is quoted as having said:

The Vanderburg video rails against the embrace of the waterfall model for earlier academics. There was always a belief that this was the dominant process model for all engineering. 


"Van Vleck was very concerned the the practice of engineering be put on a firm scientific basis. He led a vigorous shift of American engineering education away from design toward applied science. The pendulum swung too far; reactions set in; and the teaching of design has been contentious ever since."
I agree with Brook's comment and I see as a consequence that the heightened need to teach design principles now affects all engineering disciplines but is most visible in the failure of computer science professors to teach even rudimentary design concepts. And it has also shaped the expectations of these students to what the academy believes is the correct process that should be followed in industry.
The needs of students, the needs of future employers and even the material itself all argues for a fundamental rethinking of how students who wish to become software engineers are taught at university today. We must look at the distinction between engineering versus science and also redress the avoidance of good design in their education. It is our job to do the best we can and I believe we are failing to do so.