Thursday, February 14, 2019

How Much Does Wisdom Weigh?

The movie 21 Grams riffed on a 1905 study that claimed that the soul weighed 21 grams. But when someone dies we know we lose information. Secrets are taken, wisdom is lost. Where does it go?

From a purely information theoretic position we know that brain can maintain complex states and that those states encode information. Those secrets, that knowledge of the world and other people disappears once the brain can no longer maintain the complex state it once did. Entropy wins and what we once had as a society is lost.

This fact has been riffed upon countless times in sci-fi with Altered Carbon's sleeves which are merely the meat that houses the information that can be stored and transferred. Putting aside the technical details and challenges of such a technology, it is interesting to me to ponder this invisible world of representation, of things unseen that are only obliquely represented in some state within the brain, the ghosts within the machine.

Within each person lies a universe accessible only to them. Anyone else can infer what is contained within. And our inherent ability to empathize is enabled by sharing a "sleeve" very similar to the other person. As the same species we can imagine what the other person feels. But before humans had the kind of language we had today that ability was severely limited.

Language is the glue that binds people together. Language is the magic that enabled a great leap forward for the species, not necessarily the individual. For with communal behavior our species increased its chances of survival and in fact eventually prospered far beyond what any prior species had. For all of the talk of individualism, human being are very social animals.

The spoken language enabled one very important skill, the ability to pass vital information from one generation to another in a way far more malleable and effective than genetics could. These nuggets of wisdom are called "memes", a label that suggests that they operate like genes.  Putting aside the details of Dawkin's theory, it is clear what is passed is pure information that is transmitted via nurturing in a given society. This was done long before we had a written or perhaps even spoken language.

This universe in our heads is filled with objects. Many of them are representations of objects in the world of atoms. In our heads they inhabit a world of bits that mirror that reality. As human beings who must remain alive, the accuracy of that internal representation is vital and we are gifted with a brain that can do that.

The cultural memes that support the complex societies we form also are found as representations in our minds. The leaders, the people to be feared or pleased, the ideas of what constitutes justice all swirl in that complex dynamical system which encodes those abstract objects in our brain and allow us to function within the society. But they have no weight. A brain that has ceased to function weighs the same as the wisest person they were before their death. The creation of brain state does not require additional mass, only a functional brain. It is pure information.

Language in all its forms is likewise pure information. We need some medium with which to exchange that information, sound waves through air, marks on a clay tablet or paper, magnetic encodings or other solid state devices. But the medium is not the information, only the messenger. And the communications theory of Claude Shannon as adopted by communications theorists is as good as any for capturing how that is done. Taken collectively these waves of symbols are states of matter that have no meaning to the world of atoms but only the world of sentient beings. They are a shadow world.

The Enlightenment saw an explosion of knowledge and an ability to gain significant advantage over our physical world, that world of atoms, for the great benefit of our species. We could create things to extend our power both over the inanimate world but also the world of other living creatures including other men. But the extension of our power and will was largely limited by the reach of our language and life. A dead king quickly loses sway over the people as the power vacuum is filled.

With each new technological innovation we seem to be able to extend a person's power and project it farther in time and space. Whether it is propaganda or wisdom, our language has created new empires and enabled new entities that transcend individuals. We have corporations and political parties that take on a life of their own independent of the people who formed them. They exert great power over the people who interact with them. They are powered and maintained by information as much as by any physical manifestation. And the modern world of information science is morphing into a post-modern world of overt embrace of control not through the manipulation of atoms but of bits.

Public relations specialists, advertising executives, salespeople and so many others in our modern world prosper not by their brawn but by their brain. Our society is now far more driven by finance and sales than it is by manufacturing. What was once the world of factory workers is becoming the domain of automation. And what controls the automation but the world of bits that permeates our society. And those who understand how to master that world of bits are the latest class of people who are seeing their stars rise in the economic battle.

I happen to believe that the modern world that started with the Enlightenment is morphing into a post modern world by the ascendency of the masters of bits. Bloggers, politicians and yes, information specialists now both directly and through their services make an outsize difference in our world. And more often than not, there is no rearrangement of atoms in the world to mirror these tidal waves of social change. Ignoring the mass of pulp that was printed because of the sexual revolution of the 60s, there was little that overtly changed in the world. But the social changes were enormous. We now live in a world of social revolutions that allow most to keep their heads.

Of particular concern to me is the role of the software engineer in this new world order. The market for those who can create machines that automate manufacturing processes is obvious. The ability of those same engineers to create bots which impersonate humans to influence buying behavior or political opinion is just being seen. The ability to project a corporate presence into the rich and interconnected world of human communication is only starting. People are afraid of robots. But it isn't the robots that should be feared by the ability of humans to create social controls and influence that they are not even aware they are creating. For complex systems do not always have predictable behaviors. Like climate change, any complex adaptive system will find an equilibrium but predicting the steady state of that system is beyond our ability. It turns out that any machine that can predict it is unlikely to do so faster than the system itself finds its equilibrium. This in itself can be an alarming realization.

We seem to be largely unaware of the vast networks of bits in which we live. Convergence is quickly occurring yet we still see distinct media and believe that computer systems are merely tools of commerce or communication. We have only the dimmest realization that this shadow world of bits that represents not only the shards of our minds as captured by this vast web but also the agents of others and potentially even agents that become detached from any human that have the ability to live within our information networks to continue doing the will we originally programmed into them regardless of the continuing desire to have that will propagated.

We not only need the tools to actually see this shadow world of information, we need a philosophy to understand how it serves or harms our species and its environment. We need thinkers who can move beyond the modern world of markets and commerce into a new world that is a battleground for hearts and minds, and yes, control over human behavior. I am calling this a post-modern attitude toward software engineering until I happen upon a better phrase.

Friday, November 2, 2018

Arotars in Cyberspace

The basic problem of software engineering is the creation of a computer system which solves some human problem. But the satisfaction of a human problem cannot be reduced in a straightforward way but instead forces a designer to solve many problems in parallel which ofter are in contradiction or interaction with each other. This is a design challenge and has not yet given us a linear way to perform software engineering that at once ensures logical perfection (is functionally correct) while also satisfying what are called emergent properties. This essay begins to layout the structure by which I am approaching this engineering challenge to make it tractable through a series of essays that I hope will coalesce into a book which puts a human face on the role of software engineering in human society.

The Role of Functional Engineering
We have not yet achieved technical perfection in the realm of functional engineering but we have made huge strides in the past few decades in engineering and the education of software engineers. A banking system ensures balanced books and accurate balances for the customers. A hydro-electric plant operating system will ensure safe and efficient operation of the many complex components of the system. But as computer software is the manipulation of abstractions of the real world which only later are transduced into reality, the primary task is one of model building.

The marvel of the modern computer is that of the blazing fast logic circuits that reduce even complex computation into tractable solutions. The process of programming is the reduction of the physical problems into abstract mathematical models, which after all are nothing but logical models of the real world. If the model represents a useful homomorphism to the real world, the manipulation of those abstractions through logical operations will give us control over the physical universe that solves the human problem. In the vocabulary of object oriented programming and design, we create objects for computation that model the real world. We are creating cyber models for the real world.

Software Qualities

The Linguistics of Computer Communication

The Differences Between Human Processes and Machine Processes

Merriam-Webster define avatar as the incarnation of a Hindu deity in human form. This basic definition is extended to mean any concept embodied in human form such as the avatar of charity and concern for the poor, for example. The cyber models we construct are similar to yet the inverse of this definition of avatar since those avatars are physical manifestations of abstract beings. The word seems to have no prior and accepted antonym.

Word origin and history: 1784, "descent of a Hindu deity," from Sanskrit avatarana "descent" (of a deity to the earth in incarnate form), from ava-"down" + base of tarati "(he) crosses over," from PIE root *tere- "to cross over" (see through). In computer use, it seems to trace to the novel "Snowcrash" (1992) by Neal Stephenson. If my understanding of Sanscrit is correct, the opposite of ava- would be aro-. So I backform a noun, arotar to be something from the physical world lifted into the otherworld, in this case cyberspace.

I suggest that the process of software engineering is the construction of these arotars and the success or failure of the complete computer system depends upon the accuracy of this transduction between the real and cyber world.

I want to also suggest that there is reason to go back to a philosophical debate raised by Descartes in his conception of dualism. For sometime we have comfortably accepted the mind as a physical device no different from any other device in our physical universe. Yet the dominance of social constructs in our lives gives us as many objects of an abstract nature as we have in the physical universe, allowing for the abstractions we subject our universe to in making it comprehensible. After all, by the very nature of our brain we cannot comprehend all the grains of sand but subject them to collective nouns which are abstractions of the multitude of individual grains. These social constructs result in some avatars which inhabit our physical universe such as currency which is an avatar for money. But in computer systems these objects are mere informational objects whose representation is almost infinitely mutable so long as we have an agreement about that translation from one form to another. Just as we can exchange currency, we can pour cyber objects from one object into another.

The way we build software models is with the use of specialized languages which we hope are unambiguous and useful for the task of building these models. They must allow us to build structures (nouns) and processes (verbs) and then weave these together into ensembles that will collectively solve a problem in cyberspace that hopefully maintains its correspondence to the human problem in the real world. This parallel operation has such a strong similarity to the substance of mind that Descartes spoke of I question whether we must completely reject dualism as a practical matter and instead admit of a cyberspace that is a stand in for what Descartes may have meant were he familiar with modernity. So I use this as a starting point to explore an alternate way of conceptualizing the software engineering process, if only as a thought experiment. Let's make a leap and think of cyberspace as some alternative universe that is in parallel with our own and intersects through a collection of avatars and arotars that come into existence, cross over that barrier and cease to exist. Does this buy us anything in our understanding of the place and use of computer technology as it is experienced by human beings?

You can take most any major corporation today that does business with customers and you see arotars. They interact with us not as individuals but as cyber entities. It is not our actual attributes they see but the attributes that have been set in those cyber objects in their databases. Anyone who has had problems with identify theft fully understands the implications this has in our lives. It is not the reality of our lives that controls their interaction with us but their belief about us that is formed by their databases. Once those databases have been corrupted, the responsibility falls to us to struggle to bring correspondence back between the arotar and our reality, often at great expense to ourselves. For the corporation, the arotar IS the human being.

This concept has been fairly well explored in science fiction as was suggested by the definition above which used avatar to represent the "embodiment" of the real in the cyber. This was also repeated in the movie of that name. But I am choosing to make a distinction between the physical brought into the cyberspace, what I am calling arotar, with the more traditional understanding of avatar which would be the physical embodiment of some abstract cyber object into our physical and social lives. But with that small adjustment in language the science fiction works have nicely explored the way the mind grapples with this interaction between the physical and cyber worlds.

Human Society, Its Construction and Maintenance

The Creation and Maintenance of the Social Meme Structure
speech act theory
The construction of a social reality that uses abstractions that are not directly found in nature long precedes cyberspace. Those who study linguistic pragmatics recognize that humans use language for that very purpose. We construct new concepts and then use language to communicate those concepts to others. Some of those concepts construct, constrain and mould our society. Human relations, hierarchies , and organizations are formed using these words and people have a predisposition to accept many social conventions without thought. Richard Dawkins offers a compelling argument that these concepts are in fact an evolutionary step for social creatures to break the shackles of our genetic adaptation and instead have bundles of information that are passed from one generation to another which gives the society some evolutionary advantage over other societies in the competition for scarce resources. Whether or not you accept Dawkins memetic argument may not be important to my thesis but it at least offers one way in which we discuss abstract concepts which are communicated with language and have profound influence over a society and its members. This is not possible without language.

The specific ways in which language gave primitive societies an advantage is a subject of speculation. We can assume it enabled more complex forms of coordination among the hunters and gatherers. But even the earliest recorded inferences we make about language suggests it was used for means other than mere survival. There was an oral tradition of stories that served some purpose in their societies. One use was in the telling of stories. Some of these stories purported to be retellings of history. But the standards of accuracy were not strict by modern standards and the telling of an entertaining yarn were as important as staying true to historical facts, even if the memory was accurate. In time this gave way to stories that were complete fabrications and fiction was born.

The Counterfactual, Fiction and the Communal Construction of the Intended World
Even linguists stumble when discussing fiction. When someone is acting a role in a play they do not become the character despite their best attempts. And no audience member believes the player becomes the character, no matter how convincing the performance. There is an odd social transaction that the player(s) and the audience enter into a mode of suspension of belief so as to inhabit this constructed universe of the playwright. The playwright has license to create an alternate universe and populate it with anything that can be conceived by the human mind. I do not think it is too much of a stretch to suggest that from its earliest manifestations that these alternative universes were an early cyberspace.

Of course from story telling and plays passed through an oral tradition humans found a way to fix their language using physical devices like clay tablets and other forms of writing. This innovation in fixing a story helped to constrain story telling making a distinction between that which was meant to be historical from that which was allowed to be fanciful. A written history would belie the fable teller's altered version but still allowed for embellishment where the written account was silent.

While the volume of alternative universes has grown at an every increasing pace, things did not change in any fundamental way until the creation of new technologies for the capturing of visual images. Photography, cinema, sound recording, television and radio have increased the amount of information that can either be recorded or fabricated to inform and entertain. These new media extended the evocative power of story telling to allow an even greater suspension of disbelief and engagement in the alternative universe.

The realization that language and all media can be reduced to flows of information with the most minimal and effervescent of physical trace allows for the convergence that has been long anticipated. Cinema can become virtualized with constructed actors and scenes. Music need never involve physical instruments that manipulate air. While composers may be able to construct music without the realization of the sound, most people can appreciate the realization but not the idea alone. One is the design, the other the construction and implementation of that design.

Software As Memes Trying to Live
Software is in the realm of ideas that at best are realized statically as text or binary files on a disk. However they are mutated into something else once transcribed or translated into the memory of a physical machine. Once there and once given agency, they command the machine to do its will.  The machine is in the realm of the atoms while the software reaches through the machine to create an interaction between the world of ideas and the world of atoms.

One aspect of software engineer that departs significantly from computer science is the realization of the idea using physical machines. Unlike idealized logic machines, a computing device may not behave as predicted. For example all modern machines have the operating system as an intermediary. Various conditions may result in unexpected and illogical results, such as a read failure from storage because of a software failure in the OS. To the application program this cannot be anticipated and makes no sense yet it is a condition that must be programmed for and handled. The contingency of many software instructions must similarly be allowed for and has created a great deal of art. Unless and until we have physical machines that achieve their logical perfection, we must program contingently and always include a fair dose of risk management in those systems.

Software Engineering Through the Lens of Dualism
I believe it is time to revisit and old philosophical position, Cartesian dualism. No modern is willing to suggest that the abstract thoughts of humans is metaphysically separate from the physical universe of atoms. But there can be little doubt that abstract thoughts pass from human to human using language and have little or no physical presence other than their representations in human minds and the various representations they take in the physical universe such as spoken word, written symbols or the various representations they take in computer systems. By themselves they do not interact with the universe of atoms unless mediated by human beings and the effect those utterances have on us.

These abstract ideas can  of course of profound consequence on the world of atoms. One only need consider the choices societies have made since the invention of agriculture. We are now seeing the effect of anthropomorphic climate change based on the behaviors of civilizations over eons of using fossil fuels. This change only happened because of human behavior. And that human behavior was shaped by the accumulated knowledge of how to exploit the natural resources to propagate the species.

One cannot place humans in some category that is completely different from any other animal. There are after all many species which can create profound changes to their environment as well. Populations may grow beyond the limits of their ecosystem and starve. Changes in weather patterns or meteor strikes may present challenges that a species may not survive. But humans are different in their adaptability and their ability to grow in number beyond most any other species. It is this unique ability of humans that must be examined to understand how software engineering plays a pivotal role in our society today. And this requires an examination of the work of Richard Dawkins who coined the word memetics.

Computers as Machines to Propagate Memes
Unlike other animals, humans are far less bound by their genetic destiny than most any other animal. The adaptation of civilizations to the various environments in which humans moved conferred a benefit that could not be possible without some genetic pre-determinism. And that benefit is clearly bound to the ability of humans to form civilizations and work cooperatively. But many other species are social. What makes humans different?

Insect are simple creatures. It is possible to completely decipher their DNA and given time we can create models of their behavior and mimic the relatively simple behavior they exhibit given the stimuli of their environment. Yet despite their simplicity they can become a formidable species. Such is the power of social behavior. Mammalian behavior, especially the ability for complex speech, dramatically improved the ability to communicate and coordinate. Even more important is the ability to transmit information from one generation to the next. Now adaptive behavior discovered by one generation could be passed forward to the next. Now one generation could benefit from the mistakes of the past. How to grow, hunt, build, organize society, etc, was not lost. Civilizations grew and became more powerful. Dawkins called this transmitted information memetics borrowing the analog of the gene to suggest that somehow there are bits of information that the human mind could receive from parents and elders yet carry forward and pass on to their progeny. While the science of memetics is perhaps still a bit controversial and ill-formed, the basic premise that information in a society influences that society and has a remarkable ability to resist change seems hard to refute.

The Dance between the Static and the Dynamic

Communications Protocols through Linguistic Eyes

Sunday, October 28, 2018

Engineering and the Mind's Eye: an annotated bibliography

Engineering and the Mind's Eye by Eugene S Ferguson (1972, MIT Press) is a discussion of a view on engineering that differs from the mainline education of contemporary engineers in most disciplines. It emphasizes the history of engineering in design and the inherent way in which engineering uses visual or even artistic representation to communicate and develop solutions for patrons. While focused on the physical engineering disciplines, I believe it informs an approach to software engineering that is not taught today and which is poorly understood or appreciated.

"The Secret of Design
The design of Italian fortresses illustrates a fundamental tenet of engineering--one so right and proper, so self-evident (once it has been pointed out) and now so ingrained that it has become axiomatic: Whatever an engineer may be called upon to design, he know that in order for the plans to be effective the system being planned must be predictable and controllable. The designer first defines the boundaries of the system (often involving highly arbitrary judgements), just as the fortress designer set his boundaries at the limits of cannon range. Then the permissible inputs to the system and the permissible outputs from the system are carefully determined. Nothing may cross the boundaries unobserved or unaccounted for. There is no place an ideal engineering system for unpredictable actions, either by machines or by people. Thus the assumptions and essential procedures of the fortress designer--the fortress mentality--fit naturally the needs of the modern engineering design."  p 70

This passage is pregnant with implications for application in software engineering and is just one of many passages which discuss civil engineering or other physical engineering traditions but can be mined to gain insight into how software engineering is done by contrast with how we do not follow in this tradition.

The first obvious point is how systems engineering is about the identification of system boundaries with focus on that which crosses the boundaries. This is always taught to one degree or another in a standard CS curriculum but often given only an hour, at most, in the classroom. Yet there is a great deal of subtlety in where these lines get drawn and the very central issue of figure and ground that is assumed. In the 90s a popular mantra in systems consulting was the risk/reward ratio of scoping a re-engineering effort. A large percentage of system's projects are not greenfield development but instead a re-engineering of existing function. There is benefit to the patron in the re-engineering but that benefit is tied to the scope of that effort. A small scope can have only a small benefit while a larger scope may have a larger benefit. If the relationship of cost/benefit and the risk is linear, this is an easy decision to make. But it is rarely linear and it takes a measure of educated judgement to scope the project to maximize the benefit to the final cost.

This kind of management decision crosses the boundary between the technical engineering and the project management and the organization culture and upper management.

Another insightful comment has to do with the predictability of the system. Earlier in the book the point was made that much of modern engineering has to do with the manipulation of mathematical models of the system under design. The ability to accurately model a complex physical system with math varies depending upon the system. And the arc of engineering in any discipline has been the refinement of those models, often with the insight of prior failures. Engineers have always depended upon the knowledge gained from building the scale models or even complete project based on novel ideas and then refining based on experience over time. (Tacoma Narrows is only one example). Complex physical systems are often not completely modeled by the mathematical models. Insofar as we have a solid logical model of a computer system, this would not be a problem. But at some point every computer system will interface with a human being. And our understanding of HCI is in no way modeled using a mathematical model. We simply don't have the tools to do that nor do we even know whether that would be possible.


We have become so accustomed to this subordination of technology to science that it is difficult to realize that the Renaissance engineer, trained as an artist and retaining the artist's use of intuition and nonverbal thought, had significant counterparts in the United States as recently as the nineteenth century. For example, Robert Fulton of steamboat fame and Samuel Morse, inventor of the American telegraph, wee both artists before they turned to careers in technology.   (p 155)


The Crisis in Design
In the late 1950s, the declining ability of engineering graduates to carry ut design projects was becoming painfully evident to employers and to older teachers and administrators in engineering schools.

In 1961, Pay Chenea, head of the mechanical engineering department at Purdue, observed that while the engineering sciences had developed rapidly, design had "slipped back." Perhaps that was because the "faculty of several institutions have concluded that this may be an area of knowledge that cannot be successfully taught and hence have taken steps to eliminate an experience in design from the students' program." Perhaps a better explanation of the slipping back was that given by B. R. Teare, Jr.,m dean of engineering at Carnegie-Melon University:  "Design must compete with the engineering sciences for a place in the curriculum and for academic respectability and prestige."

A 1961 report issued by a faculty committee at MIT fully justified the fears of those who saw a crisis in design. Recent graduates were criticized for "unwillingness an inability" to consider a complete design problem and for preferring to tack only the fully specified problems that could be solved by analytical methods. This tendency was more marked at the M.S and Ph. D. levels, where young engineers "feel at home in solving problems which have numerical answers--the kind of problems used in school for teaching analytical techniques." Also, these "young engineers tend to consider problems which do not involve mathematics at least at the level of the calculus as beneath their dignity--something to be turned over to a technician who [is] without the benefit of a higher education."  (pp 161-162)


Wednesday, October 24, 2018

Semiology of Graphics by Jacques Bertin: An Annotated Bibliography

Semiology of Graphics: Diagrams, Networks, Maps
Jacques Bertin
Translated by William J Berg
Esri Press, Redlands, California, 2011
from the library of Carollina Song

Originally published in English by the Univ Wisconsin Press

Originally published in French as Sémiologie grahique by Jacques Bertin copyright 1967 Editions Gauthier-Villars, Paris

LOC QA90.B47513 2010

monosemic system: the meaning of each sign is known prior to use
polysemic system: the meaning of the individual signs follows and is deduced from consideration of the collection of signs. "Signification becomes subjective and thus debatable."
"...and perception consists of decoding the image. The reading operaton takes place between the sign and its meaning. The abstract painting represents an extreme form of polysemy. In its attempt to signify "everything" it no longer signifies anything precise and so becomes "pansemic"."
p 2

"On the other hand, in graphics...each element is defined beforehand.", "The reading operation takes place among the given meanings."

"But graphics and mathematics differ in the perceptual structure which characterizes each of them. It would take at least 20,000 successive instants of perception to compare to tables of 100 by 100. If the data are transcribed graphically, comparison becomes easy; it can even be instantaneous."

"Monosemy is the fundamental condition of logic, but it also defines its limits. In effect, monosemy can only exist within a finite domain of objects and relationships. Logical reasoning, therefore, can only be a moment of reflection, since there is an infinite number of finite domains, however large they may be. Logic appears, then, as a sequence of rational moments immersed in the infinite continuum of the irrational." (footnote, p 3)

"Thought can only be expressed within a system of signs." verbal and written language are codings as are digital representations. "Any transcription leads necessarily to a separation of content from form. The 'content,' those elements of the thought which can remain constant, regardless of the sign-system into which they are translated, must be distinguished from the 'container.' that is the means available in a given system and the laws which govern their use. These elements are constant, whatever the thought to be transcribed." (p 4)

Wednesday, August 15, 2018

Software Art Thou: A summary of the lecture by Glenn Vanderburg from Real Software Engineering

Software Art Thou: Real Software Engineering
A summary of the lecture by Glenn Vanderburg currently VP Engineering at

Zendesk published a Youtube in January 2018 in which Glenn Vanderburg talks about the nature of Software Engineering and offers some insightful comments. Since many of his points are important to the research I am doing I am creating a summary of this Youtube to assist in citation. Parenthetic remarks are time m

Where SE went wrong and how to fix.
(0:27) Programmers: Stop Calling Yourselves Engineers, The Atlantic. Are SEs engineers?
Software Engineering: An Idea Whose Time Has Come and Gone?, Tom DeMarco, Computer Magazine, ACM? IEEE?
"Calling what we do 'engineering' doesn't make us engineers. It makes us liars, Dave (Pragmatic) Thomas, Extreme Programming mailing list at 2003
(2:04) Arguments against SE as engineering reduce to charge that it is fundamentally different from other engineering or that it is more artistic than engineering which essentially is the same argument.

The term Software Engineering can be traced to a NATO publication from a conference in Garmisch Germany in Oct 1968. And then Rome 1969.

He calls the 1969 the launch of Academic Software Engineering and was analytical, formal and aspirational toward engineering.

All academic software engineering (ASE) modeled process models of large scale engineering projects. Requ

Allister Colbert, was proj successful? how strictly to chosen process? PM differed from coders. Would you use this process again? (mostly no)

good people who cared were motivated to subvert process

in mid 90s, attempt to find new process, Agile
less iterative, experimental, less formal, correctness by testing, rapid adoption.
Now two views: software development should be more like engineering and software development is NOT like engineering.

Both ignore what engineering process really is.

Herbert Simon, The Sciences of the Artificial,
A Rational Model of Software , Parnass

(12:50) Glenn's view:
Software development already is engineering
it always was
the rational model (Parnass) is a caricature of engineering since no engineering is really done that ways and some look nothing like that
We don't need to become engineers but refine the software engineering process which does not have to adhere to the academic software engineering processes that were first suggested.

Remainder of talk:
the errors of ASE, what is engineering?, what is SE?, what's next?

The Errors of Analy\tical software engineering
Brian Randell, newcastle ed 1968
history of software engieering conference 1996
At that history of SE cibnference, he said:
"Unlike the first conference, at which it was fully accepted that the term softare engineering expressed a need rather than a reality, in Rome there was already a slighgt tendency to talk as if the subject already existed. And it became clear during the conference that the organizers had a hidden agenda, namely that of persuading NATO to fund the setting up of an Internal Software Engineering Institute. However things did not go according to their plan. The discussion sessions which were meant to provide evidence of strong and extensive supooret for this proposal were instead marked by considerable scepticism, and led one of the participants, Tom Simpson of IBM, to write a splendid short satire on 'Masterpiece Engineering'. IOt was little surprise to any of the participants in the Rome conference that no attempt was made to continue the NATO conference series, but the software engineering bandwagon began to roll as many people started to use the term to describe their work, to my mind often with very little justification. ... I made a particular point for many years of res\fusing to use the term or be associated with any even that used it."   Brian Randell

Early dissents

Alan Perliss. in 1968
A softwae system can best be designed if the testing is interlaced with the designing instead of being used after the design. (TDD)
A simulation which matches the reqrements contains the cotrol which organizes the design of the system.
Througb successive repetititons of this process of interlaced testing and design the model ultimately becaomes the software system itself. ... in effect the testing and the replacement of simulations with modules that are deeper and more detailed goes on with the simulatin model cotrolling, as it were, the place and order in which these things are done.

Fred Brooks, The Design of Design

Winston Royce agreed that waterfall seems like it should make sense but then went on to criticize it in the same paper. (Managing the Development of Large Software Systems

"John H. Van Vleck, Nobel-laureate physicist was Dean of Harvard's Division of Engineer and Applied Science when I was a graduate student there, in Aiken's lab. Van Vleck was very concerned that the practice of engineering e put on a firmer scientific basis. He led a vigorous shift of American engineering education away from design toward applied science. The pendulum swung too far; reaction set in; and the teaching of design has been contentious every since. I am grateful that three of my Harvard teachers never lost sight of the importance of design and taught it: Philippe E Le Corbeiller,  Hary R Mimno, and Howard H Aiken, my adviser."
Preface to The Design of Design by Fred Brooks

Devalued the role of code.
What is needed is not classical mathematics, but mathematics. Systems should be built at levels and modules, which form a mathematical structure." Friedrich Bauer

[...] our basic tools are mathematical in nature. -Edsger Dijkstra

David Parnas 2007 oopsla, "In engineering...people design through documentation,", "Code cannot fill that role."

Academic software engineering emphasized correctness over cost.

graph of cost of error correction versus phase of project where error is found. But, Glenn says, emphasis on careful review of all artifacts at all stages increases costs throughout since more reviews, more quality control is needed. "The cure is worse than the disease." Low cost, high impact. Cost of long feedback loops.

(25:00) What is engineering anyway?
"Unlike the extensive analysis of the scientific method, little significant research to date has sought the philosophical foundations of engineering. Library shelves groan under the weight of books by the most scholarly, most respected people analyzing the human activity called science. No equivalent body of research treats the engineering method." -Billy Vaughn Koen

8 good books on engineering process

The sciences of the Artifiicial, Herbert A Simon
The Design of Design, Gordon L Glegg
Definition of the Engineering Method, Billy Vaughn Koen
To Engineer is Human, Henry Petroski
What Engineers Know and How They Know It, Walter G Vincenti
Engineering and the Minds's Eye, Eugene S Ferguson
The Existential Pleasures of Engineer , Samuel C Florman
The Design of Design, Fred Brooks

[Programming] is not some kind of engineering where all we have to do is put something in one end and turn the crank." -Bruce Eckel

(or paraphrased, it is not algorithmic, nor can it be)

Engineering process is inherently creative, exploratory, and will always have false starts, dead ends, and learning processes.

"The Engineering Method is the use of heuristics to cause the best change in a poorly understood or uncertain situation within the available resources." -Billy Vaughn Koen

Different engineering disciplines are different. Process Control Model, defined at one end, empirical at the other.

"The defined process control model requires that every piece of work be completely understood. A defined process can be started and allowed to run until completion, with the same results every time. The empirical process control model provides and exercises control through frequent inspection and adaptation for processes that are imperfectly defined and generate unpredictable and unrepeatable outputs." -Ken Schwaber and mike Beedle, SCRUM

Engineering which has expensive construction requires are defined.

"The conversion of an idea to an artifact, which engages both the designer and the maker, is a complex and subtle process that will always be far closer to art than to science." -Eugene S Ferguson

"At first sight the idea of any rules or principles being superimposed on the creative mind seems more likely to hinder than to help, but this is really quite untrue in practice. Disciplined thinking focuses inspiration rather than blinkers it." -Gordon Glegg

Deflection theory for suspension bridges. Tacoma Narrows bridge. Deck so thin that it had aerodynamic properties that exciting resonant frequency.

Engineers adopted formal methods to save money. Predicting failure point makes it possible to reduce safety factor.

Structural analyses (indeed any engineering calculations) must be employed with caution and judgement, because mathematical models are always less complex than actual structures, processes, or machines." -Eugene S Ferguson

Engineering driven by practitioners but refined by academics.

Robert Hook, 1671
Sagrada Familia

(49:00)What is Software Engineering?
Structural engineering is the science and art of designing and making, with economy and elegance, ... structures so that they can safely resist the forces to which they may be subjected.
-Structural Engineers Association

In structural engineering the engineers create diagrams and specifications which are given to builders. Historically we map this onto SE by thinking of the designers as creating specifications for the coders. But this hasn't worked well. Are coders really builders?

The source code is not the solution. It is a design document. It is what the compiler that constructs the system that actually provides the solution.

For structural engineering the most expensive phase was construction. But for SE the compiling is the cheapest. With continuous integration this phase is effectively free.

Jack Reeves, what is software design? (

Dave Thomas said XP processes tightly coupled.

Kent Beck wrote first XP book

If you look at the 13 processes, they can be arranged along a continuum of the amount of feedback they give and the scale at which it is given. From the processes that provide the most immediate feedback with the smallest scope is pair programming. At the other end we have practices that give valuable feedback but at a large scale of time and effort in short releases.

pair programming
unit testing
system metaphor
continuous integration
on-site customer
collective ownership
acceptance testing
planning game
short releases.

One suggestion: don't use languages that enable mutable state! (ha)

platform/application divide.

Glenn: stop using unsafe languages for applications.

formal methods to prove validity and security of small micro-kernels. formal methods now practical for well restricted domain.

Glenn Vanderburg,, or @glv

Friday, August 10, 2018

What Does Software Look Like?

I am reading the book, Engineering in the Mind's Eye by Eugene S Ferguson. The book is an in-depth discussion of the design process as found in many areas of engineering. And the focus appears to be on how those forms come to exist. He frequently cites the many paper models of the project to be created. And he stresses the visual component of how the designer tries to document what the finished product and its components look like. But how do I apply that to software engineering? If I am going to specify a system, what images help? What does software look like?

The difference between the surface appeal of a product and the inner workings is not unique to software of course. Something like a motor presents the user with a largely inscrutable object which turns an armature. But we all know that the inner workings of such an engine, or pump, or even an auto are complex. At the extreme we have the computer which has a relatively trivial surface appearance with a screen and a keyboard. But we all know the enormous complexity that exist under the cover. By analogy it is not much different than having a modest cottage that has all the inner workings of a large modern skyscraper. The appearance belies the complexity.

It is not as if the surface appearance does not matter. Quite the contrary. Working out the human-computer interface is a very important part of the design process. Not even as much because of the importance of that interface for the successful operation of the product, but because it is the way that most users can most easily explain the expected behavior, and the complex inner state transitions that are expected.

Some software product however have no human-interface design at all. Embedded systems have an interface of course. But that interface has no user perceived behavior. It becomes a piece of the infrastructure like the plumbing that we never want to deal with. And as long as it works, we don't need to.

Yet these are all important applications of software technology. And we must find a way to understand how to specify the software we need and communicate that to those who can build that for us. And much is the design literature leads us to believe that this division of designer and implementor is something artificial even while scale may require some specialization by skill.

So in this book I am hoping to find the start of some answers about what software looks like. When I engage in creating a blueprint for a software system, what should that look like? What should be visual? Are UML diagrams the right way to communicate intentions? What must be tabular? or do we need new extensions to computer languages to capture the information that right now seems to imperfectly flow from designer to builder in the processes we use today?

Monday, August 6, 2018

Software Engineering and Society

The Atlantic ran a provocative article a few years ago and I take strong exception to the points it makes. Titled Programmers: Stop Calling Yourselves Engineers ( I can agree with the point that much software engineering fails to live up to the aspirational goal of being like structural engineering. But by painting with such broad strokes the article does more to set that aspirational progress back than it does to propel it forward.

First let's consider the basic metaphysics of what a software engineer must do. Advances in steel are small and incremental. Yet changes in hardware design are still following Moore's law many decades after it was first recognized. In addition operating systems have grown in size, complexity and function just as fast as the underlying hardware. How is it possible to imagine the kind of well defined processes we find in structural engineering we see today if those engineers had such dramatic changes in their raw materials? Or if the scale of the structures they built kept pace with the growth of software systems? We would not have the dependability we see today. To suggest that software engineering is failing because it is not on par with structural engineering is a destructive false equivalence.

Next the author talks about how society has regulated some engineering disciplines. Yes, some engineering areas are subject to laws, have well defined apprenticeship tracks and continuing education requirements. But to assume that the mere adoption of those superficial forms would somehow achieve the desired effect of creating fewer software engineering failures is naive at best. Those professions themselves evolved those norms after decades, or even centuries, of practice. Software engineering today bears very little resemblance to the practice of even 30 years ago. We continue to experiment with the processes by which products are created. And the needs of the clients are so disparate that it is obvious the methods we use to created embedded systems for high safety applications bears only the vaguest similarity to the free social media apps the author cites.

Another false equivalence in the article is the criticism of the data breaches that are now getting so much criticism. The way in which we are using computer systems today is far more radical than the way we are using bridges. If society and its institutions continue to use software in novel ways we must expect that we will uncover new problems. Few thinkers in 1968 could have anticipated the kind of cyber espionage and warfare we must prepare ourselves for today. To place the blame for these failures at the feet of the software engineers who were as trusting as the people who commissioned them is hardly fair.

The author's charge of "engineerwash" is not fairly placed against engineers. Like all attempts to manipulate public trust this needs to be assessed against the organizations that profit from the work of those engineers. Engineers are rarely granted the opportunity to design the products they will build without significant input from their clients. Budget, time frames and quality are almost entirely set by the client and not the engineer. As long as we have a capitalist economy, there will be tension between engineers who may be inclined to over-engineer the product and the entrepreneurs who have incentive to under-engineer them. A great deal of legislation was introduced to affect this balance and check the worst impulses of decision makers who profit from the moral hazard.

The author does do a service to the community by pointing out the most obvious way in which engineering software systems is fundamentally different than a discipline of physical rather than information forms. Yes, changes to software alone are almost trivially made compared to structures of steel. And a great deal of rapid advance was enabled by recognizing this in our processes. If we had continued with the kind of waterfall processes that dominated most other engineering processes, many of the largest corporations on the planet today would not yet have been profitable and probably would never have been built. There is clearly a tension in practice between the need to do some level of high level design and yet embrace evolutionary design.

But process is not one-size fits all. The author glibly flits from low risk products such as social media platforms through large enterprise wide systems like airline reservation systems to individual products like spreadsheets to embedded systems with human life and safety concerns. No single process is appropriate for all of these products. The most safety critical benefit from reusing proven designs on stable platforms that have been exhaustively tested. But a great deal of wealth has been created by relaxing those constraints to explore new product categories and experimental systems. The matching of the process to the product goals is something an engineer can influence but is hardly the sole decider. And this leads into some of the most disingenuous parts of this article.

Software engineers are not unique in having clients who wish to employ them to achieve their organizational goals. A necessary feature of all engineering, as opposed to pure science, is the reality of economic constraints. These constraints are mediated by the client and are subject to limited negotiation by the engineer. We have already seen some engineering failures that were predicted to some degree of certainty by engineers such as the Saturn O-ring disaster. Engineering decisions are not made solely by engineers. To blame them for all failures is bordering on scurrilous.

Let's face facts here, Facebook is nothing like the Golden Gate Bridge. A complete failure of that platform would not end lives are throw a region into chaos. To suggest that the same standard of care should be exercised equally over both is polemic and unfair. Perhaps the notion of "move fast and break things" was in retrospect a bit naive. But the entire history of innovation is really no different. If one cannot take chances with a novel and uncritical application like a social network, then when are risks worth taking? As long as the publics health and safety is not placed at risk, the suggestion that heavy handed approach to regulating the software engineering profession betrays an officious attitude toward business.

The author also charges that software engineering has become hermetically insulated from the world. As if rocket science is not?

By page 7, the author has come back to the issue of professional licensure. I am old enough to remember these attempts. But I already pointed out how quickly the professional has changed. I looked at these early attempts with scorn at the superficiality of the certifications. Even green hiring managers could more accurately divine the capability of a potential engineer than these pseudo certifications.

I'll offer one aside on this issue. It is traditional that engineers have a four year degree. But there is scant evidence that the degree, by itself, is a reliable predictor of the engineers innate ability. While it is common that corporations will impose that restriction on hiring, it is usually out of expediency in filtering resumes more than empirical evidence that they make better engineers 4 years after hire. And given the cultural filtering of who can achieve a four years degree, it becomes more of a social filter than a technical one. This argues for the prejudice that Google and Thiel promote and which the author sneers at.

I feel I have addressed the most superficial faults of this article. But there is much more we can learn from some issues raised. One is the lure of entrepreneurship. He is absolutely correct that the ability to create a prototype system by a small group of entrepreneurial software engineers and make a great deal of money by selling this promise changes the very nature of what it means to be a software engineer. These engineers have merged the role of developer and business risk taker just as many scientists have done the same when developing their contributions to basic science. But no, this avenue is NOT turning computing into a type of speculative finance simply because the number of engineers inclined, or in a position, to take those risks is a minority. But he glosses over why this avenue is so attractive. He blithely says that engineering has always been a well-paid profession. Perhaps to a journalist.

Given the explosive growth in the technology, there has always been a severe shortage of talent compared to the appetite of the market. But like so many other labor markets in the US, this market mismatch does not explain the compensation. Consider the different economic outlooks of a hedge fund manager who can act as angel investor compared to the economic outlook of the engineer who creates the product. One can consistently outperform the other by a factor of at least 10, often far more. The basic capitalistic system encourages the use, some can say exploitation, of the engineer to remain in a risk free, but limit compensation role. How is it any wonder that young and talented developers are more inclined to explore being entrepreneurs rather than engineers?

Add to this structural bias toward risk taking the fact that powerful corporations can use selective manipulation of immigration laws to grant themselves economic relief. By keeping salaries low in the US for software engineers, they control the flow of students who choose it for a profession, When the inevitable skill shortage affects them, they claim an exception to immigration laws to allow for more software engineers to enter the US. If this were done fairly by giving them green cards I would accept it as part of the inevitable globalization of labor. But that is not what is done, these corporations go along with restrictions of these foreign workers that limit their ability to respond rationally to the US labor market and go where they can be paid the market rate for their skills. The author misses a great opportunity to make an important point about the power of the engineer in society by ignoring this.

The author cites the difference between many other professional engineers and software engineers and I made some comments about the false equivalence. However I will end by saying that I think the author does touch on something that is actually a valid criticism although I doubt they understand how valid a criticism it is. ABET is the standard by which most university programs are judged. A dirty little secret is that the computer science curriculum has fundamentally changed very little since 1968. And for prestige universities computer science is the program an aspiring software engineer must navigate to have the best chance of securing a position after graduation. Yet as the knowledge in the industry has exploded, the education system has not kept up. Once someone who wanted to study computer science got a math or physics degree. We are long past the time when meaningful distinctions can be made between computer science and software engineering. Why haven't we? Why has the gulf between academia and practice grown? Why are the first two years of a software engineer spent learning things not taught in school? And by this criticism I am not talking about skills training which by necessity is the responsibility of the employer. But instead a more mature approach to the entire software lifecycle than a very superficial mastery of basic coding, if even that.

The author accurately says that software engineering titles are aspirational more than an accurate description of what goes on in the industry or taught by the schools. But on the whole the article comes across as an uninformed hit piece that is designed to get attention by offering half truths about the role of the software engineer today. It missed a great opportunity to make some very valid criticisms of the role of software and the role of software creators in today's world.