Software Engineering and Society

The Atlantic ran a provocative article a few years ago and I take strong exception to the points it makes. Titled Programmers: Stop Calling Yourselves Engineers (https://www.theatlantic.com/technology/archive/2015/11/programmers-should-not-call-themselves-engineers/414271/) I can agree with the point that much software engineering fails to live up to the aspirational goal of being like structural engineering. But by painting with such broad strokes the article does more to set that aspirational progress back than it does to propel it forward.

First let's consider the basic metaphysics of what a software engineer must do. Advances in steel are small and incremental. Yet changes in hardware design are still following Moore's law many decades after it was first recognized. In addition operating systems have grown in size, complexity and function just as fast as the underlying hardware. How is it possible to imagine the kind of well defined processes we find in structural engineering we see today if those engineers had such dramatic changes in their raw materials? Or if the scale of the structures they built kept pace with the growth of software systems? We would not have the dependability we see today. To suggest that software engineering is failing because it is not on par with structural engineering is a destructive false equivalence.

Next the author talks about how society has regulated some engineering disciplines. Yes, some engineering areas are subject to laws, have well defined apprenticeship tracks and continuing education requirements. But to assume that the mere adoption of those superficial forms would somehow achieve the desired effect of creating fewer software engineering failures is naive at best. Those professions themselves evolved those norms after decades, or even centuries, of practice. Software engineering today bears very little resemblance to the practice of even 30 years ago. We continue to experiment with the processes by which products are created. And the needs of the clients are so disparate that it is obvious the methods we use to created embedded systems for high safety applications bears only the vaguest similarity to the free social media apps the author cites.

Another false equivalence in the article is the criticism of the data breaches that are now getting so much criticism. The way in which we are using computer systems today is far more radical than the way we are using bridges. If society and its institutions continue to use software in novel ways we must expect that we will uncover new problems. Few thinkers in 1968 could have anticipated the kind of cyber espionage and warfare we must prepare ourselves for today. To place the blame for these failures at the feet of the software engineers who were as trusting as the people who commissioned them is hardly fair.

The author's charge of "engineerwash" is not fairly placed against engineers. Like all attempts to manipulate public trust this needs to be assessed against the organizations that profit from the work of those engineers. Engineers are rarely granted the opportunity to design the products they will build without significant input from their clients. Budget, time frames and quality are almost entirely set by the client and not the engineer. As long as we have a capitalist economy, there will be tension between engineers who may be inclined to over-engineer the product and the entrepreneurs who have incentive to under-engineer them. A great deal of legislation was introduced to affect this balance and check the worst impulses of decision makers who profit from the moral hazard.

The author does do a service to the community by pointing out the most obvious way in which engineering software systems is fundamentally different than a discipline of physical rather than information forms. Yes, changes to software alone are almost trivially made compared to structures of steel. And a great deal of rapid advance was enabled by recognizing this in our processes. If we had continued with the kind of waterfall processes that dominated most other engineering processes, many of the largest corporations on the planet today would not yet have been profitable and probably would never have been built. There is clearly a tension in practice between the need to do some level of high level design and yet embrace evolutionary design.

But process is not one-size fits all. The author glibly flits from low risk products such as social media platforms through large enterprise wide systems like airline reservation systems to individual products like spreadsheets to embedded systems with human life and safety concerns. No single process is appropriate for all of these products. The most safety critical benefit from reusing proven designs on stable platforms that have been exhaustively tested. But a great deal of wealth has been created by relaxing those constraints to explore new product categories and experimental systems. The matching of the process to the product goals is something an engineer can influence but is hardly the sole decider. And this leads into some of the most disingenuous parts of this article.

Software engineers are not unique in having clients who wish to employ them to achieve their organizational goals. A necessary feature of all engineering, as opposed to pure science, is the reality of economic constraints. These constraints are mediated by the client and are subject to limited negotiation by the engineer. We have already seen some engineering failures that were predicted to some degree of certainty by engineers such as the Saturn O-ring disaster. Engineering decisions are not made solely by engineers. To blame them for all failures is bordering on scurrilous.

Let's face facts here, Facebook is nothing like the Golden Gate Bridge. A complete failure of that platform would not end lives are throw a region into chaos. To suggest that the same standard of care should be exercised equally over both is polemic and unfair. Perhaps the notion of "move fast and break things" was in retrospect a bit naive. But the entire history of innovation is really no different. If one cannot take chances with a novel and uncritical application like a social network, then when are risks worth taking? As long as the publics health and safety is not placed at risk, the suggestion that heavy handed approach to regulating the software engineering profession betrays an officious attitude toward business.

The author also charges that software engineering has become hermetically insulated from the world. As if rocket science is not?

By page 7, the author has come back to the issue of professional licensure. I am old enough to remember these attempts. But I already pointed out how quickly the professional has changed. I looked at these early attempts with scorn at the superficiality of the certifications. Even green hiring managers could more accurately divine the capability of a potential engineer than these pseudo certifications.

I'll offer one aside on this issue. It is traditional that engineers have a four year degree. But there is scant evidence that the degree, by itself, is a reliable predictor of the engineers innate ability. While it is common that corporations will impose that restriction on hiring, it is usually out of expediency in filtering resumes more than empirical evidence that they make better engineers 4 years after hire. And given the cultural filtering of who can achieve a four years degree, it becomes more of a social filter than a technical one. This argues for the prejudice that Google and Thiel promote and which the author sneers at.

I feel I have addressed the most superficial faults of this article. But there is much more we can learn from some issues raised. One is the lure of entrepreneurship. He is absolutely correct that the ability to create a prototype system by a small group of entrepreneurial software engineers and make a great deal of money by selling this promise changes the very nature of what it means to be a software engineer. These engineers have merged the role of developer and business risk taker just as many scientists have done the same when developing their contributions to basic science. But no, this avenue is NOT turning computing into a type of speculative finance simply because the number of engineers inclined, or in a position, to take those risks is a minority. But he glosses over why this avenue is so attractive. He blithely says that engineering has always been a well-paid profession. Perhaps to a journalist.

Given the explosive growth in the technology, there has always been a severe shortage of talent compared to the appetite of the market. But like so many other labor markets in the US, this market mismatch does not explain the compensation. Consider the different economic outlooks of a hedge fund manager who can act as angel investor compared to the economic outlook of the engineer who creates the product. One can consistently outperform the other by a factor of at least 10, often far more. The basic capitalistic system encourages the use, some can say exploitation, of the engineer to remain in a risk free, but limit compensation role. How is it any wonder that young and talented developers are more inclined to explore being entrepreneurs rather than engineers?

Add to this structural bias toward risk taking the fact that powerful corporations can use selective manipulation of immigration laws to grant themselves economic relief. By keeping salaries low in the US for software engineers, they control the flow of students who choose it for a profession, When the inevitable skill shortage affects them, they claim an exception to immigration laws to allow for more software engineers to enter the US. If this were done fairly by giving them green cards I would accept it as part of the inevitable globalization of labor. But that is not what is done, these corporations go along with restrictions of these foreign workers that limit their ability to respond rationally to the US labor market and go where they can be paid the market rate for their skills. The author misses a great opportunity to make an important point about the power of the engineer in society by ignoring this.

The author cites the difference between many other professional engineers and software engineers and I made some comments about the false equivalence. However I will end by saying that I think the author does touch on something that is actually a valid criticism although I doubt they understand how valid a criticism it is. ABET is the standard by which most university programs are judged. A dirty little secret is that the computer science curriculum has fundamentally changed very little since 1968. And for prestige universities computer science is the program an aspiring software engineer must navigate to have the best chance of securing a position after graduation. Yet as the knowledge in the industry has exploded, the education system has not kept up. Once someone who wanted to study computer science got a math or physics degree. We are long past the time when meaningful distinctions can be made between computer science and software engineering. Why haven't we? Why has the gulf between academia and practice grown? Why are the first two years of a software engineer spent learning things not taught in school? And by this criticism I am not talking about skills training which by necessity is the responsibility of the employer. But instead a more mature approach to the entire software lifecycle than a very superficial mastery of basic coding, if even that.

The author accurately says that software engineering titles are aspirational more than an accurate description of what goes on in the industry or taught by the schools. But on the whole the article comes across as an uninformed hit piece that is designed to get attention by offering half truths about the role of the software engineer today. It missed a great opportunity to make some very valid criticisms of the role of software and the role of software creators in today's world.