Friday, August 10, 2012

Free Software and Communism

Sac State has a program through MSDN that provides their current OS for free to the students. UC Davis does not (or at least I haven't found it yet if they do). Curious. But the bigger issue is my changing thoughts about what I expect to spend for software.

In the 30 years I have had an association with Microsoft I have probably spent in excess of $10,000 buying their products. Despite this continued relationship Microsoft doesn't know I exist and provides some of the worst customer service I have ever experienced. I have had a few times where it was a bug in their product I wanted to talk about and have been given the run-around or been asked to fork over $100 to talk to someone. Microsoft has historically charged too much for an inferior product for far too long. They seem to relate to the consumer market the way a farmer relates to a field crop.

While I myself am not the sort inclined to tinker with an OS, many people I have known are. The Linux and open source movement have been incredibly empowering for those of us who have the capability and inclination to tinker. First it is free. That of course is a huge advantage seeing that Microsoft has the audacity to float an undiscounted price for Windows 7 that is over $500. If I hadn't been getting their products for free from school I would have abandoned them long ago. There is a certain hubris that the people they might want to recruit to write operating systems would be asked to spend a significant amount of money in their education using scarce resources to buy their products. It would be in their interest to ensure that every CS major receives a generous support from them if for no other reason than good PR. So at some level good CS students must find it galling that they are asked to pay a high price for an inferior product. Is it any wonder that it is often students are the most creative in finding ways to avoid paying for their products?

Since it is CS students and the professionals they become use software as their tools of the trade daily over their careers, it makes perfect sense that they will be the most critical consumers of these tools. Any worker who uses a tool daily will choose and maintain her tools with great care. When the market provides only a mediocre tool, there is an opportunity for some worker to turn their attention to the tool. Of course many workers do not have the skills to tinker with their tools. A machine shop worker probably does not have the technology at hand to tinker with a high-precision lathe. But a software engineer with the source code for a major piece of system's software does. While it is only a minority of these engineers who are inclined to tinker, it is enough to create an alternative in the tool market. We have certainly seen that exhibited in the market for open source software and the active support it gets.

In some idealized libertarian universe, these talented tinkerers could have formed an alternative to Microsoft and made some money from their collective efforts. But creating and managing a business is not a trivial affair and certainly not what motivates talented software engineers. Engineers in general are great admirers of functionality and not of management. Their are significant intrinsic rewards to the creation of a good product but few in the marketing and sales of that product to them. Yes, there are some rare individuals who excel at both. But their scarcity relative to the number of highly talented engineers is precisely my point. The open source model was appealing to them because it gave them control over their means of production and empowered them to do things for their clients that are impossible in a closed  model.

Some have tried to cast these engineers as proto-communists contributing to a common good. (from each according to his ability, to each according to their need). But if that attitude exists in the open source community, I have not yet seen it baldly stated. Rather than being driven by some ideological position, the market seems to be moving along a highly pragmatic path of least resistance. The closed software market has not met their needs to provide high-quality, responsive solution to the clients using their tools. Instead of forming highly responsive relationships with these integrators, the companies seem to have put their needs behind the company's. Doesn't it make sense that someone responsible for providing the company with a high-quality web server would be more inclined to support Apache than IIS? It is in their self-interest to choose this simply because they have greater control over the resolution of any problem or enhancement than they would over a close product at the leading edge of its deployment. The fact that other people gain access to a high quality product as a consequence is irrelevant to them and takes nothing from them. In fact, the widespread adoption of Apache enhances their marketability since employers who desire to avoid IIS, for whatever reason, will seek them for employment.

No, I am not finding any incipient communism in the growing open source movement but instead a marketplace that is responding to the maxim 'information wants to be free" plus an enlightened self-interest among the most talented software engineers of our day. They seem to be having a blast. Now if we can only figure out where the margins of this new market lie.

Wednesday, August 1, 2012

A First Look at Commit Data for an Open Source Software Project

I got a little something to sink my teeth into. I am looking at commit data for some open source projects. This is mostly an exercise in regaining my sql chops and learning R. Here is my first plot:


The plot has the alias of the submitter along the X and the timestamp along the Y. What you can see for this one project is that there are two heavily dedicated submitters who both started working on the project about the same time, one who is more sporadic and started shortly before the two of them and two who appear to have started the project but only periodically commit although their activity is relatively consistent over the entire length of the project. What is somewhat surprising is how many there are who have almost no commit activity (there is some doubt regarding whether this alias is the submitter or committer although it is supposed to be the commit id). It seems odd that someone would have gained committer status and then stopped commiting. Will I see this pattern in other projects? Who are these heavy committers and how do they differ in role from the people who appear to have started the project?

Tuesday, July 10, 2012

Will Open Source Software Someday Rule the World?

I've been listening to the ChangeLog and reading a lot about the explosive growth of open source software. There is no doubt that this phenomenon is creating innovative products and exploring new ways for creative individuals to self-organize. I forgive young software engineers for thinking this is a complete game changer and that all software in the future will be developed using this model. Perhaps they're right but I don't think so. I want to explore this assertion and see what the data shows. Let's start with the Apache Foundation.

In their how-it-works page, they describe how the Apache server source code was taken from the NCSA HTTPD but was orphaned when the original developers moved on to other projects. Brian Behlendorf created a mailing list of other admins using the product to communicate patches on that code base. That original group continued to organize and formalize. They then began to sponsor allied projects. All the projects were organized to grant authority to volunteers after they had proven their abilities; a culture of meritocracy. This group was not only users of the products, they also had the skills to modify the product and the reasons to do so. Their working lives revolved around the functions of these products and many changes were either self-serving or would please they people who they served.

These open source projects create a resource that is designed as non-excludable and non-rivalrous. Anyone who find the repository is welcome to download the software product and use it. The use of the product by one person does not preclude the use by anyone else. In many ways this is the classic public goods model. What is interesting is how the FLOSS projects do not seem to suffer from the free loader problem or any form of tragedy of the commons.

Someone who downloads a software product from a FLOSS (Free/Libre Open Source Software) repo will often have access to a significant asset. The Apache web server is a sufficient example. This product competes quite well against products from for-profit enterprises and could possibly attain a dominant position in the marketplace, if it hasn't already. But to achieve and maintain a market position like this required a significant investment in effort at one point. In the case of the Apache webserver it was NCSA. If the creation of a superior product that is then gifted to the public, this matter would be easily understood. The Apache webserver was not a single creation that was gifted since the needs of webservers have changed dramatically over the years. In fact it was the new needs that motivated the creation of the project in the first place. Since the product was already a reasonable base upon which to build, and since the needs affected many users of the product, a natural mutual interest was created and nurtured by the communications channels that were available. This is still the model for most people of a FLOSS project.

If the common asset were open to modification by anyone who had access to it, it would surely have been doomed by the tragedy of the commons. Each person who thought they knew what they were doing would have made changes and some of them would have broken the system. The form of ownership that evolved was that only a subset of people would have commit rights and they already had established enough trust in each other to be assured that they would act for the common good of the project. Since these "managers" depended upon the product, their interests converged.

This software, unlike physical public goods, could be consumed without limit as it existed. But to remain viable in the marketplace, it needed to evolve to meet the changing requirements, lest it become obsolete. Here is where the Apache project had a distinct advantage; each consumer also had the requisite tools and knowledge to adapt the product for the changing needs. These changes were insignificant compared to the complete body of code and the cost of creating each patch was also insignificant to the cost of changing to another product. Once the patch had been created, the marginal cost to share it with the community was insignificant and its value to the individual was limited unless that developer/user chose to take on the responsibility for creating a marketable product from the FLOSS base. Since this was not compatible with that person's core mission (we assume), they chose to contribute. Since each person was acting in their self-interest, the collective effort not only allowed the product to evolve to meet new demands but could also spawn new products as needs diverged.

So what are the elements to create a successful FLOSS project? First, there must be some base of code that provides enough out-of-the-box functionality to answer some need by a user. Second, there must be some core set of users who have a shared vision of what the product is and can agree on commits to that product. It seems as if this core group must initially be the primary source of commits as well as be willing to support new users of the product. Third, the end-users of the product must also possess the tools, skills and motivation to patch the product when it does not meet their needs and offer these patches back to the project. It is possible that the work of Ronald Coase may be helpful in the analysis of FLOSS projects.

The FLOSS market has changed over the years and it appears that other models are appearing. In particular the market for ERP software in FLOSS looks very different. In this case, the initial investment to create such a product is much higher and the market much thinner. What organizations now seem to try is to offer a "community version" of their product which is appealing to some subset of the market and then use it as a loss leader or teaser to find others who might be willing to purchase a traditional license for the full-product.

Another model I see in the repos is people who have come up with a relatively small widget which they have gifted. It is not clear if their hope is to spawn a new open source project from this seed or whether they are just engaging in a form of gifting. My instinct tells me the chances of a relatively small product spawning a viable new project is small. First, it may make more sense for the the consumer to merely copy the code into their own repository rather than take the product in as a component. If that is the case, the opportunity to achieve a back channel of patches to the product will be undermined and the product will remain static. If there are any changes in the needs, the product is likely to wither from the lack of contributions to keep it viable in the market. Second, without a core of developers who see value in providing the oversight to the project that is needed to avoid code rot, the burden will fall entirely upon the original developer. If that developer does not make frequent mods to the code herself, it is likely to become a burden since there will be a limit to the time that the developer will devote to that one product.

Eclipse seems to be following a different model where instead of individuals supporting the product it is organizations. This is an area I need to research since it is not clear to me how IBM benefits from the expense in Eclipse.

Thursday, June 28, 2012

Parnas on Open Source Software

oHere is a link to a chapter that was co-authored by David Parnas on open source software. http://mitpress.mit.edu/books/chapters/0262562278chap6.pdf In it, he expresses some concerns about the ability of open source software projects to produce high-quality products.

There is also this link which I like... http://economics.mit.edu/files/1125. It has to do with qualitative research in economics

Wednesday, June 27, 2012

Does Marx offer any insight into FLOSS?

FLOSS (Free/libre open source software) has occasionally caused people to ask if it is communist over the past decade.  I'll ignore the trolling aspect that this question raises and jump right to the ways in which it can offer some frisson with FLOSS.

You cannot look at FLOSS for long before seeing the communal aspect of these projects. Most recently the question was posed on Quora and I posted a lengthy answer there. (http://www.quora.com/Is-free-open-source-software-communist/answer/Dale-Fletter). I also answered a different question that I see as closely related having to do with how FLOSS has impacted commercial software development (http://www.quora.com/Software-Engineering/What-is-the-impact-of-the-open-sourcing-philosophy-on-proprietary-closed-software/answer/Dale-Fletter). Since that forum does not reward deep intellectualism and since this journal is my primary data base of ideas, I will try to repeat or improve my answer there. 


A topic of discussion for the early economists was the tragedy of the commons. Since each person is naturally inclined to consume from the commons and make no contribution, this is a problem in a laissez-faire economy. (it is personally disturbing to me how little the current political parties that embrace libertarianism adequately deal with this fundamental problem). What I see in the current FLOSS market looks surprisingly like a reverse issue to the tragedy of the commons, let's call it the gift of the commons. In the FLOSS market consumption of the commons does not diminish it in any significant way. Yet because of the relationship people have with the commons, they are inclined to contribute to it leading to an increase in its value to everyone. This is clearly a way in which the assets are held in common for the good of everyone. It is also clear that many of these assets provide a means of production for many of the users of the commons. This forces the question, is this a form of communism?


To help me understand the question, I needed to put aside my own prejudices about communism and chose to educate myself on the source of much of its intellectual underpinnings in Karl Marx. The book I chose to read first is Karl Marx by Isaiah Berlin. I first tried to read The Capital/Das Kapital but abandoned it since his prose is painful and the progression of his ideas is glacial for a modern reader. The other advantage of Mr. Berlin's history is the cultural and intellectual context and the critical nature of his reporting. I highly recommend the book.


Marx had a deeply held point of view that has been largely debunked by the developments of the 20th century. Therefore his theory about class warfare and the inevitable fall of capitalism makes no sense any longer. However his early analysis of capitalism is surprisingly fresh and modern in its insight, if not always completely original. In my opinion, he was an empiricist, he believed that capitalism had inherent defects that were inimical to the inherent rewards of work and that without leavening, capitalism could not survive. I believe the history of the late 19th and all of the 20th century have proven this thesis to my satisfaction despite the current debate. What I find enthralling is how he seems to capture the spirit of many software engineers who contribute to FLOSS. There seem to be many motivations in their work and Marx, for me, seems to offer some insight into this emerging market. It is his intention to use a scientific method that includes economics and the other social sciences to observe, record, discuss, explain and theorize about what he saw as a model for the analysis of FLOSS.

Regarding qualitative analysis, Berlin observes, "This command to look for the most vivid expression of the universal in the particular, the concrete, the differentiated, the individual, to emulate the art and the realism of the biographer and the painter rather than the photographer and the statistician, is the peculiar legacy of German historicism." As I look at FLOSS, I intend to follow this command, at least for now.

While the broader concept of class means something very different today than it did at the time of Marx, let me first make some observations about class relative to FLOSS. Communication mediated by the software repositories created for FLOSS removed almost all traces of class that might intrude from outside. Many people involved do so anonymously and provide no meaningful picture of themselves outside of the project. This provides an opportunity for people to recreate themselves in this sphere. Coupled with the staunch support for meritocracy, someone with no power, status, or obvious class affiliation outside of the project could rise to the level of contributor. (it looks like anonymous contributors with commit rights don't exist in Apache but that needs more research). Beyond the hierarchy that exists due to this meritocracy, there does not seem to be any inherent class structure, at least within the confines of a project. Without a class structure, I don't see where any class friction can exist.

As I already observed, the commons is not diminished by the copying of the software. Also the project imposes no coercion for contribution. This is perhaps the most pure expression of the communist maxim of "from each according to their ability, to each according to their needs". (I note that the phrase, while popularized by Marx owes its origin to Louis Blanc).  Of the abilities that contribute to the projects, there are many. Some have very fundamental needs to fix the errors in these machines to keep them running for the enterprises that are important to them. There are visionaries who pursue new features with an artistic zeal. Some fall in love (eroticize) the product and take a proprietary interest in making it the best possible product in the same way that Pygmalion created his beauty. The lack of a centralized plan would seem to diminish the utility of the final product but that is difficult to see in many cases.

Much has been written about the alienating effect of monads slaving over keyboards. Perhaps a motivation for contributors is the sense of belonging that active participation in a project community brings.

One of the aspects of FLOSS that most strikes me is that the utility of the projects does not cover the universe of software uniformly. There is a decided bent toward tools and utilities versus consumer products and other products that do not have well trained individuals as their primary consumers. By this I am talking about the preponderance of languages, web development and web deployment tools, databases and other tools of the trade. From the operating system to the creation of an industrial grade online service, it is possible to avoid paying a corporation any licensing fees. I am not aware of any period of human history when the tools of production have been so freely available. This represents another basic Marxist maxim that the workers should own the means of production. We now have an entire generation of web developers who are created significant value using the tools from the commons. It is their enlightened self-interest which causes them to contribute so as to further rationalize or extend these tools. Some even find ways to tweak the tools to create new market opportunities for themselves and garner personal rewards by bringing this value to market before anyone else.

It is the interplay between the commons and the for-profit marketplace that causes many people to question the label of communism. They call it a "gift-economy" or some other such label. What distinctions are they trying to paint? I still have much research to do in the area of alternative economies, in particular those related to the gift economy. However I am going to indulge my going in positions. 

Wikipedia talks about how early file-sharing was a form of gift-exchange that fostered a sense of community. I'm not buying that. FLOSS has a very specific aspect that cannot be seen in music or most forms of art. When I buy a song, I now have a static object which will not change during the time it is of interest to me. Contrast that with a piece of complex software like an operating system. The utility of that download may last for a period of some years. However if it has become part of a larger system, I will not be inclined to abandon my investment of time in that larger system once that component is no longer completely compatible with the dynamic environment in which it exists. It is for this reason that large complex software has a value chain that includes periodic updates. And it is exactly this value chain that FLOSS most brilliantly fulfills. The collective work of the various parties who attach to that value chain ensure that the product is dynamic and continually adapting to its environment. As I become aware of new functionality that must be added to the version I have, I have some assurance that this project will have organically anticipated that need and already provided for it. Even if I am the first, there will be others who will benefit from what I may be forced to do as a matter of personal need. My contribution builds goodwill and community as I discuss my need with others who have the same or similar needs.

While I reject the idea of music file sharing as being similar to FLOSS as a form of gift economy, I completely agree that this community building is an inherent part of the FLOSS experience. Especially in their formative stages, FLOSS projects have a relatively small number of contributors. The inherent communication among these contributors creates a sense of personal intimacy that build community bonds. Trust grows. Shared context grows. Often new ideas sponaneously emerge from the discussion in one project that spawn another and a new project is born.

Another aspect of FLOSS that I find in Marx's writing has to do with the basic notion of work. In the traditional Theory X of work, an organization must provide incentives to work. Without them, people will be inclined to not produce. Both Marx and then later Theory Y suggest that work has intrinsic fulfillment. Taken with Maslow's hierarchy of needs, it appears that FLOSS attracts people who gain significant rewards from the support they give to FLOSS. What must be explored is to what extent this interacts with other forces to enable that. For example, if software developers who are most active in FLOSS are being paid by their employers to contribute, can we be assured that this has achieved some form of market stability? Software engineering salaries are still relatively high but they have begun to show some signs of market volatility as foreign countries have shown their ability to compete for this labor. There can be no doubt that some aspects of software engineering will become less financially rewarding in the future. Will this lead to a downturn in the FLOSS contribution?

Another interplay with FLOSS is those developers who were fortunate enough to have a financial success with an earlier product and now use FLOSS as an incubator for a new idea. Clearly anyone who has become independently wealthy does not represent a stable vision for the future of FLOSS.

FLOSS may become a proving ground for young talent. People in school or recently out who are still supported can afford to invest significant time contributing to a project of their choice. Is it possible FLOSS will become a form of Peace Corps where the young become apprenticed and learn their trade while simultaneously provide some societal good?

Modern economic thought does not see the world in the kind of binary fashion that was prevalent in the 1950s. Modern western economies are mixed with elements of both capitalism and socialism mixed. I cannot see any reason why there cannot be stable islands of communism that coexist in our economy. There are many utopian communities that have existed side-by-side for many years. Both models of economies seem flexible enough to allow for mutual coexistence.

The final point I see in FLOSS is the force of idealism and youthful optimism. Virtually all valley jobs involve a certain amount of kool-aid drinking. The most pure forms of capitalism involve a quasi-warfare mentality of competition in the marketplace. But just as effectively, you see many young people (and some older folks too) who become committed to an idealistic path that stands in stark opposition to the free-market capitalism. FLOSS seems to draw from this pool of talent as well. Perhaps it even offers some solace from the brutality of an unbridled capitalist environment and a place where people can go to mentally recuperate and plan their next phase.

Software is nothing like the commodities that Marx focused on. It seems reasonable that any expression of Marx's ideology that grows out of an economy based on software will differ. Perhaps it does it a disservice to call it communism. I'll need to look at these alternative economies to see what I think.

***

One of the thinkers who inspired Marx was Pierre-Joseph Proudhon who is best know by his saying that property is theft. In his essay "What is Property?" he makes a distinction between property owned but fallow versus property that is in productive use which he calls possession. That distinction reminds me of a problem that Google is facing in trying to digitize books. There are a great many books that are long out of print but still have active copyrights. The owner of that right can block the publication in any form. There is something that strikes me as inherently wrong to own something for the sole purpose of preventing anyone else from benefiting from it. If it is frictional in the sense that it is under current negotiation, then it is merely a strategy and a short-term dislocation. But if the owner is unknown or beyond negotiation, then the information is available in the limited number of books that exist but not readily available. The inevitable consequence of this is rampant copyright infringement for a popular item and the complete loss of the information from sources that no longer exist. This becomes a form of information rot. I rather admire Google for continuing to scan the books that fall into this category since they are actually performing a public good by ensuring the information will never be lost to theft or fire even while the owner may block the distribution.

Thursday, June 14, 2012

What is a computational object?


In this sense, externalizing ideas is not a matter of emptying out the mind but of actively reconstructing
it, forming new associations, and expressing concepts in external representations while lessening
the cognitive load required for remembering them: “Externalization produces a record of our mental
efforts, one that is ‘outside us’ rather than vaguely ‘in memory’. ... It relieves us in some measure from the always difficult task of ‘thinking about our own thoughts’ while often accomplishing the same end. It embodies our thoughts and intentions in a form more accessible to reflective efforts.” ((Bruner, 1996), p. 23)

http://l3d.cs.colorado.edu/~gerhard/papers/fi_ost-final.pdf


The computational viewpoint is directly concerned with the distribution of processing but not with the interaction mechanisms that enable distribution to occur. The computational specification decomposes the system into computational objects performing individual functions and interacting at interfaces. It thus provides the basis for decisions on how to distribute the jobs to be done, because objects can be located independently assuming communications mechanisms can be defined in the engineering specification to support the behaviour at the interfaces to those objects.


The heart of the computational language is the computational object model, which constrains the computational specification by defining:

  • the form of interface an object can have;
  • the way that interfaces can be bound and the forms of interaction that can take place at them;
  • the actions an object can perform, in particular the creation of new objects and interfaces, and the establishment of bindings.

The computational object model provides the basis for ensuring consistency between different engineering and technology specifications (including programming languages and communication mechanisms) since they must be consistent with the same computational object model. This consistency allows open interworking and portability of components in the resulting implementation.

The computational language enables the specifier to model constraints on the distribution of an application (in terms of environment contracts associated with individual interfaces and interface bindings of computational objects) without specifying the actual degree of distribution in the computational specification; this latter is specified in the engineering and technology specifications. This ensures that the computational specification of an application is not based on any unstated assumptions affecting the distribution of engineering and technology objects. Because of this, the configuration and degree of distribution of the hardware on which ODP applications are run can easily be altered, subject to the stated environment constraints, without having a major impact on the application software.

http://www.rm-odp.net/



Computational objects in object-oriented data models, Authors: Jianhua Zhu, David Maier


Computational object recognition: a biologically motivated approach, Authors: Tim C. Kietzmann, Sascha Lange, Martin Riedmiller

Heekyoung Jung, Youngsuk L. Altieri, and Jeffrey Bardzell. 2010. Computational objects and expressive forms: a design exploration. In Proceedings of the 28th of the international conference extended abstracts on Human factors in computing systems (CHI EA '10). ACM, New York, NY, USA, 3433-3438. DOI=10.1145/1753846.1753997 http://doi.acm.org/10.1145/1753846.1753997


Form and the computational object, by Ramia Mazé, Johan Redström




Temporal Computational Objects: A Process for Dynamic Surface Generation, Authors: Kurt W. Swanson, Kenneth A. Brakke and David E. Breen


Simulating an Economy with Smithian Production: Division of Labor as a Computational Object
By Roger A. McCain, http://faculty.lebow.drexel.edu/McCainR/top/eco/wps/RAMcCEastern.pdf


The Distributed Interactive Object Substrate (DIOS) forms the back-end of DISCOVER and is composed of two major components: (1) Interaction Objects that are co-located with computational objects and encapsulate sensors and actuators, and (2) A hierarchical control network that connects these objects with different interaction agents. http://nsfcac.rutgers.edu/TASSL/Projects/DISCOVER/DIOS/DIOS.html





Thursday, May 31, 2012

Information is food

Check out this amazing TED Talk:

JP Rangaswami: Information is food
http://www.ted.com/talks/jp_rangaswami_information_is_food.html

Monday, May 28, 2012

News: May 2012

I am taking advantage of the holiday to reduce the backlog of reading I have piled on my desk. In deference to the holiday, I am favoring the ligher fare including my backlog of New Yorkers. Today's post will feature the highlights of this reading.

InformationWeek 2012 Salary Survey
As if to acknowledge my interest in software engineering culture, this survey not only provides economic documentation of the market for software engineering and management skills, it also provides more qualitative information. Given the survey methodology I don't take this too seriously but I will not dismiss it entirely.

As I've read elsewhere, there is a continued appreciation for the role of people with deep analytic, especially statistical skills in the marketplace. This seems to be driven by the enormous growth in the amount of data out there (big data, deep analytics, business intelligence, data integration, data warehousing) and the continual movement of media to the internet where new models of advertising continue to draw attention. Facebook's need to monitize the phone app is only one such push. This is in addition to the ever growing need in financial services and in particular capital markets where trading models continue to move toward highly complex math based technques.

While this sector has done well, there is evidence in this survey for what many have seen and talked about anecdotally; the erosion of the mainline job of programmer/programmer-analyst in favor of outsourced and off-shored models. The smart people appear to be leveraging their knowledge of the data to take these more analytic jobs in favor of the development jobs they held.

There are several comments about Chevron retraining people in place. Notable since they are a local employer of grads.

The survey cites general feelings of security in the market with employees feeling that they are in no danger of losing their jobs. But after salary, company security is the next most valued attribute of their job. This salary also calls out that staffers do not depend on their company to maintain their skills. More than a third of all staff receive no training and 15% paid their own way for training. Only 25% of respondents receive tuition reimbursement. Only 10% of staffers think college-level technology or business courses would "be useful" to their careers. Technology specific training is sought by 76% of staffers. Despite the market signals, only 6% of staffers see statistics or analytics training as "most useful" for their careers.

One slide that caught my attention was the reasons for working as a contractor/consultant. For staff, the answers ranked: higher pay (45%), couldn't find full-time IT job (32%), flexible hours (30%), variety of the work (27%), broaden experience/skills (22%), other (10%). Two responses were allowed. These sum to 166% so some only provided one response. N=333. The first two responses are purely economic in my opinion. The next three speak to an engineering mindset. Hours and variety were more highly prized by the management respondants than the staff (41% and 42% respectively) which I find surprising. But staff cited broadened skills and experience two to one over their managers as was true in the category of other.

There is a slide on training valued (What type of training would you fin dmost valuable to you in developing your career?, figure 43) The categories where staff had a higher demand than management was technology specific training (76/54), certification courses (43/26), and college courses in tech or business (10/7).

Fig 46: What Matters Most to Staffers lists mostly economic issues as you might expect (base pay, job/company stability, benefits, vacation/time off, commute distance, geographic location) but what is surprising is how highly rated many of the non-economic issues are. For example 40% checked "my opinion and knowledge are valued" versus 19% for "potential for promotion". This can be interpreted to mean that the average staffer is twice as motivated by an environment where they are asked to share their opinion and feel their knowledge is valued than the yearn for a promotion. Other non-economic issues to staffers include:
* "challenge of job/responsibility" - 39%
* "job atmosphere" - 36%
* "having the tools and support to do my job well" - 30%
* "working with highly talented peers" - 25%
* "corporate culture and values" - 19%
(N=7499, respondents allowed to select up to seven)

Barely a third of the staffers felt challenged intellectually with the IT projects they are on (35%)

Of the respondents, 2895 reported they were looking for a new job. (fig 73) Of them 70% cited compensation but 50% cited "more interesting work" and 43% cited "seeking more personal fulfillment and 41% cited "don't like present company's management/culture. (multiple responses were allowed. I think this it is telling that the 2nd, 3rd, and 4th reason cited were non-economic. When you consider that few people are likely to not check more compensation when they are changing jobs, I think they suggests that IT staffers are not overly concerned with the economic incentives they receive. When this is taken together with the next figure (fig 74, reasons to accept a lower position), I think the picture is even clearer. The number one reason given was "more job satisfaction" at 55% with location second at 40%.

87%male

23% work for a company that employs more than 20,000 workers.


ACM SigSoft Software Engineering Notes, Mar 2012 vol 37,no 2
The secret life of academic papers by Robert Schaefer
He talks about requests he's received to convert his qualitative analysis "int a more empirical form" and discusses his concern about reducing "complex issues to numbers". He says he has been told by his students that this is done all the time and referred to as Q-Methodology.

Surfing the Net for Software Engineering Notes by Mark Doernhoefer at MITRE corp
He gives two links to help in learning Google's MapReduce
http://research.google.com/archive/mapreduce.html
http://code.google.com/edu/submissions/mapreduce/listing.html

Also a brief mention of Hadoop http://hadoop.apache.org, Amazon Elastic MapReduce http://aws.amazon.com/elasticmapreduce.

He cites a project called Common Crawl (http://commoncrawl.org/) which sounds way cool. If I read this page correctly, you can use their Hadoop servers and extract information for free. I gotta try that!

Risks to the Public by Peter G. Neumann and contributors
I am so tired of rereading about the Thorak software glitch. This offers new ones. Of note was a failure of the Berlin Train System on 15 Dec 2011. There was a single point of failure which lost all power. The system was completely shut down and it took 3 hours after power was restored before a any traffic could start running again. Qantas experienced two sudden altitude drops of planes in flight. At least one was attributed to a failed airspeed sensor. My favorite bit is this:
"[There's] an (unverified) assertion that the Airbus flight control system will exercise uncommanded changes to throttle settings without moving the throttle handles in the cockpit. If true: bad robot, bad, bad robot. (The Boeing system supposedly has actuators on the handles and moves them when it decides to take over throttle control.)]"

Also reported, a programming error led to the loss of a Russian Mars probe.

The Internet Hysteria Index

Tuesday, May 22, 2012

Software Engineering Culture

http://www.quora.com/What-makes-a-good-engineering-culture#ans1220471

This link leads to a post on Quora that discusses some thoughts software engineers have shared about the culture at their previous employers. Since this is one of my favorite topics, I can't let it pass without some thought and comment.

One of the common complaints is bureaucratic obstacles to creation. Having experienced this on both sides of the engineering/management divide, I feel their pain. Engineers take great pride in their creations. Given the difficulty of bringing a large, complex project to a successful conclusion, they have every right to be proud of the accomplishment. Nothing is more frustrating than to pour your effort into a project for many months or years only to have it get cancelled when the promise of a successful deployment is within sight. Even worse, there is rarely work immediately available to fill the hole left by the project's cancellation.

While a tad crude, it is not completely inappropriate to consider this a form of coitus interuptus. For an engineer, the job well done is a powerful reward and to withhold that accomplishment is a form of cruel and unusual punishment. While the situation has gotten better through the years, too many companies still do not understand the IT function or the role of software engineers in this function. The churn of CIOs illustrates how difficult it is to be at the intersection of management and technology. Most senior managers today seem to be either incapable or unwilling to make even a two year plan that they can stick to. These changing priorities often lead to the cancellation of projects that have already consumed a significant portion of their project budget.

Another source of failed projects is bad project management. In the world of bespoke software, the challenge to create something completely new is constant. But the more novel, the more risk. Managing that risk while maintaining hubris is a recipe for disaster.

No matter the reason for the cancellation, to the engineer it is all the same. This seems to go to a core value in an engineering culture; you don't work for the money, you work for the intrinsic rewards of the job. I cannot think of a better contrast to this value than that of a sales and marketing professional.


SALES v. ENGINEERING: the eternal struggle
I have grown to think of salespeople as the hunter-gatherers of the corporate world. They live for the kill. Too many times the relationship between sales and engineering is along orthogonal lines. To the salesperson, engineering is there to create the thing to be sold. When that thing is commoditized, there is a comfortable insulation between the two functions. But in the world of software, and especially in the world of bespoke software, the two are close or even combined as you can find in a consulting organization. When these two functions are separate yet forced to coordinate very closely, there is friction.

For operational efficiency, a software intensive product company will attempt to standardize and minimize their internal product line to the extent possible so as to achieve economies of scale. While this internal product line may be expressed in the marketplace as a large array of products, software allows for variation that is needed in the market while minimizing the internal complexity. This is the essence of the product line approach in software architecture.

In contrast the customer often wants a customized product that can rarely be realized from the rationalized product line that exists. Since sales people always want to say yes to the customer, they will accept orders knowing that to satisfy their customer they must ask engineering to make changes. When we are talking about the largest and most complex software intensive products, this was a common occurrence. (Since I've been out of the game for a few years, I can't be certain how prevalent it still is but I'd be very surprised if it still is not an issue for large software vendors of business solutions.) This creates a tension between sales who want to sell a product defined by the customers wants and needs, and engineering and operations who want to sell a standardized and easily maintained product line. Pre-sales support moderates the worst of this, assuming management doesn't, but like the sales people, they are not responsible for delivering on the promise, only the achievement of a signed contract. Pre-sales engineering often participate in some sales compensation commissions, although not as much as the closer, and hence are as motivated to make the sale as the closer. After the order is taken, it is the responsibility of the organization to make the customer happy by delivering the product that was promised.

It is normal for management to focus on its revenue stream. No matter how good the product, a large, complex product must be promoted, explained, negotiated and delivered. For large customizable products, this is no small feat and I have the greatest admiration for those skills. And of course the success of the company depends on those skills. But there must be a balance to the motivation to sell a customized product with no regard for the ability of the organization to deliver on that promise. This leads to the second frustration that most engineers seem to face, the demand for change to the product in an unanticipated way. Sometimes this change even undermines some of the design principles that guided its creation. An engineer will see a hack to a product to make a sale as an affront on the purity of the design, or worse, a huge risk to the reputation they have invested in the product. Since any hack is likely to be required under and unrealistic timetable (from the engineering perspective), there is no upside to the engineer but a significant downside. It is a draw/lose proposition to the engineer since any reward for their heroics will be transitory while any punishment for failure is likely to be severe. Is there any wonder engineers would always prefer to never see a sales person in their area?


SPEED
It does not take too much experience to understand the intoxicating need for speed in the valleys of innovation. From the business perspective, the need to be the first in a new market and maintain a lead is vital to success. I doubt that anyone completely dismisses the association between youth and new skills or the relationship between youth and enthusiasm. Rightly or wrongly, youth is also associated with risk taking. When you put these all together you create a culture that is wired for speed.

One of the Quora comments concerns the excitement and motivation engendered by rapid development cycles. Too many organizations still want to manage risk by using management gates which inherently drive toward a waterfall methodology. The comments clearly show an incompatibility between the management perspective on managing risk and the engineer perspective of reducing or eliminating bureaucratic (to the engineer) obstacles. The first theme raised was empowerment.

Management, especially a command and control structure, values predictability and conformance to established rules. This invariably ascribes a level of management a level of control, their formal authority, to make decisions on behalf of the entire organization. Upper- and mid-level managers must delegate some of this authority to lower levels. Since lower level managers are at the bottom of the hierarchy, are generally the less experienced or less competent managers, their ego often is embodied in their authority. If they are ambitious, they are concerned with pleasing their supervisor and doing an exemplary job so as to better their chances for promotion. For many reasons, they are disinclined to delegate the limited authority they have to line workers. This hesitancy can chafe in an engineering culture which will value rapid management decision making and a resentment of waiting for decisions that appear obvious. It is not uncommon for the delay in a management decision to cause a delay in the critical path of the project and management will both complain about the project delay while deny that they are the cause of the delay. Of course inexperienced project management that does not anticipate some predictable decision points does not help. This is both an example of what is perceived as bureaucratic obstacles and a stop/go or lethargic pace to the project.

Another theme I see in the complaints about speed are the lack of understanding of the risk management is attempting to mitigate. Take for instance the issue of committing changes to production in Google when those changes could potentially lead to bad customer experiences. Anyone in the industry knows how hard it is to build customer goodwill and how quickly it can evaporate. Consider NetFlix. Google is wise to carefully guard this intangible asset. Yet the caution that their management displays is unmotivating for an engineer, especially a young, talented engineer, who is anxious to see the results of their labor. Immaturity and a less risk-averse attitude must surely be a common theme among young, talented software engineers.

One of the many speeches I've heard was a cautionary tale I heard when being prepared for work in the field as a mainline consultant. Remember, I was told, that the people you are working with do not have the same career path you do. As a young consultant, we had reason to believe that we would see a stratospheric soar in our salaries and job prospects due to our work at THE FIRM. Meanwhile the staffs we were working with expected a 9 to 5 job and had made different choices regarding their employment. The obvious message was that we were young, ambitious and typically had no life distracting us from the martial pitch to get the job done. To expect the same motivation of the staffs we worked with was just unrealistic. I'll skip over the smugness and condescension. While the situation is different, many of the same social factors are at work among the software engineers at startups.

I did a stint at Siebel. While I was employee 1500 something and the company had been in business 4 years with most of them profitable, Tom Siebel had the managers referring to it as a start-up and expected  everyone to act like it was. It was 1999, after all, and everyone wanted to be at one. Yes, we got stock options and had all the trappings of a mature start-up. But in my opinion it wasn't and I was very dubious that the stock options would ever be worth much. As it turns out I was mostly right, although I did end up making some from the options before they became completely worthless and it made up for the lower than market wage they paid. But it gave me a taste of what I knew must be going on all over the valley. Young, naive but talented software engineers were employed by very savvy, talented guys that had an appreciation for what the technology could deliver. When they were ambitious they found ways of selling this vision to the employees and were willing to allow them to share in the expected rewards with generous stock options. This model has been working well for over a decade but I think it is far more nuanced than the media portrays.

Despite all the variations in the valley success stories, the one theme they all share is the infectious excitement of making something new and different. This type of greenfield development seems to be the ambition of every software engineer I've known. The usual preclude is maintenance programming and is something that most software engineers seek to leave behind them as soon as possible.This eroticism of the new is a theme I hope to come back to someday when i discuss design. But since this is not found in the posting I'll skip over it. However I believe that this desire to do greenfield development is inherent and helps illustrate the excitement.

When developing a product that does not yet have customers and no existing structure to restrict development, it is all promise. Further, most greenfield development, especially large, mission critical systems, enjoy a great deal of management attention. There are a great many positive incentives, fewer obstacles and a shared vision of the future. Of course the later phases are difficult as these realities begin to intrude but the human assets accumulated during the development can be used to see the project to a successful end. Such projects are often feted and people can take great pride in a job well done, or at least done. The group dynamics of the project invariably lead to rapid development.

The comments stress speed specifically in the form of rapid iterations. It isn't completely clear how rapid iterations contribute to engineer motivation so I'll parse out the comments. Iterations stand in opposition to a strict waterfall methodology. Most waterfall methodologies encourage a focus on the phase transition points and then load them with various quality control points. I doubt that each iteration has all the QA points that a waterfall plan would have and even with some QA between iterations, I doubt the total time devoted is greater in an iterative approach and I suspect a great deal less. This alone may be an explanation for an engineer's preference of iterative over waterfall.

The last point I saw in the post concerned continuous deployment. He contends that at Quora the commitment to get changes into production quickly has many positive effects. He wasn't completely clear as to what their continuous deployment infrastructure includes but presumably it includes some automated tests to ensure any change does not completely break the system. In addition, if there is any form of Test Driven Design, the testing includes some demonstration of the new feature/fixed feature that was the subject of the change.

Why would this be something an engineer would enjoy? Speculating, I would say that knowing a layer of control that is meant to shield me from my errors increases the pressure to be right. Nothing can be more humiliating than to commit a change to a code base that breaks the product. You get immediate attention and none of it is the kind you want. You might make this mistake once in your career but you are not allowed to commit the same mistake too many times and remain on a viable career path. Of course this also increases the intimacy between the effort of a software engineer and their contribution to the organization.


AUTOMATION
This poster suggests that attention to the automation of operational tasks is a key to a high-quality culture that will attract engineers. I'm disappointed he didn't connect the dots for me but I'll speculate on what he has in mind.

As I mentioned, maintenance tasks are generally not fun. The more maintenance, the less fun. People want fun. Minimize the maintenance and you increase your fun to drudge factor. Perhaps it is this simple. But I see more.

One of the standard gripes I heard on projects came from the people who would support the system after it was put into production. Generally the first briefing they had of the design was when the planning for transfer began. The project scope and the scope of design rarely included the after-turnover support. It was not uncommon for a new system to increase the burden on the operational staff, require new skills, be incompatible with other systems or any of a number of other bone-headed moves that some forethought could have avoided. Meanwhile the best projects I saw included at least some consideration of the operational environment in the planning, if not the design. Clearly the best projects will do this.

It's too frequently remarked that maintenance is the most expensive phase of a product's lifecycle. I suspect there are a great many hidden costs in operation and support as well. Where this poster gives me hope is that it sounds like there is a greater emphasis on post-implementation support, at least among the engineers. From a business perspective, this will lead to far greater efficiency of operation and contribute to the overall product profitability over the life of the project. In the end, I see this as a move to consider not only the design of the software, or even the software/hardware system, but of the complete human/machine system and the complete product lifecycle.

There was one reference that I found intriguing. http://codeascraft.etsy.com/2011/02/15/measure-anything-measure-everything/ This link expounds on the virtues of a metric driven approach to operation. I have to believe that if I dig into this meme I will find Basili. He goes on to say:
"...open-source monitoring and charting tools like graphite [5] and statsd [6] highlight an important aspect of automation -- that automation must be driven by data and monitoring. Without monitoring and logs to know what, how, or why something is going wrong, automation is difficult. A good follow-up motto would be to "measure anything, measure everything, and automate as much as possible."
A key point, I think, about the dotCom revolution and it's allied OSS, is the acceptance of putting out products that are likely to contain bugs. This is a stark break with prior practice in the commercial world. dotComs can get away with it because many of their products are free and in the formative stage and are often not mission critical.


DESIGN (build the right software abstractions)
Again, the poster neglected to tie this back to the main point which is how to build a winning software engineering culture. This is a very important topic to me in ways that go beyond its influence on the culture. However I can riff on this although I may not have the same opinion as the poster.

Who doesn't want to play with winners? One of the most ubiquitous cubicle decorations I've seen is the one that reads "It's hard to soar with eagles when you're surrounded by turkeys." This clearly communicates the idea that greatness is within reach but the individuals around us prevent it. It is never clear who the turkeys are (the co-workers? the bosses?) but the thought is clearly that outside forces hold back personal achievement.

This attitude suggests a cynical attitude and a discounting of any personal contribution to the obstacles ("If you are not part of the solution, you're part of the problem") and even worse, a defeatist attitude toward greatness. A good project manager or line manager should know how to counteract this ennui. But to the point, a highly motivated group with high performing individuals is a rewarding and fulfilling group to be in. How much is the peer relations and how much is the management which has created and maintains this group?

To what may have been the writer's main point however, what part does a good design play in the software engineer's motivation? Is it the fact that a good design suggests talented people? Is it because an engineer appreciates the beauty of the design and is rewarded by getting to know it? or learns new skills because of it? Is it because a good design makes a software engineer's job easier by anticipating the build-out of that design and does not get in the way? After all, nothing is more frustrating than being given a design that is immediately seen to be unworkable in the current context. While not every software engineer may have the skills to be a gifted designer, they certainly learn to recognize good ones. Or perhaps it is the relationship between the designer and the construction crew. Rather than having such a bifurcation of staff responsibilities, perhaps it is because the designers are themselves well respected coders who will remain intimately involved in the design through its implementation.

For me, the gold in this part of the post is its references:


These are all references I'll want to use in a post on software architecture.

STRIVE FOR GREATNESS (Develop a focus on high code quality with code reviews)
This section makes many excellent points but again does not focus on the contributions to culture I am trying to develop. Here's my riff.

Pride in workmanship is part of the culture. A staff that truly buys into code reviews is one that exhibits this quality. For code reviews to work properly, several social phenomenon must exist. First, people's egos are strong enough that they can tolerate the review, and receive recommendations, on their products. This speaks to a certain degree of maturity in the individuals and a level of comfort that has been achieved  among peers. When either of these things is lacking, dysfunctions are eventually exposed.

There are a number of quotes that come to mind when I consider this topic. Daniel Burnham, who created a master plan for the city of Chicago, is alleged to have said:
Make no little plans. They have no magic to stir men's blood and probably themselves will not be realized. Make big plans. Aim high in hope and work. Remembering that a noble, logical diagram once recorded will not die.
If he said it, I suspect it was a paraphrase of a similar quote from Machiavelli:
Make no small plans for they have no power to stir the soul.
Personally I prefer Machiavelli's pithier version although it lacks the point that a good design, once created, can live on independently. But both stress the importance of grand ideas to succeed in the leadership of large, complex projects.

I don't know that I can separate culture from leadership. A leader attracts talent to a project and makes an investment to keep everyone motivated and invested in the project. It seems self-evident that the people will contain some common characteristics even if they are nothing more than some momentary suggestion in the mind of the leader that they will make a positive contribution to the project and that they at least accept the leader and want to participate in the project.

An important aspect of project management is to ensure that everyone maintains a common vision of the goal to be reached. A common problem is miscommunication and the tendency of people to hear not what was said, but what they want to hear. Since a software-intensive project must deal with layers of abstraction and vague ideas that must be reified before any product can be realized, there is enormous room for the internal concepts of what will be delivered to wildly diverge from each other. A well managed project will reconcile these divergent visions at or before the last possible moment before which they lead to inefficiency or error.

Another problem that can infect a software engineering culture is the tendency to allow standards to slip. In my opinion, this is failure of management. In one instance, the engineers bring a decision to management that will concern some trade-off between quality and some other economic metric such as time or money. Management may be forced to accept the quality trade-off but how this is communicated to the staff can dramatically affect the culture. On the one hand, if the management accepts that they have incurred a technical debt that must be paid in the future, the engineering staff will see that management is committed to delivering a quality product but must accept certain commercial imperatives. But if managers begin to fudge, rationalize, or deny that de facto decisions that are made, the message to the engineers is that management is more concerned about reputation, money or something else more than they are in the pride they take in achieving the best possible product. Of course this issue is far more nuanced than I let on but it is sufficient to drop a marker on the interplay between decision making and how that can affect an engineering staff.

One management aspect that the poster and I seem to believe is a major contribution to a healthy and desirable engineering culture is the support of code reviews. I see this as a management contribution simply because without their support, it is impossible to create a viable code review process. But implementing this process still requires some courage and faith on the part of management since it has very measurable cost and an unmeasurable benefit. I have seen a few words written as to why people believe code reviews work and I will not take the time right now to research their opinions. Instead I will provide my own knowing full well that most, if not all of them, are the work of others and not original to me.

Socially the most important aspect of a code review process is that it forces engineers to review and discuss their work products. It is perhaps an unavoidable truth about humans that we are always more critical of the work of others than we are of our own. It is also common that people will be sensitive about comments about their work. Yet if these barriers are overcome, debate and an atmosphere of continuous learning begins to happen. Tiny decisions are made by these review meetings and conflicting points of view are aired. Over time, some consensus arises and the I contend that the work of that staff will begin to converge on a common style and uniformity more effectively than any style manual could hope to achieve. This is particularly important where staff members must back each other up to support production. In my case, the list of people who participated in the code review became a matter of record and any member of the review team could be called upon to deal with a problem in that module when the primary was not available. This potential gave an urgency to the work.

The poster suggests that non FTF code reviews can be done and he is absolute right. However I do not believe that they have the same contribution to the staff culture that FTF meetings have. To be in the same room as someone brings a hightened tension to the need to make critical observations or even question someone about their work product. While the improvement to the work product could perhaps be done via computer mediated communication, it does not build the same social skills and interpersonal relationships that FTF meetings have. If you want a cohesive staff, FTF code reviews is one sure way to build strong morale as well as encourage a high-quality, high-performing environment. In the social science literature, this would be due to group norming.


R-E-S-P-E-C-T (Maintain a respectful work environment)
I hope I made the point in the above section on FTF code reviews that they become a vital tool for allowing groups to overcome the obstacles that exist to their effective team effort. Lacking some serious dysfunction, group will naturally move toward some behavioral norms regarding new people or new revelations about people. But the focus on the work product and the mission of the organization often cause people to reevaluate prejudices and try harder to engage in real interpersonal communication.

The poster uses this heading to also discuss how groups can contribute synergy to a design task, a conclusion I endorse. I suppose the cultural issue here is that if engineers are either not asked for their opinions when they should be, or are not listened to when they are, it is understandable why the engineering culture of that organization may flag.


CODE OWNERSHIP (Build shared ownership of code)
The natural tendency for an engineering organization is to divide responsibility for the code along module, sub-system, system boundaries. While logical in that assembly line way of thinking about specialization, it fails to address the stifling aspect of the job. As I've noted elsewhere, software engineers, especially highly talented and young software engineers, are receiving a great deal of intrinsic reward from their job. Asking them to focus on one piece of code to exclusion of the remainder of the system removes the excitement of newness from the job and leads to the inevitable growth of job dissatisfaction. Engineering organizations should take a page from management training programs that move managers frequently so as to give them the broadest possible exposure to the organization. While the organization may lose some instant recall of the internal workings of the code due to the lack of specialization, over time the engineers should become more capable of recognizing the potential causes of systemic and intermittent errors that cross module boundaries.

At a young start-up, compensating different teams differently will lead to strong boundaries between the teams. The greater the distance from one team to another the greater the distance between the code modules. (Conway's law). If Agile methods are ever to reach their maximum effectiveness, people from different areas must be capable of leaving their functional groups and joining together into a cohesive team. The broader the social comfort each team member has with the broader organization, the faster the norming of the group and the sooner they can begin performing.

The poster references the term "bus number" for a project. The reference was another Quora post and I haven't taken the time to research the origin of this phrase. However it is a reference to the number of people in a project who could have a calamitous event (like being hit with a bus) without impacting the team mission. Too often this is 1. For non-mission critical projects, this may make economic sense however it is worth considering how helpful it is to adopt a journeyman/apprentice approach or even a pair-programming approach.


INVEST IN AUTOMATED TESTING
He's right. However I think the poster stops short here. Software engineering is one of the least capital intensive industries in the US today. (I need to find evidence for this but I did once come across a chart showing capital per worker and I remember it being near the bottom). It would be an interesting empirical study to demonstrate how much this means to an organization over the amortization period of the investment in the testing software. The usual obstacle is that most development is done with project level accounting and the project manager can never demonstrate the payback on the investment. Project level accounting for testing software is just boneheaded. It should be at the enterprise level and few senior level executives are convinced that expensive test suites are worth the money. What does he think? That his staff is too lazy to test the products? That people enjoy the often tedious work of testing? This should be a part of every organization's quality program.

The products delivered now are larger and more complex than their predecessors. The skills needed to test them are more complex both because of the complexity of the structures but also because of the greater subject domain they encompass. To deny professionals the tools they need to do their best work just doesn't make any sense. It is also not in keeping with several other goals for a high-quality engineering culture; rapid deployment and Agile processes.

The current practice is to release a product to testing and use basic system's analysis to ensure that it meets its specifications. Once the process is complete it is released to production. But more recent practices are calling for smaller releases that do not lend themselves to the former waterfall approach of holding back changes until a release. The releases are smaller and more targeted. It is difficult to perform the best possible recursion test in this environment without some automated help. In fact, most organizations do not perform complete recursion testing on releases because of the cost of testing it with people.

I think one of the most compelling reasons for an automated test suite is its support for test driven design. While I am not necessarily a proponent, I see many benefits in this approach. First and foremost, it provides an actionable definition of what is expected in the final product. As soon as a feature or error is found, the test to demonstrate that feature is added to the test suite. If full recursion testing is done, all future releases are assured of passing that test or knowing that the release fails that test.

More broadly, the same obstacles that test suites face with enterprise management are faced by many software engineering tools and training. Ironically, the very biggest enterprise purchases such as a database and customer relationship management have a much easier time in the executive suite. This is due to the far greater cost, the smaller number of decisions asked of the executives and the power of the sales and marketing organizations behind these products. Will it be necessary for software engineering tools to coalesce into a mega-application before they can get this kind of attention?


20% Time
This is one section I will quote rather than paraphrase. Since he speaks from experience and provides good citations, it needs no further commentary:
Gmail found its roots in Paul Buchheit's 20% project, and he hacked together the first version in a single day. [http://paulbuchheit.blogspot.com/2009/01/communicating-with-code.html ] Google News, Google Transit, and Google Suggest also started and launched as 20% projects. I used 20% time while at Google to write a python framework that made it significantly easier to build search page demos.  While Google's 20% time may be less productive now than during the early days of the company [http://www.quora.com/Engineering-in-Silicon-Valley/How-does-Googles-Innovation-Time-Off-20-time-work-in-practice ], the notion of letting engineers spend 20% of their time working on something not on their product map remains a cradle of innovation for smaller engineering organizations.

Ooyala didn't officially have 20% time while I was there, but I took some anyway and wrote a command-line build tool for Flex and Actionscript that sped up the team's build times, just as Adobe's Flex Builder tool chain started to degrade, and the tool's still in use today even though the engineering team has nearly tripled in size. Atlassian adopted 20% time after experimenting it for year. [http://www.atlassian.com/company/careers/life ] A variation of 20% time that Facebook's fond of and that Ooyala added later is periodic hackathons -- all-night events where the rule is that you can work on anything except your normal project. [http://gigaom.com/2011/12/16/exclusive-inside-facebooks-final-palo-alto-hackathon/ ]

Top-down approaches to product planning, while necessary for focusing the overall direction of the company, can't account to for the multitude of ideas that might arise from engineers closer to the ground. As long as engineers are accountable for their 20% time and focus on what can be high-impact changes, these projects can lead to large steps forward in progress. Without official 20% time, it's still possible but much more difficult for engineers and designers to try out crazy ideas -- the dedicated ones basically have to find weekends or vacation days to do it.
BUILD A CULTURE OF LEARNING AND CONTINUOUS IMPROVEMENT
The poster makes three valid points. First, software engineers, like everyone, find satisfaction when they achieve what Mihaly Csikszentmihalyi calls a state of "flow", second, weekly tech talks, third, a good mentoring program. Let me take these in order.

I agree that flow is a vital to job satisfaction although I do not see how it relates to culture so much as software engineeer psychology. I can see that when a software engineer is comfortable in her environment that some obstacles to flow have been removed. However I don't see how the culture, per se, can make this happen.

The recommendation for tech talks is one I can support from my own experience. Let's explore the cause and effect. Why would a technical meeting lead to improved culture? It shows management support and recognition for the software engineering profession. It takes time away from other tasks (in the eyes of management) and thus represents a perceived cost to the organization. Whether it ultimately does or not would be a matter of empirical research. But the perception from the engineer is likely to be that of encouragement of new skill development, and a connection to the broader profession. Too many organizations that do not have software engineering as their prime revenue generator do not view software engineers as professionals. This denigration of their skills and fear of the business function leads to an isolation of this group from the broader organizational goals and alienation from the other staff. This alienation is often absent when the software products are the chief output of the organization. Here the software engineers are more valued for their expertise although this is not universal. Software consulting organizations are perhaps the best at both recognizing the value of the software engineers and taking an active interest in their continued growth.

I have seen many mentoring programs and I do not believe they work. I say this with great sadness since I think the relationships can make an enormous difference to the individuals and the organization. Any professional who is a minority in an organization will feel some degree of alienation from the other staff. The job function requires considerable education and training, it is introspective by nature and forces a form of interpersonal communication that is unnatural to the non-professional. Having a mentor is one way to overcome alienation since the humanity of that relationship overcomes the inherently unsatisfying relationship to the organization as a whole. (I had one friend who once remarked that all organizations are inherently dysfunctional maternal objects).

So why do I think mentoring programs don't work? At its core is the growth of an interpersonal relationship between two people, one with more power than the other. This power difference creates interpersonal distance that must be overcome before the benefits of mentoring are seen. This cannot be achieved when the relationship is forced on two people and cannot happen when there is distrust on either side. It cannot be someone in the management chain to which this person reports which creates many obstacles for smaller companies. This represents too many barriers and consequently is rarely effective as a program. What companies should do instead is create opportunities for different levels to mix as equals and to work on creating environments where people are comfortable speaking candidly. It is a rare company that does even this.


HIRE THE BEST
This sound completely reasonable but I do not think this maxim holds up to critical analysis. Does this mean that all these high-performing silicon valley organization all employ the best engineers? If so, how many "best" engineers are there? Are we conflating best with competent? There is no doubt that there is a great difference between engineering talent at any level (Sackman's Second Law, [Enders03] ). I would suggest that "the best" isn't much more than a slogan or a bit of organizational jingoism.

Clearly no one should be given responsibility to create a product that is beyond their means. If the new hire has so little experience or talent so as to not be a contributing member of the team, they should not be given the position. I think it is denigrating to dismiss them as not being the best, something lacking, instead of recognizing that they do not fit the requirements. There are too many skill sets in the marketplace now. Some very talented programmers may have a difficult time coming up to speed on a set of technologies that are unfamiliar to them. Does that mean they are not "the best" or just not the best at that set of tasks?

If technical expertise were all that was required to be a productive member of a team a strict meritocracy would evolve. But there is more to being a highly skilled software engineer than being technically proficient in whatever technologies are needed. The ability to both adhere to group norms and to provide leadership is one pair of traits that can be in conflict. Interpersonal skills are vital to being able to navigate the sometimes difficult terrain of corporate social structures. If you doubt this, just imagine someone with no English skills trying to work as a software engineer in a high-performing organization. Human communication plays a much bigger role in competence than is sometimes recognized.

As to the contribution that hiring practices can have, I will share one anecdote. As a manager of a testing group I was saddled with an incompetent employee who had been hired because of political connections and shuttled around until he found an area where he was unlikely to do much harm. It was difficult to get him fired and it took almost a year but in the end I feared that his departure would impact the group morale since he was well liked. I was wrong. Almost immediately after his departure, the morale significantly improved and productivity went up. Conversely, when I hired someone very competent shortly before his departure, there was also a notable uptick in morale and productivity. The conclusion I came to was that the self perception of the staff was helped by the positive improvements and recognition of both competence and incompetence. Incompetent workers not only create technical debt, they also weigh down a group.


THE RESEARCH PERSPECTIVE
This post would be pointless beyond my self-indulgent rant about the problems in the industry if I did not bring it to closure with an improved focus on my research agenda. My first filter for looking at this material is the question of what open source software is bringing us, how it is doing that and some way of looking at its future. There is ample suggestion that software engineers are inclined to freely volunteer significant talents to the creation, nurturing and maintenance of open source software products. Why? This post explored some key themes the Quora post suggests are on the minds of talented software engineers in the field. My own experience supports the posters claims. While suggestive, this is hardly incontrovertible empirical evidence and needs more careful collection and analysis.

Putting aside the flimsy evidence for the moment and making a guess as to what will be found, I can see many other research questions. What first comes up in my mind is the motivation of contributors. Do software engineers who volunteer for open software projects find the work intrinsically rewarding? What are the components of this instrinsic reward? Is it the sense of ownership they derive? Is it the participatory management style of the projects? Is it the reward of having a seat at the decision making table? the ability to scratch their own itch and create the tools they want to work with instead of allowing management to chose their tools and present obstacles for the acquisition of tools that software engineers believe will be helpful? Is it an altruistic impulse to give back to a community that gives them so many software products that would otherwise be economically impossible to use? Is it a self-serving mechanism to impress a future employer? a way to get mentored by experienced professionals? Is it a form of idol worship where people are motivated by the star status or charisma of a project leader? Is this just part of a job that employer is paying for? If so, what is the motivation of the employer? Why does IBM support Eclipse? Why does Oracle support Java and MySQL? What will happen to these motivations over time? Is this a sustainable model for development? If the motivations are completely self-serving, will they poison the motivations that non-employees feel about contributing? I have heard it said that OSS engineers are scratching their own itch. To what extent are contributors only working on products that they themselves use or had used in the past? Is participation in infrastructure products, like DBMS, web servers, IDEs, etc, more prevalent than end-user products like word processors, graphics tools or web browsers? How many work hours are contributed to projects per year? What is the skill set of the contributors?


software architecture,

LAWS OF SOFTWARE ENGINEERING
Glass's Law: Requirement deficiences are the prime source of project failures.

Boehm's First Law: Errors are most frequent during the requirements and design activities and
are the more expensive the later they are removed.

Boehm's Second Law: Prototyping (significantly) reduces requirement and design errors, especially
for user interfaces.

Davis Law: The value of a model depends on the view taken, but none is best for all purposes.

Curtis' Law: Good designs require deep application domain knowledge.

Many more laws can be found in what looks like an excellent text on software engineering mentioned by MG.
(found in [Enders03] )




FURTHER TOPICS
The software engineering of vaporware
"the teeth of time", "an encounter with the ghosts of the past" ~Stephen Greenblatt author of The Swerve
What is the best measure for program length

REFERENCES
[Enders03] Handbook of Software and System's Engineering: Empirical Observations, Laws, and Theory by Albert Endres and Dieter Rombach, 2003, Pearson available at http://dev.co.ua/docs/a-handbook-of-software-and-systems-engineering-empirical-observations-laws-and-theories.9780321154200.29795.pdf)



THE POSTER

Edmond Lau, Quora Engineer

I work as an engineer at Quora, and I'm loving it.  Prior to Quora, I led the video analytics and monetization team at Ooyala, worked on query suggestions and UI experiments on the search quality team at Google, and studied at Massachusetts Institute of Technology.

In Defense of Clippy

For those of you who met Clippy, I'm sure you're groaning already. But let me explain. While Clippy was a flawed character I will still defend the basic idea that underlied the design and contend that Clippy is out there waiting for his day. In this defense is a tale of the current state of customer service and an insight into why I am dubious of the grand claims for the future of AI.

While living in SF I shopped at the Safeway at Potrero and 16th. One of the notable characteristics of shopping there, and I suspect at all Safeway stores, is how each and every employee would greet you as you passed them in the aisles. It was obviously a requirement on them and I found it annoying since I loose myself in wandering the aisles looking at products I don't know. Each greeting was an unwelcome interruption to my reverie. One day I needed to find a product that could logically have been shelved in one of three areas. After checking all three I decided that I would take them up on their continual offers to help me find something. I got that deer-in-the-headlights look from the clerk and I thought he would start to cry. But he did get me in touch with someone that eventually figured out that the product was in a fourth category that I would never have thought to check. So while there was enforced chiperness in the interest of improving customer experience, it was not backed up by deep knowledge of the merchandise.

Here in Sacramento there is an old line grocery store with an excellent butcher and a deli counter that beats any single shop I knew in the city. The employees are sometimes chatty and friendly, sometimes hurried and focused on their tasks and sometimes even a bit diffident. The variability of their personalities doesn't bother me a bit. I can imagine that some people are intimidated by some of the employees but I also think some people are too sensitive. I say, get over it. What I love about this store is that every time I have a question I get an answer that is almost encyclopedic about the product in question. What I lose in chipperness I get back in competency. In the end, the only thing I want is for them to pay attention to me when I need it and take care of my need in the most efficient way possible. If I want a friend I'll sign onto Facebook.

Once I recognized this I see it all over my life, especially in tech and telephone support. Too many companies give their operators scripts that they tightly enforce and very little deep training in their products. I suspect that this has something to do with the bad reputation that Indian call centers have earned. Perhaps it is the more hierarchical social structures of their history or just the rapid fire growth of the industry there, but the bone headed questions I've gotten from some operators would fill a volume of comedy. ("Hi, my name is Dale, I'm calling about order xyz.", "Yes sir. I'd be glad to help you. But first can I get your name?") What managements have done is substitute rote scripts for true management, supervision and training.

So enter Clippy. Even the name suggests some perky young thing that is eager to help. It's continual dance in the corner of the screen just screamed, "Please, please, let me help you!" But as soon as you ask it something, if you weren't lucky enough to ask it the two things it knew, you got useless information. People blamed the mode of delivery. But what was lacking was the competence. If Clippy was able to have even a fraction of the success at answering a real question about the products it was embedded into, it would not have become the brunt of jokes.

Consider what Clippy had going for it; it was an observer of what you were doing across products. It knew the context. It had access to databases of information. Now we have much better search products that could potentially make Clippy far more capable of intuiting what advice we might be seeking. It could potentially recognize repetitive behavior and offer real help in making our interaction with the product better. Will it? I suspect any employee at Microsoft who suggests anything that sounds like Clippy is immediately taken out back and shot. Too bad. RIP Clippy. I hope you come back in another life. You were much more endearing than the clerks at Safeway.

What's the best way to read a file in Java?

The title of this post was an innocuous and seemingly straightforward question in StackOverflow that caught my eye. (http://stackoverflow.com/questions/4716503/best-way-to-read-a-text-file)  The poster had noticed that there was more than one way to read an ASCII file and as a noob was curious as to what the "best" way was. There was hardly a consensus in the answer and I believe there is something to be learned in that lack of a rapid consensus.

First some background. I do not profess to be a Java expert. Since my needs in this department are simple I usually mimic what I find in the first text I grab. Since there are different ways in different texts, I'll lay those out first.

From Savitch:
Scanner inputStream = new Scanner (new FileInputStream("stuff.txt"));

From Gaddis:
BufferedReader inputFile = new BufferedReader(new FileReader("stuff.txt"));

So pangea suggests Gaddis' approach as does Knubo for his first suggestion. However Knubo offers an alternative that only uses the FileInputStream rather than encapsulating it into a Scanner object. Jesus Ramos also agrees with the first suggestion. Juan Carlos Kuri Pinto suggests his way is better but it still uses the Gaddis classes.

Peter Lawrey offered a novel approach.

for(String line: FileUtils.readLines("my-text-file"))
    System.out.println(line);


as did Claude


The methods within apache.commons.io.FileUtils may also be very handy, e.g.:
/**
 * Reads the contents of a file line by line to a List
 * of Strings using the default encoding for the VM.
 */
static List readLines(File file)

One poster, jzd, referenced a page at Oracle which attempts to provide an overview of the different methods and their differences. (http://docs.oracle.com/javase/tutorial/essential/io/file.html)

So if I were to reply to this poster with all of this information, what would I say? All these suggestions will result in code that will read an ASCII file. What are the differences? What makes this interesting to me is that the differences are not in the gross functionality of the code but rather the non-functional qualities that the resulting system will have.

Let's take a trivial and obvious example, the try-catch block. If we know that all our data is good and want a quick-and-dirty piece of code, we might drop the coding of this block since we want to get the code written quickly and have enough control over the file that the code isn't needed. Here we are trading off coding speed (time-to-market) with robustness. The first exception will cause the code to halt. In many cases we don't care and choose for the quick-and-dirty.

There are two important qualities that vary in these treatments. First, is the relationship between the operating code (the dynamic structures) and time. The buffering feature will provide for a more efficient operation for a static file that is to be read and processed as it might in a batch system. But presumably there is some price to be paid for this feature in terms of processor and memory load. In a modern system it is likely that this is negligible in most systems but for very large, high-performance systems, this cannot always be taken for granted. The quality trade-off may become an architectural consideration if this file reading is in the critical path of a high-performance system or one embedded with limited hardware performance.

Another quality difference between the coding options is maintainability. As jzd observes, Oracle now offers some classes that provide much faster processing but at the cost of readability. This illustrates how quality choices are often trade-offs between two or more different requirements. I suspect in the end, we first code what we know best in the absence of any requirement forcing consideration of other options.

Normally I need to find more complicated examples to illustrate this concept. As the design gets more complex, the design task becomes more complicated