Thursday, May 31, 2012

Information is food

Check out this amazing TED Talk:

JP Rangaswami: Information is food
http://www.ted.com/talks/jp_rangaswami_information_is_food.html

Monday, May 28, 2012

News: May 2012

I am taking advantage of the holiday to reduce the backlog of reading I have piled on my desk. In deference to the holiday, I am favoring the ligher fare including my backlog of New Yorkers. Today's post will feature the highlights of this reading.

InformationWeek 2012 Salary Survey
As if to acknowledge my interest in software engineering culture, this survey not only provides economic documentation of the market for software engineering and management skills, it also provides more qualitative information. Given the survey methodology I don't take this too seriously but I will not dismiss it entirely.

As I've read elsewhere, there is a continued appreciation for the role of people with deep analytic, especially statistical skills in the marketplace. This seems to be driven by the enormous growth in the amount of data out there (big data, deep analytics, business intelligence, data integration, data warehousing) and the continual movement of media to the internet where new models of advertising continue to draw attention. Facebook's need to monitize the phone app is only one such push. This is in addition to the ever growing need in financial services and in particular capital markets where trading models continue to move toward highly complex math based technques.

While this sector has done well, there is evidence in this survey for what many have seen and talked about anecdotally; the erosion of the mainline job of programmer/programmer-analyst in favor of outsourced and off-shored models. The smart people appear to be leveraging their knowledge of the data to take these more analytic jobs in favor of the development jobs they held.

There are several comments about Chevron retraining people in place. Notable since they are a local employer of grads.

The survey cites general feelings of security in the market with employees feeling that they are in no danger of losing their jobs. But after salary, company security is the next most valued attribute of their job. This salary also calls out that staffers do not depend on their company to maintain their skills. More than a third of all staff receive no training and 15% paid their own way for training. Only 25% of respondents receive tuition reimbursement. Only 10% of staffers think college-level technology or business courses would "be useful" to their careers. Technology specific training is sought by 76% of staffers. Despite the market signals, only 6% of staffers see statistics or analytics training as "most useful" for their careers.

One slide that caught my attention was the reasons for working as a contractor/consultant. For staff, the answers ranked: higher pay (45%), couldn't find full-time IT job (32%), flexible hours (30%), variety of the work (27%), broaden experience/skills (22%), other (10%). Two responses were allowed. These sum to 166% so some only provided one response. N=333. The first two responses are purely economic in my opinion. The next three speak to an engineering mindset. Hours and variety were more highly prized by the management respondants than the staff (41% and 42% respectively) which I find surprising. But staff cited broadened skills and experience two to one over their managers as was true in the category of other.

There is a slide on training valued (What type of training would you fin dmost valuable to you in developing your career?, figure 43) The categories where staff had a higher demand than management was technology specific training (76/54), certification courses (43/26), and college courses in tech or business (10/7).

Fig 46: What Matters Most to Staffers lists mostly economic issues as you might expect (base pay, job/company stability, benefits, vacation/time off, commute distance, geographic location) but what is surprising is how highly rated many of the non-economic issues are. For example 40% checked "my opinion and knowledge are valued" versus 19% for "potential for promotion". This can be interpreted to mean that the average staffer is twice as motivated by an environment where they are asked to share their opinion and feel their knowledge is valued than the yearn for a promotion. Other non-economic issues to staffers include:
* "challenge of job/responsibility" - 39%
* "job atmosphere" - 36%
* "having the tools and support to do my job well" - 30%
* "working with highly talented peers" - 25%
* "corporate culture and values" - 19%
(N=7499, respondents allowed to select up to seven)

Barely a third of the staffers felt challenged intellectually with the IT projects they are on (35%)

Of the respondents, 2895 reported they were looking for a new job. (fig 73) Of them 70% cited compensation but 50% cited "more interesting work" and 43% cited "seeking more personal fulfillment and 41% cited "don't like present company's management/culture. (multiple responses were allowed. I think this it is telling that the 2nd, 3rd, and 4th reason cited were non-economic. When you consider that few people are likely to not check more compensation when they are changing jobs, I think they suggests that IT staffers are not overly concerned with the economic incentives they receive. When this is taken together with the next figure (fig 74, reasons to accept a lower position), I think the picture is even clearer. The number one reason given was "more job satisfaction" at 55% with location second at 40%.

87%male

23% work for a company that employs more than 20,000 workers.


ACM SigSoft Software Engineering Notes, Mar 2012 vol 37,no 2
The secret life of academic papers by Robert Schaefer
He talks about requests he's received to convert his qualitative analysis "int a more empirical form" and discusses his concern about reducing "complex issues to numbers". He says he has been told by his students that this is done all the time and referred to as Q-Methodology.

Surfing the Net for Software Engineering Notes by Mark Doernhoefer at MITRE corp
He gives two links to help in learning Google's MapReduce
http://research.google.com/archive/mapreduce.html
http://code.google.com/edu/submissions/mapreduce/listing.html

Also a brief mention of Hadoop http://hadoop.apache.org, Amazon Elastic MapReduce http://aws.amazon.com/elasticmapreduce.

He cites a project called Common Crawl (http://commoncrawl.org/) which sounds way cool. If I read this page correctly, you can use their Hadoop servers and extract information for free. I gotta try that!

Risks to the Public by Peter G. Neumann and contributors
I am so tired of rereading about the Thorak software glitch. This offers new ones. Of note was a failure of the Berlin Train System on 15 Dec 2011. There was a single point of failure which lost all power. The system was completely shut down and it took 3 hours after power was restored before a any traffic could start running again. Qantas experienced two sudden altitude drops of planes in flight. At least one was attributed to a failed airspeed sensor. My favorite bit is this:
"[There's] an (unverified) assertion that the Airbus flight control system will exercise uncommanded changes to throttle settings without moving the throttle handles in the cockpit. If true: bad robot, bad, bad robot. (The Boeing system supposedly has actuators on the handles and moves them when it decides to take over throttle control.)]"

Also reported, a programming error led to the loss of a Russian Mars probe.

The Internet Hysteria Index

Tuesday, May 22, 2012

Software Engineering Culture

http://www.quora.com/What-makes-a-good-engineering-culture#ans1220471

This link leads to a post on Quora that discusses some thoughts software engineers have shared about the culture at their previous employers. Since this is one of my favorite topics, I can't let it pass without some thought and comment.

One of the common complaints is bureaucratic obstacles to creation. Having experienced this on both sides of the engineering/management divide, I feel their pain. Engineers take great pride in their creations. Given the difficulty of bringing a large, complex project to a successful conclusion, they have every right to be proud of the accomplishment. Nothing is more frustrating than to pour your effort into a project for many months or years only to have it get cancelled when the promise of a successful deployment is within sight. Even worse, there is rarely work immediately available to fill the hole left by the project's cancellation.

While a tad crude, it is not completely inappropriate to consider this a form of coitus interuptus. For an engineer, the job well done is a powerful reward and to withhold that accomplishment is a form of cruel and unusual punishment. While the situation has gotten better through the years, too many companies still do not understand the IT function or the role of software engineers in this function. The churn of CIOs illustrates how difficult it is to be at the intersection of management and technology. Most senior managers today seem to be either incapable or unwilling to make even a two year plan that they can stick to. These changing priorities often lead to the cancellation of projects that have already consumed a significant portion of their project budget.

Another source of failed projects is bad project management. In the world of bespoke software, the challenge to create something completely new is constant. But the more novel, the more risk. Managing that risk while maintaining hubris is a recipe for disaster.

No matter the reason for the cancellation, to the engineer it is all the same. This seems to go to a core value in an engineering culture; you don't work for the money, you work for the intrinsic rewards of the job. I cannot think of a better contrast to this value than that of a sales and marketing professional.


SALES v. ENGINEERING: the eternal struggle
I have grown to think of salespeople as the hunter-gatherers of the corporate world. They live for the kill. Too many times the relationship between sales and engineering is along orthogonal lines. To the salesperson, engineering is there to create the thing to be sold. When that thing is commoditized, there is a comfortable insulation between the two functions. But in the world of software, and especially in the world of bespoke software, the two are close or even combined as you can find in a consulting organization. When these two functions are separate yet forced to coordinate very closely, there is friction.

For operational efficiency, a software intensive product company will attempt to standardize and minimize their internal product line to the extent possible so as to achieve economies of scale. While this internal product line may be expressed in the marketplace as a large array of products, software allows for variation that is needed in the market while minimizing the internal complexity. This is the essence of the product line approach in software architecture.

In contrast the customer often wants a customized product that can rarely be realized from the rationalized product line that exists. Since sales people always want to say yes to the customer, they will accept orders knowing that to satisfy their customer they must ask engineering to make changes. When we are talking about the largest and most complex software intensive products, this was a common occurrence. (Since I've been out of the game for a few years, I can't be certain how prevalent it still is but I'd be very surprised if it still is not an issue for large software vendors of business solutions.) This creates a tension between sales who want to sell a product defined by the customers wants and needs, and engineering and operations who want to sell a standardized and easily maintained product line. Pre-sales support moderates the worst of this, assuming management doesn't, but like the sales people, they are not responsible for delivering on the promise, only the achievement of a signed contract. Pre-sales engineering often participate in some sales compensation commissions, although not as much as the closer, and hence are as motivated to make the sale as the closer. After the order is taken, it is the responsibility of the organization to make the customer happy by delivering the product that was promised.

It is normal for management to focus on its revenue stream. No matter how good the product, a large, complex product must be promoted, explained, negotiated and delivered. For large customizable products, this is no small feat and I have the greatest admiration for those skills. And of course the success of the company depends on those skills. But there must be a balance to the motivation to sell a customized product with no regard for the ability of the organization to deliver on that promise. This leads to the second frustration that most engineers seem to face, the demand for change to the product in an unanticipated way. Sometimes this change even undermines some of the design principles that guided its creation. An engineer will see a hack to a product to make a sale as an affront on the purity of the design, or worse, a huge risk to the reputation they have invested in the product. Since any hack is likely to be required under and unrealistic timetable (from the engineering perspective), there is no upside to the engineer but a significant downside. It is a draw/lose proposition to the engineer since any reward for their heroics will be transitory while any punishment for failure is likely to be severe. Is there any wonder engineers would always prefer to never see a sales person in their area?


SPEED
It does not take too much experience to understand the intoxicating need for speed in the valleys of innovation. From the business perspective, the need to be the first in a new market and maintain a lead is vital to success. I doubt that anyone completely dismisses the association between youth and new skills or the relationship between youth and enthusiasm. Rightly or wrongly, youth is also associated with risk taking. When you put these all together you create a culture that is wired for speed.

One of the Quora comments concerns the excitement and motivation engendered by rapid development cycles. Too many organizations still want to manage risk by using management gates which inherently drive toward a waterfall methodology. The comments clearly show an incompatibility between the management perspective on managing risk and the engineer perspective of reducing or eliminating bureaucratic (to the engineer) obstacles. The first theme raised was empowerment.

Management, especially a command and control structure, values predictability and conformance to established rules. This invariably ascribes a level of management a level of control, their formal authority, to make decisions on behalf of the entire organization. Upper- and mid-level managers must delegate some of this authority to lower levels. Since lower level managers are at the bottom of the hierarchy, are generally the less experienced or less competent managers, their ego often is embodied in their authority. If they are ambitious, they are concerned with pleasing their supervisor and doing an exemplary job so as to better their chances for promotion. For many reasons, they are disinclined to delegate the limited authority they have to line workers. This hesitancy can chafe in an engineering culture which will value rapid management decision making and a resentment of waiting for decisions that appear obvious. It is not uncommon for the delay in a management decision to cause a delay in the critical path of the project and management will both complain about the project delay while deny that they are the cause of the delay. Of course inexperienced project management that does not anticipate some predictable decision points does not help. This is both an example of what is perceived as bureaucratic obstacles and a stop/go or lethargic pace to the project.

Another theme I see in the complaints about speed are the lack of understanding of the risk management is attempting to mitigate. Take for instance the issue of committing changes to production in Google when those changes could potentially lead to bad customer experiences. Anyone in the industry knows how hard it is to build customer goodwill and how quickly it can evaporate. Consider NetFlix. Google is wise to carefully guard this intangible asset. Yet the caution that their management displays is unmotivating for an engineer, especially a young, talented engineer, who is anxious to see the results of their labor. Immaturity and a less risk-averse attitude must surely be a common theme among young, talented software engineers.

One of the many speeches I've heard was a cautionary tale I heard when being prepared for work in the field as a mainline consultant. Remember, I was told, that the people you are working with do not have the same career path you do. As a young consultant, we had reason to believe that we would see a stratospheric soar in our salaries and job prospects due to our work at THE FIRM. Meanwhile the staffs we were working with expected a 9 to 5 job and had made different choices regarding their employment. The obvious message was that we were young, ambitious and typically had no life distracting us from the martial pitch to get the job done. To expect the same motivation of the staffs we worked with was just unrealistic. I'll skip over the smugness and condescension. While the situation is different, many of the same social factors are at work among the software engineers at startups.

I did a stint at Siebel. While I was employee 1500 something and the company had been in business 4 years with most of them profitable, Tom Siebel had the managers referring to it as a start-up and expected  everyone to act like it was. It was 1999, after all, and everyone wanted to be at one. Yes, we got stock options and had all the trappings of a mature start-up. But in my opinion it wasn't and I was very dubious that the stock options would ever be worth much. As it turns out I was mostly right, although I did end up making some from the options before they became completely worthless and it made up for the lower than market wage they paid. But it gave me a taste of what I knew must be going on all over the valley. Young, naive but talented software engineers were employed by very savvy, talented guys that had an appreciation for what the technology could deliver. When they were ambitious they found ways of selling this vision to the employees and were willing to allow them to share in the expected rewards with generous stock options. This model has been working well for over a decade but I think it is far more nuanced than the media portrays.

Despite all the variations in the valley success stories, the one theme they all share is the infectious excitement of making something new and different. This type of greenfield development seems to be the ambition of every software engineer I've known. The usual preclude is maintenance programming and is something that most software engineers seek to leave behind them as soon as possible.This eroticism of the new is a theme I hope to come back to someday when i discuss design. But since this is not found in the posting I'll skip over it. However I believe that this desire to do greenfield development is inherent and helps illustrate the excitement.

When developing a product that does not yet have customers and no existing structure to restrict development, it is all promise. Further, most greenfield development, especially large, mission critical systems, enjoy a great deal of management attention. There are a great many positive incentives, fewer obstacles and a shared vision of the future. Of course the later phases are difficult as these realities begin to intrude but the human assets accumulated during the development can be used to see the project to a successful end. Such projects are often feted and people can take great pride in a job well done, or at least done. The group dynamics of the project invariably lead to rapid development.

The comments stress speed specifically in the form of rapid iterations. It isn't completely clear how rapid iterations contribute to engineer motivation so I'll parse out the comments. Iterations stand in opposition to a strict waterfall methodology. Most waterfall methodologies encourage a focus on the phase transition points and then load them with various quality control points. I doubt that each iteration has all the QA points that a waterfall plan would have and even with some QA between iterations, I doubt the total time devoted is greater in an iterative approach and I suspect a great deal less. This alone may be an explanation for an engineer's preference of iterative over waterfall.

The last point I saw in the post concerned continuous deployment. He contends that at Quora the commitment to get changes into production quickly has many positive effects. He wasn't completely clear as to what their continuous deployment infrastructure includes but presumably it includes some automated tests to ensure any change does not completely break the system. In addition, if there is any form of Test Driven Design, the testing includes some demonstration of the new feature/fixed feature that was the subject of the change.

Why would this be something an engineer would enjoy? Speculating, I would say that knowing a layer of control that is meant to shield me from my errors increases the pressure to be right. Nothing can be more humiliating than to commit a change to a code base that breaks the product. You get immediate attention and none of it is the kind you want. You might make this mistake once in your career but you are not allowed to commit the same mistake too many times and remain on a viable career path. Of course this also increases the intimacy between the effort of a software engineer and their contribution to the organization.


AUTOMATION
This poster suggests that attention to the automation of operational tasks is a key to a high-quality culture that will attract engineers. I'm disappointed he didn't connect the dots for me but I'll speculate on what he has in mind.

As I mentioned, maintenance tasks are generally not fun. The more maintenance, the less fun. People want fun. Minimize the maintenance and you increase your fun to drudge factor. Perhaps it is this simple. But I see more.

One of the standard gripes I heard on projects came from the people who would support the system after it was put into production. Generally the first briefing they had of the design was when the planning for transfer began. The project scope and the scope of design rarely included the after-turnover support. It was not uncommon for a new system to increase the burden on the operational staff, require new skills, be incompatible with other systems or any of a number of other bone-headed moves that some forethought could have avoided. Meanwhile the best projects I saw included at least some consideration of the operational environment in the planning, if not the design. Clearly the best projects will do this.

It's too frequently remarked that maintenance is the most expensive phase of a product's lifecycle. I suspect there are a great many hidden costs in operation and support as well. Where this poster gives me hope is that it sounds like there is a greater emphasis on post-implementation support, at least among the engineers. From a business perspective, this will lead to far greater efficiency of operation and contribute to the overall product profitability over the life of the project. In the end, I see this as a move to consider not only the design of the software, or even the software/hardware system, but of the complete human/machine system and the complete product lifecycle.

There was one reference that I found intriguing. http://codeascraft.etsy.com/2011/02/15/measure-anything-measure-everything/ This link expounds on the virtues of a metric driven approach to operation. I have to believe that if I dig into this meme I will find Basili. He goes on to say:
"...open-source monitoring and charting tools like graphite [5] and statsd [6] highlight an important aspect of automation -- that automation must be driven by data and monitoring. Without monitoring and logs to know what, how, or why something is going wrong, automation is difficult. A good follow-up motto would be to "measure anything, measure everything, and automate as much as possible."
A key point, I think, about the dotCom revolution and it's allied OSS, is the acceptance of putting out products that are likely to contain bugs. This is a stark break with prior practice in the commercial world. dotComs can get away with it because many of their products are free and in the formative stage and are often not mission critical.


DESIGN (build the right software abstractions)
Again, the poster neglected to tie this back to the main point which is how to build a winning software engineering culture. This is a very important topic to me in ways that go beyond its influence on the culture. However I can riff on this although I may not have the same opinion as the poster.

Who doesn't want to play with winners? One of the most ubiquitous cubicle decorations I've seen is the one that reads "It's hard to soar with eagles when you're surrounded by turkeys." This clearly communicates the idea that greatness is within reach but the individuals around us prevent it. It is never clear who the turkeys are (the co-workers? the bosses?) but the thought is clearly that outside forces hold back personal achievement.

This attitude suggests a cynical attitude and a discounting of any personal contribution to the obstacles ("If you are not part of the solution, you're part of the problem") and even worse, a defeatist attitude toward greatness. A good project manager or line manager should know how to counteract this ennui. But to the point, a highly motivated group with high performing individuals is a rewarding and fulfilling group to be in. How much is the peer relations and how much is the management which has created and maintains this group?

To what may have been the writer's main point however, what part does a good design play in the software engineer's motivation? Is it the fact that a good design suggests talented people? Is it because an engineer appreciates the beauty of the design and is rewarded by getting to know it? or learns new skills because of it? Is it because a good design makes a software engineer's job easier by anticipating the build-out of that design and does not get in the way? After all, nothing is more frustrating than being given a design that is immediately seen to be unworkable in the current context. While not every software engineer may have the skills to be a gifted designer, they certainly learn to recognize good ones. Or perhaps it is the relationship between the designer and the construction crew. Rather than having such a bifurcation of staff responsibilities, perhaps it is because the designers are themselves well respected coders who will remain intimately involved in the design through its implementation.

For me, the gold in this part of the post is its references:


These are all references I'll want to use in a post on software architecture.

STRIVE FOR GREATNESS (Develop a focus on high code quality with code reviews)
This section makes many excellent points but again does not focus on the contributions to culture I am trying to develop. Here's my riff.

Pride in workmanship is part of the culture. A staff that truly buys into code reviews is one that exhibits this quality. For code reviews to work properly, several social phenomenon must exist. First, people's egos are strong enough that they can tolerate the review, and receive recommendations, on their products. This speaks to a certain degree of maturity in the individuals and a level of comfort that has been achieved  among peers. When either of these things is lacking, dysfunctions are eventually exposed.

There are a number of quotes that come to mind when I consider this topic. Daniel Burnham, who created a master plan for the city of Chicago, is alleged to have said:
Make no little plans. They have no magic to stir men's blood and probably themselves will not be realized. Make big plans. Aim high in hope and work. Remembering that a noble, logical diagram once recorded will not die.
If he said it, I suspect it was a paraphrase of a similar quote from Machiavelli:
Make no small plans for they have no power to stir the soul.
Personally I prefer Machiavelli's pithier version although it lacks the point that a good design, once created, can live on independently. But both stress the importance of grand ideas to succeed in the leadership of large, complex projects.

I don't know that I can separate culture from leadership. A leader attracts talent to a project and makes an investment to keep everyone motivated and invested in the project. It seems self-evident that the people will contain some common characteristics even if they are nothing more than some momentary suggestion in the mind of the leader that they will make a positive contribution to the project and that they at least accept the leader and want to participate in the project.

An important aspect of project management is to ensure that everyone maintains a common vision of the goal to be reached. A common problem is miscommunication and the tendency of people to hear not what was said, but what they want to hear. Since a software-intensive project must deal with layers of abstraction and vague ideas that must be reified before any product can be realized, there is enormous room for the internal concepts of what will be delivered to wildly diverge from each other. A well managed project will reconcile these divergent visions at or before the last possible moment before which they lead to inefficiency or error.

Another problem that can infect a software engineering culture is the tendency to allow standards to slip. In my opinion, this is failure of management. In one instance, the engineers bring a decision to management that will concern some trade-off between quality and some other economic metric such as time or money. Management may be forced to accept the quality trade-off but how this is communicated to the staff can dramatically affect the culture. On the one hand, if the management accepts that they have incurred a technical debt that must be paid in the future, the engineering staff will see that management is committed to delivering a quality product but must accept certain commercial imperatives. But if managers begin to fudge, rationalize, or deny that de facto decisions that are made, the message to the engineers is that management is more concerned about reputation, money or something else more than they are in the pride they take in achieving the best possible product. Of course this issue is far more nuanced than I let on but it is sufficient to drop a marker on the interplay between decision making and how that can affect an engineering staff.

One management aspect that the poster and I seem to believe is a major contribution to a healthy and desirable engineering culture is the support of code reviews. I see this as a management contribution simply because without their support, it is impossible to create a viable code review process. But implementing this process still requires some courage and faith on the part of management since it has very measurable cost and an unmeasurable benefit. I have seen a few words written as to why people believe code reviews work and I will not take the time right now to research their opinions. Instead I will provide my own knowing full well that most, if not all of them, are the work of others and not original to me.

Socially the most important aspect of a code review process is that it forces engineers to review and discuss their work products. It is perhaps an unavoidable truth about humans that we are always more critical of the work of others than we are of our own. It is also common that people will be sensitive about comments about their work. Yet if these barriers are overcome, debate and an atmosphere of continuous learning begins to happen. Tiny decisions are made by these review meetings and conflicting points of view are aired. Over time, some consensus arises and the I contend that the work of that staff will begin to converge on a common style and uniformity more effectively than any style manual could hope to achieve. This is particularly important where staff members must back each other up to support production. In my case, the list of people who participated in the code review became a matter of record and any member of the review team could be called upon to deal with a problem in that module when the primary was not available. This potential gave an urgency to the work.

The poster suggests that non FTF code reviews can be done and he is absolute right. However I do not believe that they have the same contribution to the staff culture that FTF meetings have. To be in the same room as someone brings a hightened tension to the need to make critical observations or even question someone about their work product. While the improvement to the work product could perhaps be done via computer mediated communication, it does not build the same social skills and interpersonal relationships that FTF meetings have. If you want a cohesive staff, FTF code reviews is one sure way to build strong morale as well as encourage a high-quality, high-performing environment. In the social science literature, this would be due to group norming.


R-E-S-P-E-C-T (Maintain a respectful work environment)
I hope I made the point in the above section on FTF code reviews that they become a vital tool for allowing groups to overcome the obstacles that exist to their effective team effort. Lacking some serious dysfunction, group will naturally move toward some behavioral norms regarding new people or new revelations about people. But the focus on the work product and the mission of the organization often cause people to reevaluate prejudices and try harder to engage in real interpersonal communication.

The poster uses this heading to also discuss how groups can contribute synergy to a design task, a conclusion I endorse. I suppose the cultural issue here is that if engineers are either not asked for their opinions when they should be, or are not listened to when they are, it is understandable why the engineering culture of that organization may flag.


CODE OWNERSHIP (Build shared ownership of code)
The natural tendency for an engineering organization is to divide responsibility for the code along module, sub-system, system boundaries. While logical in that assembly line way of thinking about specialization, it fails to address the stifling aspect of the job. As I've noted elsewhere, software engineers, especially highly talented and young software engineers, are receiving a great deal of intrinsic reward from their job. Asking them to focus on one piece of code to exclusion of the remainder of the system removes the excitement of newness from the job and leads to the inevitable growth of job dissatisfaction. Engineering organizations should take a page from management training programs that move managers frequently so as to give them the broadest possible exposure to the organization. While the organization may lose some instant recall of the internal workings of the code due to the lack of specialization, over time the engineers should become more capable of recognizing the potential causes of systemic and intermittent errors that cross module boundaries.

At a young start-up, compensating different teams differently will lead to strong boundaries between the teams. The greater the distance from one team to another the greater the distance between the code modules. (Conway's law). If Agile methods are ever to reach their maximum effectiveness, people from different areas must be capable of leaving their functional groups and joining together into a cohesive team. The broader the social comfort each team member has with the broader organization, the faster the norming of the group and the sooner they can begin performing.

The poster references the term "bus number" for a project. The reference was another Quora post and I haven't taken the time to research the origin of this phrase. However it is a reference to the number of people in a project who could have a calamitous event (like being hit with a bus) without impacting the team mission. Too often this is 1. For non-mission critical projects, this may make economic sense however it is worth considering how helpful it is to adopt a journeyman/apprentice approach or even a pair-programming approach.


INVEST IN AUTOMATED TESTING
He's right. However I think the poster stops short here. Software engineering is one of the least capital intensive industries in the US today. (I need to find evidence for this but I did once come across a chart showing capital per worker and I remember it being near the bottom). It would be an interesting empirical study to demonstrate how much this means to an organization over the amortization period of the investment in the testing software. The usual obstacle is that most development is done with project level accounting and the project manager can never demonstrate the payback on the investment. Project level accounting for testing software is just boneheaded. It should be at the enterprise level and few senior level executives are convinced that expensive test suites are worth the money. What does he think? That his staff is too lazy to test the products? That people enjoy the often tedious work of testing? This should be a part of every organization's quality program.

The products delivered now are larger and more complex than their predecessors. The skills needed to test them are more complex both because of the complexity of the structures but also because of the greater subject domain they encompass. To deny professionals the tools they need to do their best work just doesn't make any sense. It is also not in keeping with several other goals for a high-quality engineering culture; rapid deployment and Agile processes.

The current practice is to release a product to testing and use basic system's analysis to ensure that it meets its specifications. Once the process is complete it is released to production. But more recent practices are calling for smaller releases that do not lend themselves to the former waterfall approach of holding back changes until a release. The releases are smaller and more targeted. It is difficult to perform the best possible recursion test in this environment without some automated help. In fact, most organizations do not perform complete recursion testing on releases because of the cost of testing it with people.

I think one of the most compelling reasons for an automated test suite is its support for test driven design. While I am not necessarily a proponent, I see many benefits in this approach. First and foremost, it provides an actionable definition of what is expected in the final product. As soon as a feature or error is found, the test to demonstrate that feature is added to the test suite. If full recursion testing is done, all future releases are assured of passing that test or knowing that the release fails that test.

More broadly, the same obstacles that test suites face with enterprise management are faced by many software engineering tools and training. Ironically, the very biggest enterprise purchases such as a database and customer relationship management have a much easier time in the executive suite. This is due to the far greater cost, the smaller number of decisions asked of the executives and the power of the sales and marketing organizations behind these products. Will it be necessary for software engineering tools to coalesce into a mega-application before they can get this kind of attention?


20% Time
This is one section I will quote rather than paraphrase. Since he speaks from experience and provides good citations, it needs no further commentary:
Gmail found its roots in Paul Buchheit's 20% project, and he hacked together the first version in a single day. [http://paulbuchheit.blogspot.com/2009/01/communicating-with-code.html ] Google News, Google Transit, and Google Suggest also started and launched as 20% projects. I used 20% time while at Google to write a python framework that made it significantly easier to build search page demos.  While Google's 20% time may be less productive now than during the early days of the company [http://www.quora.com/Engineering-in-Silicon-Valley/How-does-Googles-Innovation-Time-Off-20-time-work-in-practice ], the notion of letting engineers spend 20% of their time working on something not on their product map remains a cradle of innovation for smaller engineering organizations.

Ooyala didn't officially have 20% time while I was there, but I took some anyway and wrote a command-line build tool for Flex and Actionscript that sped up the team's build times, just as Adobe's Flex Builder tool chain started to degrade, and the tool's still in use today even though the engineering team has nearly tripled in size. Atlassian adopted 20% time after experimenting it for year. [http://www.atlassian.com/company/careers/life ] A variation of 20% time that Facebook's fond of and that Ooyala added later is periodic hackathons -- all-night events where the rule is that you can work on anything except your normal project. [http://gigaom.com/2011/12/16/exclusive-inside-facebooks-final-palo-alto-hackathon/ ]

Top-down approaches to product planning, while necessary for focusing the overall direction of the company, can't account to for the multitude of ideas that might arise from engineers closer to the ground. As long as engineers are accountable for their 20% time and focus on what can be high-impact changes, these projects can lead to large steps forward in progress. Without official 20% time, it's still possible but much more difficult for engineers and designers to try out crazy ideas -- the dedicated ones basically have to find weekends or vacation days to do it.
BUILD A CULTURE OF LEARNING AND CONTINUOUS IMPROVEMENT
The poster makes three valid points. First, software engineers, like everyone, find satisfaction when they achieve what Mihaly Csikszentmihalyi calls a state of "flow", second, weekly tech talks, third, a good mentoring program. Let me take these in order.

I agree that flow is a vital to job satisfaction although I do not see how it relates to culture so much as software engineeer psychology. I can see that when a software engineer is comfortable in her environment that some obstacles to flow have been removed. However I don't see how the culture, per se, can make this happen.

The recommendation for tech talks is one I can support from my own experience. Let's explore the cause and effect. Why would a technical meeting lead to improved culture? It shows management support and recognition for the software engineering profession. It takes time away from other tasks (in the eyes of management) and thus represents a perceived cost to the organization. Whether it ultimately does or not would be a matter of empirical research. But the perception from the engineer is likely to be that of encouragement of new skill development, and a connection to the broader profession. Too many organizations that do not have software engineering as their prime revenue generator do not view software engineers as professionals. This denigration of their skills and fear of the business function leads to an isolation of this group from the broader organizational goals and alienation from the other staff. This alienation is often absent when the software products are the chief output of the organization. Here the software engineers are more valued for their expertise although this is not universal. Software consulting organizations are perhaps the best at both recognizing the value of the software engineers and taking an active interest in their continued growth.

I have seen many mentoring programs and I do not believe they work. I say this with great sadness since I think the relationships can make an enormous difference to the individuals and the organization. Any professional who is a minority in an organization will feel some degree of alienation from the other staff. The job function requires considerable education and training, it is introspective by nature and forces a form of interpersonal communication that is unnatural to the non-professional. Having a mentor is one way to overcome alienation since the humanity of that relationship overcomes the inherently unsatisfying relationship to the organization as a whole. (I had one friend who once remarked that all organizations are inherently dysfunctional maternal objects).

So why do I think mentoring programs don't work? At its core is the growth of an interpersonal relationship between two people, one with more power than the other. This power difference creates interpersonal distance that must be overcome before the benefits of mentoring are seen. This cannot be achieved when the relationship is forced on two people and cannot happen when there is distrust on either side. It cannot be someone in the management chain to which this person reports which creates many obstacles for smaller companies. This represents too many barriers and consequently is rarely effective as a program. What companies should do instead is create opportunities for different levels to mix as equals and to work on creating environments where people are comfortable speaking candidly. It is a rare company that does even this.


HIRE THE BEST
This sound completely reasonable but I do not think this maxim holds up to critical analysis. Does this mean that all these high-performing silicon valley organization all employ the best engineers? If so, how many "best" engineers are there? Are we conflating best with competent? There is no doubt that there is a great difference between engineering talent at any level (Sackman's Second Law, [Enders03] ). I would suggest that "the best" isn't much more than a slogan or a bit of organizational jingoism.

Clearly no one should be given responsibility to create a product that is beyond their means. If the new hire has so little experience or talent so as to not be a contributing member of the team, they should not be given the position. I think it is denigrating to dismiss them as not being the best, something lacking, instead of recognizing that they do not fit the requirements. There are too many skill sets in the marketplace now. Some very talented programmers may have a difficult time coming up to speed on a set of technologies that are unfamiliar to them. Does that mean they are not "the best" or just not the best at that set of tasks?

If technical expertise were all that was required to be a productive member of a team a strict meritocracy would evolve. But there is more to being a highly skilled software engineer than being technically proficient in whatever technologies are needed. The ability to both adhere to group norms and to provide leadership is one pair of traits that can be in conflict. Interpersonal skills are vital to being able to navigate the sometimes difficult terrain of corporate social structures. If you doubt this, just imagine someone with no English skills trying to work as a software engineer in a high-performing organization. Human communication plays a much bigger role in competence than is sometimes recognized.

As to the contribution that hiring practices can have, I will share one anecdote. As a manager of a testing group I was saddled with an incompetent employee who had been hired because of political connections and shuttled around until he found an area where he was unlikely to do much harm. It was difficult to get him fired and it took almost a year but in the end I feared that his departure would impact the group morale since he was well liked. I was wrong. Almost immediately after his departure, the morale significantly improved and productivity went up. Conversely, when I hired someone very competent shortly before his departure, there was also a notable uptick in morale and productivity. The conclusion I came to was that the self perception of the staff was helped by the positive improvements and recognition of both competence and incompetence. Incompetent workers not only create technical debt, they also weigh down a group.


THE RESEARCH PERSPECTIVE
This post would be pointless beyond my self-indulgent rant about the problems in the industry if I did not bring it to closure with an improved focus on my research agenda. My first filter for looking at this material is the question of what open source software is bringing us, how it is doing that and some way of looking at its future. There is ample suggestion that software engineers are inclined to freely volunteer significant talents to the creation, nurturing and maintenance of open source software products. Why? This post explored some key themes the Quora post suggests are on the minds of talented software engineers in the field. My own experience supports the posters claims. While suggestive, this is hardly incontrovertible empirical evidence and needs more careful collection and analysis.

Putting aside the flimsy evidence for the moment and making a guess as to what will be found, I can see many other research questions. What first comes up in my mind is the motivation of contributors. Do software engineers who volunteer for open software projects find the work intrinsically rewarding? What are the components of this instrinsic reward? Is it the sense of ownership they derive? Is it the participatory management style of the projects? Is it the reward of having a seat at the decision making table? the ability to scratch their own itch and create the tools they want to work with instead of allowing management to chose their tools and present obstacles for the acquisition of tools that software engineers believe will be helpful? Is it an altruistic impulse to give back to a community that gives them so many software products that would otherwise be economically impossible to use? Is it a self-serving mechanism to impress a future employer? a way to get mentored by experienced professionals? Is it a form of idol worship where people are motivated by the star status or charisma of a project leader? Is this just part of a job that employer is paying for? If so, what is the motivation of the employer? Why does IBM support Eclipse? Why does Oracle support Java and MySQL? What will happen to these motivations over time? Is this a sustainable model for development? If the motivations are completely self-serving, will they poison the motivations that non-employees feel about contributing? I have heard it said that OSS engineers are scratching their own itch. To what extent are contributors only working on products that they themselves use or had used in the past? Is participation in infrastructure products, like DBMS, web servers, IDEs, etc, more prevalent than end-user products like word processors, graphics tools or web browsers? How many work hours are contributed to projects per year? What is the skill set of the contributors?


software architecture,

LAWS OF SOFTWARE ENGINEERING
Glass's Law: Requirement deficiences are the prime source of project failures.

Boehm's First Law: Errors are most frequent during the requirements and design activities and
are the more expensive the later they are removed.

Boehm's Second Law: Prototyping (significantly) reduces requirement and design errors, especially
for user interfaces.

Davis Law: The value of a model depends on the view taken, but none is best for all purposes.

Curtis' Law: Good designs require deep application domain knowledge.

Many more laws can be found in what looks like an excellent text on software engineering mentioned by MG.
(found in [Enders03] )




FURTHER TOPICS
The software engineering of vaporware
"the teeth of time", "an encounter with the ghosts of the past" ~Stephen Greenblatt author of The Swerve
What is the best measure for program length

REFERENCES
[Enders03] Handbook of Software and System's Engineering: Empirical Observations, Laws, and Theory by Albert Endres and Dieter Rombach, 2003, Pearson available at http://dev.co.ua/docs/a-handbook-of-software-and-systems-engineering-empirical-observations-laws-and-theories.9780321154200.29795.pdf)



THE POSTER

Edmond Lau, Quora Engineer

I work as an engineer at Quora, and I'm loving it.  Prior to Quora, I led the video analytics and monetization team at Ooyala, worked on query suggestions and UI experiments on the search quality team at Google, and studied at Massachusetts Institute of Technology.

In Defense of Clippy

For those of you who met Clippy, I'm sure you're groaning already. But let me explain. While Clippy was a flawed character I will still defend the basic idea that underlied the design and contend that Clippy is out there waiting for his day. In this defense is a tale of the current state of customer service and an insight into why I am dubious of the grand claims for the future of AI.

While living in SF I shopped at the Safeway at Potrero and 16th. One of the notable characteristics of shopping there, and I suspect at all Safeway stores, is how each and every employee would greet you as you passed them in the aisles. It was obviously a requirement on them and I found it annoying since I loose myself in wandering the aisles looking at products I don't know. Each greeting was an unwelcome interruption to my reverie. One day I needed to find a product that could logically have been shelved in one of three areas. After checking all three I decided that I would take them up on their continual offers to help me find something. I got that deer-in-the-headlights look from the clerk and I thought he would start to cry. But he did get me in touch with someone that eventually figured out that the product was in a fourth category that I would never have thought to check. So while there was enforced chiperness in the interest of improving customer experience, it was not backed up by deep knowledge of the merchandise.

Here in Sacramento there is an old line grocery store with an excellent butcher and a deli counter that beats any single shop I knew in the city. The employees are sometimes chatty and friendly, sometimes hurried and focused on their tasks and sometimes even a bit diffident. The variability of their personalities doesn't bother me a bit. I can imagine that some people are intimidated by some of the employees but I also think some people are too sensitive. I say, get over it. What I love about this store is that every time I have a question I get an answer that is almost encyclopedic about the product in question. What I lose in chipperness I get back in competency. In the end, the only thing I want is for them to pay attention to me when I need it and take care of my need in the most efficient way possible. If I want a friend I'll sign onto Facebook.

Once I recognized this I see it all over my life, especially in tech and telephone support. Too many companies give their operators scripts that they tightly enforce and very little deep training in their products. I suspect that this has something to do with the bad reputation that Indian call centers have earned. Perhaps it is the more hierarchical social structures of their history or just the rapid fire growth of the industry there, but the bone headed questions I've gotten from some operators would fill a volume of comedy. ("Hi, my name is Dale, I'm calling about order xyz.", "Yes sir. I'd be glad to help you. But first can I get your name?") What managements have done is substitute rote scripts for true management, supervision and training.

So enter Clippy. Even the name suggests some perky young thing that is eager to help. It's continual dance in the corner of the screen just screamed, "Please, please, let me help you!" But as soon as you ask it something, if you weren't lucky enough to ask it the two things it knew, you got useless information. People blamed the mode of delivery. But what was lacking was the competence. If Clippy was able to have even a fraction of the success at answering a real question about the products it was embedded into, it would not have become the brunt of jokes.

Consider what Clippy had going for it; it was an observer of what you were doing across products. It knew the context. It had access to databases of information. Now we have much better search products that could potentially make Clippy far more capable of intuiting what advice we might be seeking. It could potentially recognize repetitive behavior and offer real help in making our interaction with the product better. Will it? I suspect any employee at Microsoft who suggests anything that sounds like Clippy is immediately taken out back and shot. Too bad. RIP Clippy. I hope you come back in another life. You were much more endearing than the clerks at Safeway.

What's the best way to read a file in Java?

The title of this post was an innocuous and seemingly straightforward question in StackOverflow that caught my eye. (http://stackoverflow.com/questions/4716503/best-way-to-read-a-text-file)  The poster had noticed that there was more than one way to read an ASCII file and as a noob was curious as to what the "best" way was. There was hardly a consensus in the answer and I believe there is something to be learned in that lack of a rapid consensus.

First some background. I do not profess to be a Java expert. Since my needs in this department are simple I usually mimic what I find in the first text I grab. Since there are different ways in different texts, I'll lay those out first.

From Savitch:
Scanner inputStream = new Scanner (new FileInputStream("stuff.txt"));

From Gaddis:
BufferedReader inputFile = new BufferedReader(new FileReader("stuff.txt"));

So pangea suggests Gaddis' approach as does Knubo for his first suggestion. However Knubo offers an alternative that only uses the FileInputStream rather than encapsulating it into a Scanner object. Jesus Ramos also agrees with the first suggestion. Juan Carlos Kuri Pinto suggests his way is better but it still uses the Gaddis classes.

Peter Lawrey offered a novel approach.

for(String line: FileUtils.readLines("my-text-file"))
    System.out.println(line);


as did Claude


The methods within apache.commons.io.FileUtils may also be very handy, e.g.:
/**
 * Reads the contents of a file line by line to a List
 * of Strings using the default encoding for the VM.
 */
static List readLines(File file)

One poster, jzd, referenced a page at Oracle which attempts to provide an overview of the different methods and their differences. (http://docs.oracle.com/javase/tutorial/essential/io/file.html)

So if I were to reply to this poster with all of this information, what would I say? All these suggestions will result in code that will read an ASCII file. What are the differences? What makes this interesting to me is that the differences are not in the gross functionality of the code but rather the non-functional qualities that the resulting system will have.

Let's take a trivial and obvious example, the try-catch block. If we know that all our data is good and want a quick-and-dirty piece of code, we might drop the coding of this block since we want to get the code written quickly and have enough control over the file that the code isn't needed. Here we are trading off coding speed (time-to-market) with robustness. The first exception will cause the code to halt. In many cases we don't care and choose for the quick-and-dirty.

There are two important qualities that vary in these treatments. First, is the relationship between the operating code (the dynamic structures) and time. The buffering feature will provide for a more efficient operation for a static file that is to be read and processed as it might in a batch system. But presumably there is some price to be paid for this feature in terms of processor and memory load. In a modern system it is likely that this is negligible in most systems but for very large, high-performance systems, this cannot always be taken for granted. The quality trade-off may become an architectural consideration if this file reading is in the critical path of a high-performance system or one embedded with limited hardware performance.

Another quality difference between the coding options is maintainability. As jzd observes, Oracle now offers some classes that provide much faster processing but at the cost of readability. This illustrates how quality choices are often trade-offs between two or more different requirements. I suspect in the end, we first code what we know best in the absence of any requirement forcing consideration of other options.

Normally I need to find more complicated examples to illustrate this concept. As the design gets more complex, the design task becomes more complicated

Friday, May 18, 2012

Research Questions: April 2012

In today's symposium, a comment was tossed to me concerning a question of programmer productivity. In the discussion P drew a diagram showing the economic 1% (Xaxis, income, Yaxis, number of people) and compared it to the number of contributions in a typical open source project (Xaxis, contributions, Yaxis number of people). Why, he asked, should the shape of these two curves be so similar? U suggested that it was because of the lower cost of creation that this 1% experienced; that their ability to crank out code faster gave them an advantage. V suggested that it was due to the distribution of talent in the population. I objected to the suggestion that talent was the primary factor and that led P to suggest that some basic research into the factors that influence the contributions to open source projects. Perhaps he's right.

For anyone who has worked in a software shop, we easily accept the oft made assertion that there is a factor of ten between the productivity of a good programmer and the productivity of a poor programmer. I, like I suspect most people who have written code, believe that there is clearly a talent component in one's ability to perform the kind of abstract reasoning and design needed to create computer code. However I also see people generalize far too quickly and accept these assertions uncritically. I, for one, can suggest many reasons that can explain differences in contributions in addition, possibly instead of, talent.

In my own experience, my periods of highest coding productivity happened when coding was my full time job. It seems pretty self-evident that if all you are doing is coding day-after-day you get pretty good at it. This alone can explain a lot of variation in open source software contribution.

Not everyone who contributes to an open source project may be a full-time coder. (Q: How many hours a week does a contributor spend coding? How many of those are for the open source project?)

Even when a contributor is a full-time coder, there may be differences between the coding environment of the open source project and the coding environment of their other coding work. Unlike a shop where everyone is using a small set of languages, possibly only one, with significant support for that language, an open source project will pull in a wide variety of people using a wide variety of languages at various levels. Where the contributor either works full-time on the open source project or happens to code in the same environment in their other coding work, this may contribute to high-contribution levels. (Q: When not all coding is for the open source project, what is the environment of the open source project? What is the environment for the other coding work? Language, version, IDE, )

The Cathedral and the Bazaar

Against all odds, I see hits against this blog. So for those of you who may be following me I'd like to offer a few words about my absence. I am teacshing a course at a local JC titled Introduction to Computer Science which has no articulation to any four year CS curriculum. While the book the supervisor of this course has chosen is probably the best in its class, that is very faint praise. These texts fall into one of two camps: the either mirror a first course in computer science which is often a programming methodology or they are computer literacy. I rebel against the thought that this course should fall into either of those two camps. I cannot countenance the thought of making non-cs majors competent at any level of programming to the exclusion of all other topics. Nor do I accept the "dummies" approach to teaching skills that the average high-school student has already mastered. Computer literacy is not something that should earn a college credit, even at a JC. Instead I approach the course as teaching core topics that are resonant with material from other non-CS classes but which also illustrate important concepts, and vocabulary, from CS. I sometimes spend way too much time thinking about this and trying to twist the material to my liking.

Were this all I am doing, I'd still have plenty of time to blog. But as I'm sure I've already mentioned, I start a PhD program in the fall and am looking to ease into that program with the least amount of trauma and stress possible. To that end, I enrolled in the natural language processing course offered online by the Stanford professors. I took the AI class last semester which did not include programming assignments. I looked forward to the programming of this course. What I did not count on was the linguistic background this course seems to take for granted. So while I was getting up to speed with the mechanics of downloading their scaffolding code, I was mildly challenged to complete their assignments in the week they were given. For me, two assignments, admittedly the hardest of the course, just became too time consuming to take seriously. The first was the creation of a probabilistic context free grammar for a restricted lexicon and the other a probabilistic CYK parser that would be trained from a tree bank they provide. I love both the assignments but I find that I am not willing to spend the time right now to get an acceptable solution to these assignments to the exclusion of the other things going on. I expect to complete them after the course is through but no in time to get any credit for them. Not that the credit matters to me anyway.

So I am posting today to give my thoughts on an article I just read that, in part, addresses a question P. posed some weeks ago, "What does an open source project lose by not having a traditional project manager?". I think the question is one that deserves a good answer since if I cannot articulate the differences between open source projects and traditional closed source projects, I am not as familiar with this form of organization as I need to be.

In my research into open source software, I came across a paper by Eric Steven Raymond titled "The Cathedral and the Bazaar" from about 1996. I suspect this is well know to people more familiar with open source software than I am but I have a tendency to enjoy the search for beginnings and this looked like a good place to start. What I find in this paper is the articulation of many core beliefs I have about the right way to develop software in the context of his own experience developing the fetchmail program. I'll review some of his key points and my thoughts about them.

His first lesson: every good work of software starts by scratching a developer's personal itch. I don't agree with this formulation of the thought but see a basic truth expressed here about why open source software can be so powerful. Systems workers use many tools to do their work: operating systems, compilers, integrated development environments, databases, etc. We become intimately familiar with these tools including their shortcomings and strengths. In the world before open source software these were products that needed to be purchased from an organization, mostly for-profit organizations. Prior to the PC, the tools of production were too expensive for the average person to afford and corporations had an incentive to price them for corporations and not people. This placed the cost of production for software outside the hands of a hobbyist or even a garage entrepreneur. But the open source movement spawned by the transparancy brought by UNIX and the dramatic reduction in the cost of hardware changed that. We now have the ability to change what we don't like in our tools and many incentives to do so. When we have an itch to add a new feature, we now have the ability to scratch it. For those who have the requisite skills and motivation, we can craft our own tools by making small modifications to what is already available. This is very empowering and I believe a major motive force behind open source software. Now I just need to find the evidence to support it or refute it.

Where I disagree with this maxim is the suggestion that this broadly applies to all software. Is it reasonable to assume that there will always be a group of people who will have an itch to develop any imaginable piece of software and be able to grow a circle of supports around it? I find this hard to accept. That would suggest that any commercial product out there could eventually fall to an open sourced alternative. I'm not yet confident enough to say that this is unrealistic, yet I believe this is treating open source software as a silver bullet, and we all know how successful those have been in the history of software engineering. Yes, a great many software products are likely to be at least partly open source software in the future but I believe this will be far more graduated than this maxim suggests.

I cannot consider this maxim without also considering Karl Marx. One of the popular conceptions from his is that the workers should own the means of production. Sadly I have not yet studied his work yet and this is something I will want to do to see if this explains some of the open source software movement.

2. "Good programmers know what to write. Great ones know what to rewrite (and reuse). "
On the surface, this maxim acknowledges that great programmers recognize good code and see how to reuse it. What I immediately get out of this is how other programmers cannot quickly see the value of an existing piece of code and can only understand code that they have written. Alternatively other programmers, in hubris, feel they can always do it better and dismiss code written by others. I think this poses a good research question, "What factors are cited by programmers who have access to other similar code when they duplicate functions?". I also believe, without good evidence, that better programmers are better designers and have greater powers of abstraction for seeing how existing components can be reassembled for a novel application.

3. "Plan to throw one away: you will anyhow." from Mythical Man-Month by Brooks
This maxim is often cited but I have not yet seen a good theory for why this should be so. In my opinion, this happens because the design space is novel to the programmer and the process of creating the first version is an exploration of that space. Often early design decisions necessitate later design choices and block others. It is not uncommon for the programmer to realize an alternative version only after a significant amount of effort and the resistance to the refactoring of the code that would be needed. The second time, if starting from a blank page, the programmer will avoid the bad decisions  of the first construction while retaining to good decisions made in that design.

4. "If you have the right attitude, interesting problems will find you."
It is really not clear from his essay what this maxim really means to him. I can mean that embracing a collaborative mindset will make you open to seeing possibilities for interesting work. I am not sure this is really a software engineering maxim as it is a philosophy of life. There is so much need for high-quality software that the opportunities are boundless. Yet few people find software development interesting. Even those who do can take a parochial attitude that if they are going to do work, it should be for a company that will pay them for their effort. In software, we are lucky enough to enjoy a lifestyle that both rewards us monetarily while offers us an opportunity to do what we love. But as for artists or philosophers, the show Cabaret tells us what a happens to love for something when there is no money "...and the fat little pastor tells you to love everymore, and the hunger comes a rat-tat-a-tat at your window, and your love flies out the door. Money makes the world go around..."

5. "When you lose interest in a program your last duty to it is to hand it off to a competent successor."
How is this software engineering? This is social consciousness 101 and it applies to any work for public good and is seen in boards all across the country. Many a good organization was built by a competent lead and languished under the leadership, or lack thereof, by subsequent people. If you really love something, you have an obligation that goes beyond your own interests.

6. Treating your users as co-developers is your least-hassle route to rapid code improvement and effective debugging.
There may be a software engineering truth here but first let's separate the business truth. No business can be assured of continued customer goodwill if they do not treat their customers with respect. The organization must create value for its customers or else it will be vulnerable. The world is not static and an organization has probably never gotten the function set exactly right in the first place. Either an organization will be responsive to its customers or not. If it is not responsive to its customers, it is vulnerable. Where is the software engineering principle that is separate from the business truth?

Customer intimacy is practiced by organizations that are committed to being responsive. In open source software, the customers are the users who are often the potential developers or at least valued members of the "hallow". The collapsing of these relationships dramatically reduces the communications noise and filtering that would exist in a close source environment. There is no product manager, or any management for that matter, to block or alter a message. Since the communication comes from someone who is more invested in the product, they are willing to give more to see that feature added. This can be extended discussion to work out details of behavior, prototyping or testing. For many reasons, this is often not the case in close-source software and the additional contributions are made at the right place and time to speed the work of the developer.

One aspect that is not discussed in the paper though is the decision making process that a suggestion must go through. In a closed source environment, this is part of the organization processes. I understand there are some standard processes to be found in open source software (OSS) projects but the reality is that someone must decide if this suggestion become acted upon if the requester does not have the skills needed to perform the addition themselves. Earlier I talked about how OSS can differ in the self-selected nature of the people who will work on a project. They are often drawn to work on the tools of their own production. What if the product were a legal database system that served the needs of lawyers? While developers who earned a living consulting to law firms would have a vested interest in the product, the end-users are the lawyers. How would this work in an OSS environment? Would lawyers themselves be posting their suggestions? Would they be articulate enough to provide actionable specifications for the new features they need? Would they remain engaged with the developer long enough to see their suggestion be developed into a product feature? These are all questions in my mind regarding the applicability of OSS outside the domain of system's software.

7. Release early. Release often. And listen to your customers.
I think the listen to your customers is a repeat of a prior point.

This software engineering maxim has proven its worth but I don't think this essay really explains why. Nothing demonstrates an organization's commitment to their customers than an immediate response. OSS projects differ from ordinary organizations in that the submitter is likely to also be the person who submits the proto-solution, whether for a bug-fix or for an enhancement. Whatever the form of leadership, their job is far easier than in a closed source software environment. First, the submitter is motivated to submit a working solution and has most likely demonstrated it in their environment. The OSS project has less of an investment to make. Second, the philosophy of the project is to get the new code in front of as many eyeballs as possible as quickly as possible. There is no expectation of exhaustive quality assurance before the code is seen by a set of users who are inclined to use the test version. The more the code is exercised as a form of black box testing, and examined by other programmers, as a form of white box testing or static analysis, the faster any questionable code will be found. This keeps the process streamlined to production and imposes a caveat emptor on the product pushing more responsibility to test the product against their own acceptance criteria.

While I can see the benefit on the customer side, what I believe is the bigger justification for this maxim is the impact it has on the development side. Long product life-cycles were the norm in the traditional waterfall methodologies.