My Notes on Qualitative Techniques in Empirical Software Engineering Research - Part 1

I think my current line of study can best be summarized by a paragraph from the book, Guide to Advanced Empirical Software Engineering by Shull, singer, Sjøberg. This paragraph is from chapter two which is a paper by Carolyn Seaman titled Qualitative Methods:

The study of software engineering has always been complex adn difficult. The complexity arises from technical issues, from the awkward intersection of machine and human capabilities, and from the central role of the people performing software engineering tasks. The first two aspects provide more than enough complex problems to keep empirical software engineering researchers busy. ut the last factor, the people themselves, introduces aspects that are especially difficult to capture. However, studies attempting to capture human behavior as it relates to software engineering are increasing and, not surprisingly, are increasingly employing qualitative methods.

This post is my first attempt to distill some of what I am learning about qualitative techniques in software engineering research. My first stop is from a text I have from a software metrics class which seems to frame this well for me. In Software Metrics: A Rigorous and Practical Approach by Norman Fenton and Shari Lawrence Pfleeger they state that there are three investigative techniques: survey, case study, and formal experiment. They characterize surveys as research in the large, formal experiments as research in the small, and case studies as research in the typical. They also point out that surveys are most often retrospective while case studies and formal experiments require a decision regarding what will be investigated. They present four principles of investigation that are common to all three types of investigation. The principles of investigation are:

1. choosing an investigative technique
2. stating the hypothesis
3. maintaining control over variables
4. making your investigation meaningful

At this time, I am more interested in looking at case studies than the other two techniques. On page 148, section 4.3 is on Planning Case Studies.

While many issues are shared between formal experiments and case studies, the book explores some differences.

A case study usually compares one situation with another: the results of using one method or tool with the results of using another, for example. To avoid bias and make sure that you are testing the relationship you hypothesize, you can organize your study in one of three ways: sister project, baseline, or random selection.

Sister Projects
Suppose your organization is interested in modifying the way it performs code inspections. You decide to perform a case study to assess the effects of using a new inspection technique. To perform such a study, you select two projects, called sister projects, each of which is typical of the organization and has similar values for the state variables that you have planned to measure. for instance, the projects may be similar in terms of application domain, implementation language, specification technique, and design method. Then, you perform inspections the current way on the first project, and the new way on the second project. By selecting projects that are as similar as possible, you are controlling as much as you can. This situation allows you to attribute any differences in result to the difference in inspection technique.
Baselines
If you are unable to find two projects similar enough to be sister projects, you can compare your new inspection technique with a general baseline. Here, your company or organization gathers data from its various projects, regardless of how different one project is from another. In addition to the variable information mentioned above, the data can include descriptive measures, such as product size, effort expended, number of faults discovered, and so on. then, you can calculate measures of central tendency and dispersion on the data in the database, so you have some idea of the "average" situation that is typical in your company. Your case study involves completing a project using the new inspection technique, and then comparing the results with the baseline. in some cases, you may be able to select from the organization database a subset of projects that is similar to the one using the new inspection technique; again, the subset adds a degree of control to your study, giving you more confidence that any differences in result are caused by the difference in inspection technique.
Random selection
Sometimes, it is possible to partition a single project into parts, where one part uses the new technique while the other does not. Here, the case study resembles a formal experiment, because you are taking advantage of randomization and replication in performing your analysis. It is not a formal experiment, however, because the project was not selected at random from among the others in the company or organization. In this case, you randomly assign the code components to either the old inspection technique or the new. As with an experiment, the randomization helps to reduce the experimental error and balance out the confounding factors.
This type of case study design is particularly useful for situation where the method being studied can take on a variety of values. For example, you may want to determine whether preparation time affects the effectiveness of the inspections. You record the preparation time as well as component size and faults discovered. You can then investigate whether increased preparation time result sin a higher detection rate. 

 At the end of this chapter, there some suggestions for further reading. The ones that catch my eye are:

Curtis, B., "Measurement and experimentation in software engineering," Proceedings of the IEEE, 68(9), pp. 1144-57, 1980

Basili, V.R., Selby, R.W., and utchens, D.H., "Experimentation in software engineering," IEEE Transactions on Software Engineering, 12(7), pp. 733-43, 1986

Shen, V.Y., Conte S.D., and Dunsmore, H.E., "Software science revisted: A critical analysis of the theory and its empirical support," IEEE Transactions on Software Engineering, 9(2), pp. 155-65, 1983

Swanson, E.B. and Beath, C.M., "The use of cast study data in software management research," Journal of Systems and Software, 8, pp.63-71, 1988

Kitchenham, B., Pickard L., and Pfleeger, S.L., "Case studies for method and tool evaluation," IEEE Software, 12(4), pp. 52-62, 1995