Controlled experiments in software engineering
Compare techniques while managing developer variation
Controlled software engineering experiments compare tools, languages or development practices under planned conditions. Treatments may be assigned to programmers, teams or tasks; outcomes can include correctness, effort, completion time and maintainability. Developer experience, task difficulty, learning and carry-over require explicit design decisions. The design tests a bounded comparison rather than the intrinsic superiority of a technology.
Choose it when a specific intervention can be manipulated and meaningful tasks can be assigned fairly, with participants whose expertise and working conditions relate to the intended population.
Strengths
- Supports causal comparisons under defensible assignment
- Can examine how expertise and task complexity modify effects
Limitations
- Laboratory tasks may omit teamwork and maintenance realities
- Student samples may differ from professional developers
Know the boundary
More developer-hours and shorter elapsed time are different outcomes; one cannot substitute for the other.