Surveys are not free, and this is a real constraint.
So far, I’ve briefly reviewed the total survey error framework, which itself is the subject of book length treatments (Weisberg 2005; Groves et al. 2009). Although this framework is comprehensive, it generally causes researchers to leave out an important factor: cost. Although cost—which can be measured by either time or money—is rarely explicitly discussed by academic researchers, it is a real constraint that we ignore at our peril. In fact, the reason researchers interview samples of people rather than entire population is to save money. Thus, cost is fundamental to the process of survey research (Groves 2004). A single-minded devotion to minimizing error while completely ignoring costs is not always in our best interest.
The limitations of an obsession with reducing error are illustrated by the landmark study of Scott Keeter and colleagues (2000) on the effects of expensive field operations in order to reduce non-response in telephone surveys . Keeter and colleagues ran two simultaneous surveys, one using “Standard” procedures and one using “Rigorous” procedures. Although the “Rigorous” procedures did produce a lower rate of non-response, estimates from both samples were basically the same. However, the “Rigorous” procedures cost roughly twice as much and took 8 times as long. Are we better off with 2 reasonable surveys or 1 pristine survey? What about 10 reasonable surveys or 1 pristine survey? What about 100 reasonable surveys or 1 pristine survey? At some point cost advantages must outweigh vague, non-specific concerns about quality.
Many of the opportunities created by the digital age are not about creating estimates that obviously have lower error. Rather, these opportunities are about creating estimates cheaper and faster, but perhaps with errors that are currently higher or harder to measure. As many of the examples in this chapter will show, researchers who insist on a single-minded obsession with minimizing error at the expense of other dimensions of quality are going to miss out on exciting opportunities. Given this background about the total survey error framework, we will now turn to three main areas of the third era of survey research: new approaches to representation (Section 3.4), new approaches to measurement (Section 3.5), and new strategies for combining surveys with digital traces (Section 3.6).
