Surveys are not free, and this is a real constraint.
So far, I’ve briefly reviewed the total survey error framework, which itself is the subject of book-length treatments (Weisberg 2005; Groves et al. 2009). Although this framework is comprehensive, it generally causes researchers to overlook an important factor: cost. Although cost—which can be measured by either time or money—is rarely explicitly discussed by academic researchers, it is a real constraint that should not be ignored. In fact, cost is fundamental to the entire process of survey research (Groves 2004): it is the reason researchers interview a sample of people rather than an entire population. A single-minded devotion to minimizing error while completely ignoring cost is not always in our best interest.
The limitations of an obsession with reducing error are illustrated by the landmark project of Scott Keeter and colleagues (2000) on the effects of expensive field operations on reducing nonresponse in telephone surveys. Keeter and colleagues ran two simultaneous studies, one using “standard” recruitment procedures and one using “rigorous” recruitment procedures. The difference between the two studies was the amount of effort that went into contacting respondents and encouraging them to participate. For example, in the study with “rigorous” recruitment, researchers called the sampled households more frequently and over a longer period of time and made extra callbacks if participants initially refused to participate. These extra efforts did in fact produce a lower rate of nonresponse, but they added to the cost substantially. The study using “rigorous” procedures was twice as expensive and eight times slower. And, in the end, both studies produced essentially identical estimates. This project, as well as subsequent replications with similar findings (Keeter et al. 2006), should lead you to wonder: are we better off with two reasonable surveys or one pristine survey? What about 10 reasonable surveys or one pristine survey? What about 100 reasonable surveys or one pristine survey? At some point, cost advantages must outweigh vague, nonspecific concerns about quality.
As I’ll show in this rest of the chapter, many of the opportunities created by the digital age are not about making estimates that obviously have lower error. Rather, these opportunities are about estimating different quantities and about making estimates faster and cheaper, even with possibly higher errors. Researchers who insist on a single-minded obsession with minimizing error at the expense of other dimensions of quality are going to miss out on exciting opportunities. Given this background about the total survey error framework, we will now turn to three main areas of the third era of survey research: new approaches to representation (section 3.4), new approaches to measurement (section 3.5), and new strategies for combining surveys with big data sources (section 3.6).
