4.5 Making it happen

Even if you don’t work at a big tech company you can run digital experiments. You can either do it yourself or partner with someone who can help you (and who you can help).

By this point, I hope that you are excited about the possibilities of doing your own digital experiments. If you work at a big tech company, you might already be doing these experiments all the time. But if you don’t work at a tech company, you might think that you can’t run digital experiments. Fortunately, that’s wrong: with a little creativity and hard work, everyone can run a digital experiment.

As a first step, it is helpful to distinguish between two main approaches: doing it yourself or partnering with the powerful. And there are even a few different ways that you can do it yourself: you can experiment in existing environments, build your own experiment, or build your own product for repeated experimentation. As you’ll see from the examples below, none of these approaches is best in all situations, and it’s best to think of them as offering trade-offs along four main dimensions: cost, control, realism, and ethics (figure 4.12).

Figure 4.12: Summary of trade-offs for different ways that you can make your experiment happen. By cost I mean cost to the researcher in terms of time and money. By control I mean the ability to do what you want in terms of recruiting participants, randomization, delivering treatments, and measuring outcomes. By realism I mean the extent to which the decision environment matches those encountered in everyday life; note that high realism is not always important for testing theories (Falk and Heckman 2009). By ethics I mean the ability of well-intentioned researchers to manage ethical challenges that might arise.