4.5 Making it happen

Even if you don’t work at a big tech company you can run digital experiments. You can either do it yourself or partner with someone who can help you (and who you can help).

By this point, I hope that you are excited about the possibilities of doing your own digital experiments. If you work at a big tech company you might already be doing these experiments all the time. But, if you don’t work at a tech company you might think that you can’t run digital experiments. Fortunately, that’s wrong; with a little creativity and hard work, everyone can run a digital experiment.

As a first step, it is helpful to distinguish between two main approaches: doing it yourself or partnering with the powerful. And, there are even a couple of different ways that you can do it yourself; you can experiment in existing environments, build your own experiment, or build your own product for repeated experimentation. I’ll illustrate these approaches with lots of examples below, and while you are learning about them you should notice how each approach offers trade-offs along four main dimensions: cost, control, realism, and ethics (Figure 4.11). No approach is the best in all situations.

Figure 4.11: Summary of trade-offs for different ways that you can make your experiment happen. By cost I mean cost to the researcher in terms of time and money. By control I mean the ability to do what you want in terms of recruiting participants, randomization, delivering treatments, and measuring outcomes. By realism I mean the extent to which the decision environment matches those encountered in everyday life; note that high realism is not always important for testing theories (Falk and Heckman 2009). By ethics I mean the ability of well-intentioned researchers to manage ethical challenges that might arise.