Whether you are doing things yourself or working with a partner, I’d like to offer four pieces of advice that I’ve found particularly helpful in my own work. The first two pieces of advice apply to any experiment, while the second two are much more specific to digital-age experiments.
My first piece of advice for when you are doing an experiment is that you should think as much as possible before any data has been collected. This probably seems obvious to researchers accustomed to running experiments, but it is very important for those accustomed to working with big data sources (see chapter 2). With such sources most of the work is done after you have the data, but experiments are the opposite: most of the work should be done before you collect data. One of the best ways to force yourself to think carefully before you collect data is to create and register an pre-analysis plan for your experiment in which you basically describe the analysis that you will conduct (Schulz et al. 2010; Gerber et al. 2014; Simmons, Nelson, and Simonsohn 2011; Lin and Green 2016).
My second piece of general advice is that no single experiment is going to be perfect, and, because of that, you should consider designing a series of experiments that reinforce each other. I’ve heard this described as the armada strategy; rather than trying to build one massive battleship, you should build lots of smaller ships with complementary strengths. These kinds of multi-experiment studies are routine in psychology, but they are rare elsewhere. Fortunately, the low cost of some digital experiments makes multi-experiment studies easier.
Given that general background, I’d now like to offer two pieces of advice that are more specific to designing digital age experiments: create zero variable cost data (section 4.6.1) and build ethics into your design (section 4.6.2).
