Whether you are doing it yourself or working with a partner, I’d like to offer two pieces of advice that I’ve found particularly helpful in my own work. First, think as much as possible before any data has been collected. This advice probably seems obvious to researchers accustomed to running experiments, but it is very important for researchers accustomed to working with big data sources (see Chapter 2). With big data sources most of the work happens after you have the data, but experiments are the opposite; most of the work should happen before you collect data. One of the best ways to force yourself to think carefully about your design and analysis is to create and register an analysis plan for your experiment. Fortunately, many of the best-practices for the analysis of experimental data have been formalized into reporting guidelines, and these guidelines are a great place to start when creating your analysis plan (Schulz et al. 2010; Gerber et al. 2014; Simmons, Nelson, and Simonsohn 2011).
The second piece of advice is that no one experiment is going to be perfect, and because of that, you should try to design a series of experiments that reinforce each other. I’ve even heard this described as the armada strategy; rather than trying to build one massive battleship, you might be better building lots of smaller ships with complementary strengths. These kinds of multi-experiment studies are routine in psychology, but they are rare elsewhere. Fortunately, the low cost of some digital experiments makes these kind of multi-experiment studies easier.
Also, I’d like to offer two pieces of advice that are less common now but are particularly important for designing digital age experiments: create zero marginal cost data and build ethics into your design.
