4.4.3 Mechanisms

Experiments measure what happened. Mechanisms explain why and how it happened.

The third key idea for moving beyond simple experiments is mechanisms. Mechanisms tell us why or how a treatment caused an effect. The process of searching for mechanisms is also sometimes called looking for intervening variables or mediating variables. Although experiments are good for estimating causal effects, they are often not designed to reveal mechanisms. Digital age experiments can help us identify mechanisms in two ways: 1) they enable us to collect more process data and 2) they enable us to test many related treatments.

Because mechanisms are tricky to define formally (Hedström and Ylikoski 2010), I’m going to start with a simple example: limes and scurvy (Gerber and Green 2012). In the 18th century doctors had a pretty good sense that when sailors ate limes they did not get scurvy. Scurvy is a terrible disease so this was powerful information. But, these doctors did not know why limes prevented scurvy. It was not until 1932, almost 200 years later, that scientists could reliably show that vitamin C was the reason that lime prevented scurvy (Carpenter 1988, p 191). In this case, vitamin C is the mechanism through which limes prevent scurvy (Figure 4.9). Of course, identifying the mechanism is very important scientifically—lots of science is about understanding why things happen. Identifying mechanisms is very important practically. Once we understand why a treatment works, we can potentially develop new treatments that work even better.

Figure 4.9: Limes prevent scurvy and the mechanism is Vitamin C.

Unfortunately, isolating mechanisms is very difficult. Unlike limes and scurvy, in many social settings, treatments probably operate through many interrelated pathways, which makes isolation of mechanisms extremely difficult. However, in the case of social norms and energy use, researchers have tried to isolate mechanisms by collecting process data and testing related treatments.

One way to test possible mechanisms is by collecting process data about how the treatment impacted possible mechanisms. For example, recall that Allcott (2011) showed that Home Energy Reports caused people to lower their electricity usage. But, how do these reports lower electricity usage? What were the mechanisms? In a follow-up study, Allcott and Rogers (2014) partnered with a power company that, through a rebate program, had acquired information on which consumers upgraded their appliances to more energy efficient models. Allcott and Rogers (2014) found that slightly more people receiving the Home Energy Reports upgraded their appliances. But, this difference was so small that it could only account for 2% of the decrease in energy use in the treated households. In other words, appliance upgrades were not the dominant mechanism through which the Home Energy Report decreased electricity consumption.

A second way to study mechanisms is to run experiments with slightly different versions of the treatment. For example, in the experiment of Schultz et al. (2007) and all the subsequent Home Energy Report experiments, participants were provided with a treatment that has two main parts 1) tips about energy savings and 2) information about their energy use relative to their peers (Figure 4.6). Thus, it is possible that the energy saving tips are what caused the change, not the peer information. To assess the possibility that the tips alone might have been sufficient, Ferraro, Miranda, and Price (2011) partnered with a water company near Atlanta, GA, and ran a related experiment on water conservation involving about 100,000 households. There were four conditions:

• a group that received tips on saving water.
• a group that received tips on saving water + a moral appeal to save water.
• a group that received tips on saving water + a moral appeal to save water + information about their water use relative to their peers.
• a control group.

The researchers found that the tips only treatment had no effect on water usage in the short (one year), medium (two year), and long (three year) term. The tips + appeal treatment caused participants to decrease water usage, but only in the short-term. Finally, the tips + appeal + peer information treatment caused decreased usage in the short, medium, and long term (Figure 4.10). These kinds of experiments with unbundled treatments are a good way to figure out which part of the treatment—or which parts together—are the ones that are causing the effect (Gerber and Green 2012, Sec. 10.6). For example, the experiment of Ferraro and colleagues shows us that water saving tips alone are not enough to decrease water usage.

Figure 4.10: Results from Ferraro, Miranda, and Price (2011). Treatments were sent May 21, 2007, and effects were measured during the summers of 2007, 2008, and 2009. By unbundling the treatment the researchers hoped to develop a better sense of the mechanisms. The tips only treatment had essentially no effect in the short (one year), medium (two years), and long (three years) term. The tips + appeal treatment caused participants to decrease water usage, but only in the short-term. The advice + appeal + peer information treatment caused participants to decrease water usage in the short, medium, and long term. Vertical bars are estimated confidence intervals. See Bernedo, Ferraro, and Price (2014) for actual study materials.

Ideally, one would move beyond the layering of components (tips; tips + appeal; tips + appeal + peer information) to a full factorial design—also sometimes called a \(2^k\) factorial design—where each possible combination of the three elements is tested (Table 4.1). By testing every possible combination of components, researchers can fully assess the effect of each component in isolation and in combination. For example, the experiment of Ferraro and colleagues does not reveal whether peer comparison alone would have been sufficient to lead to long term changes in behavior. In the past, these full factorial designs have been difficult to run because they require a large number of participants and they require researchers to be able to precisely control and deliver a large number of treatments. But, the digital age removes these logistical constraints in some situations.

In summary, mechanisms—the pathways through which a treatment has an effect—are incredibly important. Digital age experiments can help researchers learn about mechanisms by 1) collecting process data and 2) enabling full factorial designs. The mechanisms suggested by these approaches can then by tested directly by experiments specifically designed to test mechanisms (Ludwig, Kling, and Mullainathan 2011; Imai, Tingley, and Yamamoto 2013; Pirlott and MacKinnon 2016).

In total, these three concepts—validity; heterogeneity of treatment effects; and mechanisms—provide a powerful set of ideas for designing and interpreting experiments. These concepts help researchers move beyond simple experiments about what “works” to richer experiments that have tighter links to theory, that reveal where and why treatments work, and might even help researchers design more effective treatments. Given this conceptual background about experiments, I’ll now turn to how you can actually make your experiments happen.