Why is random assignment important for cause and effect?

Editor’s note: This blog post is part of an ongoing series entitled “Technically Speaking.” In these posts, we write in a way that is understandable about very technical principles that we use in reading research. We want to improve busy practitioners’ and family members’ abilities to be good consumers of reading research and to deepen their understanding of how our research operates to provide the best information.

Making Causal Inferences with Random Assignment

In our previous “Technically Speaking” blog post, we explained the first of two important goals when conducting a study that attempts to measure the effectiveness of a reading intervention on student outcomes:

1. Be able to generalize results to a wider student population
2. Establish a causal relationship between the reading intervention and changes observed in student reading

If random sampling addresses the goal of generalizing results as we wrote previously, what do researchers implement to make causal inferences about the reading intervention being studied? A key design component to facilitate this is random assignment. Whereas random sampling helps facilitate the external validity of a study (i.e., the degree to which findings can be appropriately generalized from a sample to a population), random assignment helps establish a study’s internal validity. This refers to the degree to which one can conclude that changes to the instruction based on the treatment and changes in students’ outcomes reflect a cause-effect relationship between the two (Shadish, Cook, & Campbell, 2002). In other words, just how sure can we be that the changes in student reading performance observed during the study are a result of the reading intervention that was implemented?

Randomly assigning students to treatment conditions means every student has an equally likely chance to end up in either the treatment/intervention group or a comparison group. A comparison group is important for establishing internal validity because it serves as a way to determine what the outcome would be in the absence of the intervention. Without a comparison group, it becomes difficult to disentangle potential effects of an intervention from factors such as general student maturation (particularly if the study extends over time), biases in how student participants were selected for the study conditions, or other uncontrollable factors. What may serve as a reasonable comparison group in a reading intervention study? Depending on the study, a comparison group could be a control group of students that do not receive any reading intervention at all (e.g., a treatment group attends summer school and a control group does not), a classroom that receives the “status quo” or usual reading instruction that students receive, or a classroom that receives an alternative reading intervention that differs from the intervention of interest in the study.

Random assignment, then, establishes that—before the intervention begins—the treatment and comparison groups are equal on expectation. On expectation means that while student characteristics may not be necessarily the same if a handful of students from each group is selected at random, on average over all possible assignments of students to conditions, the important characteristics between treatment and control group would be similar (Shadish et al., 2002). In other words, in the long run groups will be similar to one another on average for all characteristics other than the treatment condition itself. This expected equality of groups reduces the likelihood that other factors that could affect the outcome of a study are confounded with the treatment. In other words, if groups were randomly equated prior to the intervention, then any differences we observe after the intervention is implemented could not be the result of any preexisting differences prior to intervention. By removing these potential confounding factors, random assignment reduces the plausibility of alternative explanations for any observed relationship between the reading intervention and student outcomes (Shadish et al., 2002).

Consider the alternative to random assignment—where students or teachers are assigned to groups in some planned or systematic way (e.g., only highly experienced teachers being chosen to pilot a new curriculum) or where the teachers are allowed to choose the group to which their students would belong. This leads to a selection bias, where there are systematic differences in teacher or student qualities in the two groups that are completely independent of treatment condition. Random assignment removes, or at the very least diminishes, these selection biases.

Random assignment to groups can occur at different levels—schools, classrooms, students within classrooms, or a combination of these. For example, a study’s design may have one school in a district implement an intervention of interest while another school in the district will serve as a comparison group – ideally, the researcher would then randomly assign the intervention of interest to one of these schools, while the other school receives a different intervention. On the other hand, if different classrooms within the same school will be receiving different interventions, then random assignment of conditions to each classroom is needed. There are strengths and weaknesses for designing implementation of random assignment at each of these levels within a study.

Perhaps the biggest challenges of random assignment arise when the study design calls for students within the same classroom to receive different intervention conditions. For one, if different interventions are implemented within the same classroom, there is the concern of treatment contamination (i.e., students receiving components of the treatment when they were supposed to receive the control condition, or vice versa). On the other hand, there may be less of a teacher effect when both interventions are implemented within the same classroom by the same teacher. Another challenge is that the differences in student abilities within classrooms typically are larger than the average differences in student abilities between classrooms. This large variability within classrooms means there is a fairly high chance that students’ reading abilities will not be balanced between the treatment and comparison groups despite randomly assigning students within the class to conditions. Although there are ways statistically to try to account for these imbalances during the analysis phase of a study, there are also ways this can be addressed within the assignment process itself. Pairing students by ability and then randomly assigning one member of the pair to a condition (and thereby placing the other member in the other condition) is one method that can provide some level of group balancing while still using random assignment.

An Example of Random Assignment in Reading Research

Building on the example from our previous “Technically Speaking” blog post on random sampling, recall that the colored dots represent a sample of 100 students with different characteristics. We now assume that we will randomly allocate half of our sample (50 units or students) to the treatment group (Figure 1) and half of them (50 units) to the comparison group (Figure 2).

Figure 1. Treatment Group Using Simple Random Assignment

Figure 2. Comparison Group Using Simple Random Assignment

The figures above illustrate the treatment and comparison groups if simple random assignment were used. As the visual representation suggests, the important student characteristics represented by the colored dots do not appear similarly in both figures. This means that the students in these two groups may not be similar enough to compare their results, depending on how important it is for these student characteristics to be distributed evenly between the two groups. A way to address this will be discussed after this example.

Let us look at this another way. Below, we have broken down each group by the numbers. Each letter A through J represents an important student characteristic.

Treatment Group

Comparison Group

We can see above that the number of students with each characteristic in some cases differs substantially between the groups (e.g., there 6 students with characteristic J in the treatment group but only 2 students with characteristic J in the comparison group). This demonstrates that random assignment equates groups on expectation, but perhaps not always in reality. If we are sure that these student characteristics must be evenly represented across samples, we typically rely on block randomization. In short, this means we would create a pool or block of students by each characteristic of interest and randomly allocate students within each block. Figures 3 and 4 display the same samples as displayed in Figures 1 and 2, but in this case, block randomization was used.

Figure 3. Treatment Group Using Block Randomization

Figure 4. Comparison Group Using Block Randomization

Looking back at the breakdown of the groups by the numbers of students with each lettered characteristic, we can confirm that block randomization was more successful in equating our samples than simple random assignment (e.g., there are now 4 students with characteristic J in both the treatment and comparison groups). Remember, each letter represents an important student characteristic that should be evenly distributed between the two groups.

Random sampling and random assignment are two distinct techniques used in research. As we explained in our previous post, random sampling aids in our ability to create representative samples from a population of interest. This post addressed how random assignment aids in creating treatment and control groups that are equated on expectation for important characteristics. Random sampling is related to the ability to generalize our study results to a wider universe of students, and random assignment is related to establishing the causal effect of a reading intervention. When used together, these techniques provide researchers sound evidence of whether or not a reading intervention has the potential to help students become more skilled readers.

References

Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Cengage Learning: Boston, MA.

How does random assignment help cause and effect?

Does random assignment determine cause and effect?

What is random assignment Why is this so important?

What is the importance of random sampling and random assignment in research?