Randomized Controlled Trials Remain the Gold Standard for Ed Tech Research

Get stories like this delivered straight to your inbox. Sign up for The 74 Newsletter

In a recent interview published by The 74, Leanlab founder Katie Boody Adorno says randomized controlled trials may be “an outdated mode of research.” I wholeheartedly disagree. 

RCTs remain the gold standard for effective research for good reason. They reduce sources of bias that plague other designs by accounting for observed and unobserved characteristics between the groups being studied. They can often be done quickly and provide the strongest evidence of whether a product’s impact varies based on students’ racial, ethnic and socioeconomic backgrounds. Most important, RCTs answer the key question about any ed tech product: Does it work? This is not to say RCTs are always appropriate; for instance, using one to evaluate a program to be used from elementary school through high school would be unworkable. But, where feasible, RCTs are the best approach to determining whether a product or program functions as intended.

Ed tech is a rapidly growing $150 billion industry that shows no signs of slowing down. In fact, it will likely double or triple in size over the next decade. It is absolutely essential that ed tech companies work with researchers throughout the development process to ensure that their products are as effective for educators and students as their marketing materials claim. A range of research designs, including RCTs, is required to ensure a product works as intended in the courses and classroom for which it was created.

Randomized controlled trials are uniquely structured to determine whether a product’s impact is the result of the product itself or other factors. This matters — a product that works for students with a teacher who’s enthusiastic about it may not be effective if the teacher is less familiar or uninterested. In an RCT, the product is randomly assigned to a group during a fixed period of time, while a comparison group continues with its usual activities. The groups have similar characteristics, including the degree of interest in the product.

This means that RCTs can reliably reveal which interventions are working. This is especially important in efforts to bolster the success of the most vulnerable students. For example, a randomized controlled trial of a statewide public prekindergarten program in Tennessee found that it was actually degrading future academic growth in elementary school for low-income kids — an alarm bell that weaker methods would not have been able to detect because they could not have controlled for the fact that families who self-select into public pre-K are different from families who don’t. Only through randomized control was it possible to determine whether the program was producing the desired outcomes for all students regardless of their families’ economic status or focus on early education.

Beyond producing rigorous evidence of impact, RCTs have other advantages over different designs.

First, they are simple and transparent. Because they compare two similar groups whose only difference is the use of a product, RCTs clearly and directly demonstrate whether the kids in Group A tend to do better than the kids in Group B. The answer can be clearly understood without the use of statistical models or nonreproducible “insights” of the type that come from classroom observations.

Second, because conducting RCTs requires only as much statistical training as one gets in a typical master’s program involving math and statistics, such as finance or economics, they can easily be set up by state and local educational agencies. This frees local leaders from relying on studies done in other schools and districts that have different student demographics and serve different communities, allowing them to move from asking, “Does this work overall?” to, “Does this work for my students?” 

Third, RCTs can strengthen studies that are conducted to help shape the final product. When ed tech products are in at the start of the development process, it’s often unknown whether version A or B works better. Suppose you’re building an app that focuses on fractions and you want to find out if one visual representation engages students more than another. Randomly assigning students within a classroom to see either version A or B will produce better information than letting the students decide on their own which one to view.

Fourth, RCTs eliminate inherent bias. Many districts and schools pilot programs and products with volunteers before deciding whether to scale them more broadly. RCTs provide a more realistic picture of implementation because the group testing the product is assigned randomly rather than because its members have already expressed interest in it.

It is undeniable that RCTs require some time to complete, but that is the cost associated with rigor, and developers, educators and students all deserve to benefit from the most rigorous research methods available. The ed tech cemetery is rife with products that couldn’t deliver on their promises. The nation’s children and educators deserve educational tools backed by the strongest evidence possible to support learning and academic success. RCTs provide that evidence better than any other method does.

Get stories like these delivered straight to your inbox. Sign up for The 74 Newsletter