Study: Chicago Tutoring Program Delivered Huge Math Gains; Personalization May Be the Key

A year after mayors and governors announced the first school closures related to COVID-19, many have turned to personalized tutoring to cope with disruptions to learning. Families that could afford to hire private instructors began doing so even before the 2020-21 school year began, while governments in Europe launched full-fledged programs to work with thousands of students at risk of falling behind.

The latest research findings indicate that those efforts could be among the most effective means of helping students make up for lost time in the classroom. A working paper released today by the National Bureau of Economic Research points to the enormous mathematics gains achieved by high schoolers through the Saga Education tutoring initiative. The effects were found in combined samples including over 5,000 children, suggesting the possibility that they could be further expanded to serve many more students.

Across two randomized controlled trials conducted in the 2013-14 and 2014-15 school years, ninth and tenth graders in Chicago who received regular tutoring saw significant improvements on both math test scores and course grades. While the upward movement was sizable in both experiments, the gains were much larger during the program’s second year of implementation.

Monica Bhatt, one of the study’s authors and the senior research director at the University of Chicago’s Education Lab, called the results “remarkable,” particularly because they came in a study of older students, whose primary years of academic and social-emotional development have already passed and for whom education interventions are typically less effective. That persistent phenomenon generates “a lot of pessimism among some social scientists,” she said in an interview.

“Results for interventions at the high school level are…often not as successful, at least in rigorous studies,” Bhatt said. “That leads people to think it’s too late, and I think our study shows that it’s not too late to intervene, and in fact you can do so very successfully.” 

No shortage of previous evidence has confirmed the power of tutoring to lift student academic achievement. A meta-analysis circulated last year found huge average effects across nearly 100 experimental tutoring studies, the overwhelming majority of which focused on programs that worked with primary school students.

Saga’s own performance was included in that analysis, whose authors characterized the program’s successes as “among the most noteworthy” they encountered. The working paper released today synthesizes the previous findings while adding additional data and some evidence of how the results were achieved.

In each of the two experimental trials, over 2,500 students in representative Chicago public high schools were randomly selected to receive high-dosage tutoring from Saga, a nonprofit enterprise that originated in Boston’s Match Charter Public School. The offerings were somewhat typical of the supplemental instruction that is often required at high-performing charters like Match: Two students were assigned to each tutor for hour-long sessions during the school day; the tutors themselves were recent college graduates without teaching credentials who worked at the program over the course of nine months.

Test scores from the first study showed that participants roughly doubled the math growth of an average high schooler over one year, according to previous estimates; improvements from the subsequent year roughly doubled that result. In percentile terms, their scores improved by six points in the first study and 14 points in the second.

Course performance improved commensurately, as the students’ math grades grew by about .56 points on a 0-4 scale. In the first trial, failures in math courses dropped by 49 percent among participating students. And in follow-up data, Bhatt added that she and her fellow researchers found little sign that the effects faded over time; 11th-grade math course grades among students who received tutoring were .18 GPA points higher than in the control group.

The working paper also offers some informed theorizing about how the gains were achieved.

One explanation of how tutoring delivers academic improvement hinges on the bond between instructor and pupil: by spending long periods in close contact with a mentor, especially when isolated from the social and behavioral distractions that often take teachers off-task in classrooms, students gain more effective instruction and learn more content.

But in studying the variation between the high schools where Saga’s students were enrolled, the authors didn’t find a strong correlation between fewer disruptions and larger academic improvements. Instead, they argue in favor of an alternative theory — that working with just two students allows each tutor to personalize instruction much more than a teacher, working with as many as 30 teenagers, is able to do. The testing data provides some support for this hypothesis, as lower-performing students within the sample were found to improve more on easier assessment items, while their relatively higher-performing peers saw bigger jumps on the harder questions.

Bhatt said that she and her collaborators would continue studying the benefits of Saga’s tutoring strategies, including by looking at the results of a pilot program to deliver instruction online (some of which was still being administered when the pandemic first closed schools last spring). But increasingly, she added, discussion should turn to the question of how more students can receive access to it. 

“We’re just scratching the surface of how to do this at scale, but I think it’s incredibly promising. I really do think we have to stop asking ourselves, ‘Well, what really works?’ because we have more indications of what works. Now we have to figure out how to actually do it in the context of U.S. public schooling.”

Disclosure: The Bill & Melinda Gates Foundation, the Chan Zuckerberg Initiative and The Walton Family Foundation provide financial support to Saga Education and The 74.

Get stories like these delivered straight to your inbox. Sign up for The 74 Newsletter