Math Study Shows Difficulty in Motivating Teachers to Change Behaviors

Get stories like this delivered straight to your inbox. Sign up for The 74 Newsletter

Like an online retailer trying to woo a customer back by offering a 10% discount on the boots they’ve been eyeing, education researcher Angela Duckworth wanted to understand how to incentivize teachers to log in regularly to an online math platform that aims to help them improve their students’ academic performance.

“Today is perfect for checking your Pace Report!”

“Keep Zearning!”

“By opening this email, you’ve earned another 100 digital raffle tickets in the Zearn Math Giveaway!”

In partnership with Zearn Math, a nonprofit online math instruction platform used by roughly 25% of U.S. elementary school students, Duckworth and a team of researchers from the University of Pennsylvania’s Behavior Change for Good Initiative launched a megastudy that peppered 140,000 teachers with different types of email prompts to log into the platform’s dashboard each week and check their students’ progress.

Behavioral scientists like Duckworth, who popularized the “power of grit” about a decade ago, spend a lot of time trying to pinpoint what, exactly, it is that prompts an individual to sign a form, become an organ donor or click an ad that promises a secure and safe retirement now.

“In the case of education there’s the idea of nudging the students directly,” Duckworth said. “But there’s also the idea that’s less commonly studied, which is, what do you do to nudge the teachers, who are not in complete charge, but have a lot of authority about what is going to happen in the classroom that day? It was clear to us that if we could get the students onto the Zearn platform that their learning would progress. But are they actually going to log in?”

To that end, the team developed 15 different types of intervention emails featuring things like planning prompts, teaching tips, learning goals, digital swag and celebrity endorsements. The goal was to change behavior without mandates, bans or substantial financial incentives — though teachers were enrolled in a giveaway and earned digital raffle tickets every time they opened an email, increasing their chances of winning such prizes as autographed children’s books, stickers and gift cards. 

The researchers then compared the average number of lessons the teachers’ students completed on the Zearn Math platform over four weeks to a control group using Zearn that received only a simple weekly email.

So did it work? Did the emails prompt teachers to log in more regularly? And if so, did the number of lessons their students completed increase? To some degree, yes, it did work. But not at all to the extent that Duckworth and researchers had anticipated. 

The best-performing intervention, which encouraged teachers to log into Zearn Math for an updated report on how their students were doing that week, produced a 5% increase in students’ math progress. Emails that referenced data specific to a teacher’s students — versus those without that information — boosted students’ progress by 2.3%. And teachers who received any of the behaviorally informed email nudge saw their students’ math progress increase by an overall average of 1.9%

Duckworth was sure that the emails featuring famed astrophysicist Neil deGrasse Tyson and literary rockstar Judy Blume would move the needle more than anything else. But teachers were virtually unaffected. 

“We had sexier treatment conditions,” she said. “But no, it turns out, a simple message that says, ‘Hey, your students’ data are here, remember to log in,’ that is what worked the best.”

Notably, the intervention effects were consistent across school socioeconomic status and school type, both public and private. Moreover, they persisted for eight weeks after the email intervention period ended. Collectively, the reminders resulted in students completing an estimated 80,424 additional lessons during the four weeks their teachers received emails, and an estimated 156,117 additional lessons during the following eight weeks.

Yet the limited impact of the email reminders surprised virtually everyone involved with the study: Students whose teachers received any type of behaviorally-informed email reminder only marginally outperformed students whose teachers received a simple email reminder. In fact, the effect was at least 30 times smaller than forecasted by the behavioral scientists who designed interventions, by Zearn Math staff and by a sample of elementary school teachers. 

“It’s a sober reminder that big effects are very rare,” said Duckworth. “In general, we’re finding in our megastudies and what’s emerging across the social sciences is that intervention effects tend to be very small.”

“One of the things that this megastudy has reinforced is a kind of humility about how complicated human beings are and how challenging it is to durably change behavior. A kid is a complicated organism. Teachers are complicated. Schools are complicated,” she continued. “It would be naive to think that you could radically change behavior with these like light touch interventions.”

The findings not only underscore the difficulty of changing behavior, but also the need, Duckworth said, for large-scale, rigorous, empirical research on how to drive impact in math, which is a high-priority subject for education policy experts at the moment. 

Indeed, the findings come at an inflection point for math in the U.S. 

The most recent release of the National Assessment of Educational Progress showed that, nationally, average mathematics scores in 2024 were lower by 3 points among fourth-grade students and lower by 8 points among eighth-grade students compared to their scores in 2019 – the most significant drop since 1990. School districts have struggled to rebound after significant academic setbacks incurred by the COVID-19 pandemic. For math in particular, by the spring of 2022, the average public school student in grades three to eight had lost the equivalent of a half-year of learning.

Compared to students in other developed countries, Americans have ranked in the bottom 25% of students globally on standardized tests of mathematics for decades. U.S. students saw a 13-point drop in their 2022 Programme for International Student Assessment math results when compared to the 2018 exam — “among the lowest ever measured by PISA in mathematics” for the U.S., according to the Organisation for Economic Co-operation and Development, which administers the exam. 

As a result, a contentious debate has erupted surrounding whether educators are effectively teaching the subject — and whether they themselves are being effectively taught how to teach it. 

“There was a dawning realization that there’s a real urgency around math achievement in the United States,” Duckworth said when her team decided to design the megastudy. “This very light touch nudge was helpful, but it does underscore how hard behavior is to change. And if there are bigger levers to influence teacher behavior, I think we would have found a bigger downstream effect on student achievement.”

Get stories like these delivered straight to your inbox. Sign up for The 74 Newsletter