‘No One Is Above the Law’: Divisive Trump Surrogate Carl Paladino Removed From Buffalo School Board

Veto Override Uncertain as Fight Over Funding Illinois Schools Moves to the House

Noble Network of Charter Schools: It’s Not Just About Going to College, but About Global Perspective & Leaving Chicago

74 Interview: David Hardy on Putting Purpose Before Politics and Kids Before Adults in Leading Ohio’s 2nd State-Takeover District

For Schools, an Eclipse Conundrum: To Open or Close? For Fun or for Science?

New Poll Shows Sharp Decline in Support for Public Charter Schools Over Past Year

A Massachusetts Teachers Union Votes to Kill a Successful Charter School, as Families Scramble for Answers

WATCH: Mission to Mars Video Wins $10,000 and Visit to NASA for 4 NJ Middle Schoolers

Jason Botel Reportedly Out at Education Dept. as Feds Reject ESSA Plan From DeVos’s Home State

2 in 3 High School Students Know of Kids Who Cheat Using Digital Devices — but Few Admit Doing It Themselves

Fewer Than 1 in 3 Americans Support Kids Opting out of Tests; About Half Confused on What ‘Opt Out’ Means

Call Her RoboKid: How a Cutting-Edge Robot Is Helping an Ohio Student Attend Classes While She’s Sick at Home

LearnLaunch Accelerator Gives a Boost to Ed Tech Startups Worldwide From Its Boston Home

No More School Daze? California Weighs Making Middle & High Schools Start Later So Students Can Sleep In

This Week in ESSA: Pennsylvania Looks to Cut Testing Time, Indiana Reformats A–F Grades & 3 More Approvals

What Our Kids Made at Summer Camp: Proud Parents Posting Adorable Photos of Arts & Crafts on Social Media

74 Interview: Michael Lomax, CEO of the United Negro College Fund, on Guiding Low-Income Students Through College

Los Angeles Schools Launch Campaign and Resource Guides to Protect Immigrant Students

300 Tutors, Working With Students 2 Hours a Day: One School Network’s Investment in Personalized Learning

Data Sharing, Data Dumping & Claims of ‘Academic Fraud’ in Tweetstorm Over Story About Louisiana Vouchers

$8 Billion a Year on Ed Tech, but Does It Work? Experts Call for Better Research at Unique D.C. Symposium

May 4, 2017

Talking Points

Only about half of districts ask for evidence before purchasing #edtech #ShowTheEvidence

“Politics affect decision-making more than cost, effectiveness, or any other rational input” #ShowTheEvidence

Sign Up for Our Newsletter

Washington, D.C.

When it comes to proving their worth, education technology companies are struggling to put their money where their mouths are.

The $8 billion–plus ed tech industry has ballooned in recent years as tech tools flood K-12 and higher education to adapt classrooms now largely occupied by digital natives. The primary source for federal ed tech funding under the Every Student Succeeds Act is authorized at $1.65 billion. And while investments in the market slowed last year, private dollars continue to drive massive growth and innovation in the sector.

But in an industry that raised more than $1 billion across 138 venture deals last year, there exists an incongruence in evidence to support that explosion of investor and consumer confidence. Also missing is purchaser demand for proof that these innovations work once they make into the classroom. Experts believe, however, that better ed tech research would likely be driven by demand from schools and districts, and not because investors and tech companies seek evidence to grow business.

(The 74: #ShowTheEvidence: Building a Movement Around Research, Impact in EdTech)

In a new study of 63 ed tech companies, 90 percent of respondents reported that they were very or extremely confident that their product has its intended impact, while just over half of those who made claims of efficacy had research to back up their assertion. Overlaid is the finding that only 48 percent of consumers inquire about the relevant research.

“Most surprising to us – only about half of the school districts actually asked for evidence,” said Mahnaz Charania, director of strategic planning in research and program evaluation for Georgia’s Fulton County Schools. “We’re not asking the right questions. So hopefully this research will get us to say, ‘We need to look at this a little deeper and figure out what we need to do about this big gap in what we call quality and what we call evidence.’ ”

A graphic illustrating evidence and solutions presented at the EdTech Efficacy Research Academic Symposium in Washington, D.C.
Photo: Emmeline Zhao

The research was presented during the first day of the two-day EdTech Efficacy Research Academic Symposium Wednesday in Washington, D.C. The findings come from just one of 10 working groups identifying, analyzing, and prescribing recommendations for the role of efficacy research in ed tech development, implementation, and decision-making across sectors. The symposium – hosted by the University of Virginia’s Curry School of Education, Digital Promise, and the Jefferson Education Accelerator — brought together 275 researchers, teachers, entrepreneurs, professors, administrators, and philanthropists with a focus on actionable solutions.

Charania and David Daniel, a professor of psychology at James Madison University in Virginia, led the study. Also among their findings, 57 percent of ed tech developers reported conducting their efficacy research internally, while 20 percent said they use third-party evaluators. And in assessing their products, most companies don’t conduct efficacy research during product development, instead favoring user feedback and engagement data to improve a product over time. Just 42 percent of respondents said they have a standard process for evaluating the impact of their product.

“Our findings suggest that ed tech developers value research to inform their products but, for a variety of reasons, are not conducting rigorous research,” Charania and Daniel write. “Although most developers are focusing their outcomes on implementation fidelity and ease of use, these important aspects must be viewed as a start towards rigorous research and not the end goal.”

 

The rigorous research that Charania and Daniel call for still faces obstacles. The pace of the research cycle is slow — at times requiring years — and some of the best-researched technology often becomes obsolete after surviving the extended process. Researchers for a second working group found that nearly 90 percent of respondents to their survey did not insist on research evidence to make an ed tech purchasing or implementation decision.

“We need a paradigm shift in terms of how we’re working with schools,” said Michael Kennedy, an assistant professor at the University of Virginia and research lead on that study. A universal ratings system could be a solution, he said, to create more accountability in the sector.

Not only are those purchasing decisions often made without the basis of evidence, they’re also frequently the result of lip service paid by colleagues or counterparts in neighboring districts, a third working group found.

“Politics affect decision-making more than cost, effectiveness, or any other rational input,” working group lead Fiona Hollands said. Hollands is associate director and senior researcher for the Center for Benefit-Cost Studies of Education at Columbia University’s Teachers College in New York City.

The confidence and assumptions that ­trusted associates rigorously test recommended technologies are often misplaced, said Bart Epstein, founding CEO of Jefferson Education Accelerator and an associate professor at the University of Virginia’s education school.

Companies know that products multiply through and across communities as a result of that proximity, and Epstein said developers frequently ask him who key influencers are, or whether he can connect them with major school districts. If those mammoths pick up a particular product, it’s more likely to succeed.

Particularly in a torpid ed tech market with capricious investors in the driver’s seat, the onus is on researchers not to influence backers on the supply side but to develop educated mechanisms that foster collaboration and move the needle with schools and districts on the demand side, Epstein said.

“When you ask them publicly, every ed investor will only want to fund the things that work – they’re mission-driven,” Epstein said. “But if you ask them behind closed doors, they’ll tell you that unless they have specific social capital given to them by a foundation or family office, they invest in what provides returns. When research drives returns – meaning more sales – investors will do it.”

“Until then,” he added, “it’s not a ... productive use of our time to cajole or guilt investors into doing more research. What they’re doing is generally more rational. Investors tell us over and over what they want is smart consumers. We need better consumers. We need better tools to support our consumers so they make better decisions.”