That’s not the plot of an education-themed sci-fi show, but the aspiration of a new Fordham Institute report attempting to identify “risk factors” that predict the failure of a charter school. The study, based on data from four states, concludes that charters are more likely to be low-performing when their application does not identify a principal, states a commitment to a “child-centered instructional model,” and indicates plans to serve disadvantaged students without providing individualized tutoring.
In the foreword to the report, Fordham’s Amber Northern and Michael Petrilli say the findings should be helpful to policymakers but must be interpreted carefully. “If our results are used to automatically reject or fast-track an application, they have been misused,” they write. “Yet they ought, at minimum, to lead to considerably deeper inquiry, heightened due diligence, and perhaps a requirement for additional information.”
The study, “Three Signs That a Proposed Charter School Is at Risk of Failing,” was conducted by Anna Nicotera and David Stuit and released by the Fordham Institute, a conservative education think tank that generally backs charter schools as well as rules to hold them accountable for test score performance.
The analysis relies on data from Colorado, Indiana, North Carolina, and Texas, four states with large charter school sectors. Using information on charter school applications between 2011 and 2014,1 the researchers coded for a number of factors to see which predicted approval, and, in turn, performance.
In general, approval rates were fairly low, ranging from 46 percent in Colorado to 11 percent in Texas. The authors identified a number of indicators that predicted which applications would get the green light. For instance, schools that did not identify any plan for evaluating teachers or intend to hire an external management company were more likely to be rejected.
Then, looking at schools that did open, the authors identified them as low-performing if they met two criteria based on standardized tests: below-average scores on student growth and rankings in the bottom quarter of the state in student proficiency.
This approach has a number of limitations. Most researchers say proficiency is a poor measure of school performance because it doesn’t account for out-of-school factors. It also relies solely on test scores as a measure of quality, even though other factors are also important.
Nicotera, lead author of the study, said the goal was to include common measures for judging schools: “We wanted to make sure we were capturing two elements that are typically used in every school’s scorecard. Schools are very rarely measured on just growth, even if that might be ideal.”
Using this definition, the report pointed to three indicators that predict low performance:
Failure to name a school leader or contract with a management organization was a particularly strong predictor.
“For authorizers overwhelmed by extensive applications that can run to one hundred pages of content or more, these risk factors provide a good starting point for flagging applications that need an especially thorough review,” the authors write.
But although the study describes the risk factors as “hard to game,” it’s not clear that that’s true. If applicants know they’ll be dinged for, say, promising a student-centered curriculum, they might simply omit their planned approach. Charter applications do not always reflect what happens when a school opens. For instance, in North Carolina, half of charter schools that promised in their application to offer transportation for students did not actually do so, researchers found.
The findings are quite different from a similar study looking at charter schools in New Orleans. That analysis, released last year, found that no observable factors from a school application seemed to consistently predict charters’ effectiveness in raising test scores. This report looked at largely different factors than the Fordham study. However, unlike the latest paper, the New Orleans research did not find that naming a school leader was related to future performance.
An important caveat to both studies is that by their nature, they can’t examine the performance of schools whose application is denied and therefore never open. It may be the case that the results would be different if it were possible to determine how those (hypothetical) schools would perform.
The latest research highlights a deeper philosophical debate: What is the purpose of school choice? To some, like the Fordham Institute and the report’s authors, the goal seems to be to maximize academic performance, as measured predominantly or exclusively by standardized tests.
“When a charter school consistently produces sub-par academic results for its students, it is a sign that the latter half of the ‘charter school bargain’ (better results in return for more autonomy) is not being met,” the authors write. “Failing schools can have profound political and financial implications, but the foremost concern is that they harm students.”
But others view the purpose of charter schools quite differently: as allowing families to make choices for their children based on whatever dimension of quality they care about. Research shows that parents value a school’s academic performance but also consider other factors.
There remains some academic debate around how well test score gains predict longer-run student outcomes. One recent study showed that charter schools in Texas whose students made smaller gains on tests also had lower incomes as adults (though bigger test score improvements didn’t seem to produce much income gain).
The finding that test scores are lower in schools with a child-centered approach crystallizes the debate about the purpose of charters: Parents may appreciate this approach for a variety of reasons beyond achievement gains, and advocates for this approach may, likewise, judge its effectiveness by other measures.
“Theoretically, it does worry me, because these schools could be really adding value to a community and to kids who don’t have the opportunity to have that kind of instruction,” Nicotera said. “We were attempting to have a proxy for what an authorizer would be looking at in terms of determining whether this school is successful — and in this day and age, it’s test scores.”
Disclosure: The report was funded in part by the Walton Family Foundation, which is also a funder of The 74.
1. For Colorado, the researchers had data from 2009 to 2014 (return to the story).