The Fact-Check is The Seventy Four’s ongoing series that examines the ways in which journalists, politicians and leaders misuse or misinterpret education data and research. See our complete Fact-Check archive.
The misinformation, a-contextual data on New Orleans’ school reform will not stop. The latest edition comes from Andrea Gabor, writing in The New York Times that “it is more important than ever to accurately assess the results, the costs and the continuing challenges” of the New Orleans reforms.
That’s reasonable enough, and Gabor makes a few important points. In particular, she is right to urge caution to those trying to replicate New Orleans’ virtually all-charter model in other cities. But much of the piece — far too much — is a stream of ill-supported arguments designed to discredit the significant progress made in New Orleans schools.
Strap in, as we correct the record:
If not progress, then what?
One of the chief problems with commentaries that seek to downplay the success of New Orleans’ school reforms is a refusal to focus on measurable progress. Gabor’s piece is a case study in precisely that. She briefly acknowledges test score gains — based on Tulane economist Doug Harris’ rigorous research — before quickly downplaying them, pointing to the city’s low ACT scores and the alarming number of young adults in the New Orleans metro area who are neither in school nor employed (“disconnected youth”).
But these data tell us absolutely nothing about the success or failure of school reform in New Orleans — it’s possible that these numbers would be even worse in the absence of the reforms. Gabor makes no attempt to consider the data over time.
In fact, the average ACT scores in the city have gone up significantly — although without more sophisticated analyses it would be inappropriate to credit the reforms for these gains. The city also compares favorably to other urban districts that require all students to take the test. (Gabor does not mention that Louisiana requires all high school juniors to take the ACT, even those who do not intend to go to college.)
On the question of disconnected youth, there is no data available showing whether the problem has gotten worse or has improved in the New Orleans metro area since Hurricane Katrina. I confirmed as much with with Kristen Lewis, of Measures of America, which put out the report that Gabor cites. In an interview Monday, Lewis also emphasized the fact that New Orleans should be viewed in the context of the entire state — Louisiana is last in the country among all fifty states, and Baton Rouge actually has a slightly higher number of disconnected youths than New Orleans, particularly among black residents. Again, though, any inferences about school quality based on these numbers are entirely unsupported.
To say the city still has problems or that the schools could improve further is a truism. But as an indictment of the New Orleans reform, such claims fall short. And if the progress that New Orleans has made — Harris says he has “never seen an effect of this size before” — is not good enough, then what would be?
No evidence that data is skewed
Gabor suggests the data showing school success in not to be trusted, in part because of the disconnected youth numbers. She claims that students “often disappear from school entirely and, thus, are no longer included in the data.”
But when looking at cohort graduation rates — which measure the percentage of students starting ninth grade who graduate, and is thus unlikely to be affected by the number of disconnected youth — New Orleans has made significant progress. Graduations rates have increased from 54 percent in the 2003-04 school year to 73 percent in 2013-14.
Moreover, insofar as the disconnected youth create an upward bias, this would have also occurred in earlier data as well. Without evidence showing increases in disconnected youth — evidence which Gabor fails to cite, and which doesn’t appear to exist — attempts to debunk New Orleans’ progress are simply conjecture.
Teacher diversity: reformers agree it’s a problem
Gabor writes: “A key part of the New Orleans narrative is that firing the unionized, mostly black teachers after Katrina cleared the way for young, idealistic (mostly white) educators who are willing to work 12- to 14-hour days.”
It is absolutely true that after Katrina, the number of teachers in New Orleans who are black dropped precipitously, but the teaching force remains half black. Moreover, contrary to Gabor’s suggestion, supporters of reform are not celebrating the decline in black teachers, but trying to fix it. For example, a recent report from the pro-charter group New Schools for New Orleans acknowledges a “major teacher diversity problem.” To be fair, talk is cheap, but the data suggest that although newly hired teachers remain disproportionately white, more black teachers have been hired in recent years.
School takeovers – problem or solution?
Gabor says: “Consider Joseph S. Clark Preparatory High School, one of the city’s last traditional public schools to be ‘taken over.’ Most of its 366 students declined to re-enroll when it reopened under new management in the fall of 2011. During its first year under FirstLine, a charter management organization, Clark had only 117 ‘persisters,’ or returning students, according to a study by Stanford University’s Center for Research on Education Outcomes, known as Credo.”
Here, Gabor simply assumes her premise — that the disruption inherent in closing a community school — is a bad thing. But when researchers from MIT looked at schools in New Orleans taken over by charters, they found large gains in achievement. That does not preclude the possibility that some students were made worse off by closing of traditional public schools — research on this question in other parts of the country is mixed — but Gabor shares precisely zero evidence to make this case.
The reforms failed and they only worked because of the money
Gabor says that “one of the most important post-Katrina reforms” was “a big increase in both government and philanthropic funding.” She is quite right on this front, but it’s a bizarre point for Gabor to underscore. After all, she has just gotten done explaining how the system has been a failure to a large extent, but now we’re told that the infusion of money was a big success. How does Gabor know this? No evidence on this point is offered.
Least advantaged students suffer?
To close her piece Gabor writes, “Privatization may improve outcomes for some students, but it has hurt the most disadvantaged pupils.” Previously she claims, “There is also growing evidence that the reforms have come at the expense of the city’s most disadvantaged children.”
But Gabor’s “growing evidence” is fabricated. Harris found broad gains in achievement across New Orleans, and the writ large gains for the largely low-income, largely black students are a huge victory for anyone who cares about educational equity. There is simply no empirical evidence on offer suggesting that the New Orleans reforms harmed disadvantaged students in terms of achievement or any other educational outcome. None.
Gabor does cite research showing that disadvantaged students are less likely to enroll in the highest performing schools. This is a real problem and squares with evidence from other cities, but it falls far short of showing that disadvantaged students are actually worse off. Paradoxically some of the same research finds that disadvantaged students who do enroll in charters benefit the most. If Gabor’s concern is that too few of these disadvantaged students have access to charters, well, she should be praising their massive expansion in New Orleans. (For a more extensive discussion on this question including research from other parts of the country, see here.)
You say miracle, everyone else says progress
The heart of Gabor’s piece is this: “The New Orleans miracle is not all it seems.” Yet purveyors of the city’s reform explicitly say that no miracles have occurred — just real progress.
Perhaps this framing explains why Gabor thinks the New Orleans experiment has been such a disaster. If she is looking for miracles, any reform effort will inevitably come up short.
Photo by Getty Images: A student attends a dance class at the Encore Academy charter school in May in New Orleans.