NewsThe Fact-Check

The Fact-Check: Why John Merrow’s Misleading DC Rant Ignores Facts, Research and the Full Truth

By Matt Barnum | December 10, 2015

Photo: Getty Images
The Fact-Check is The Seventy Four’s ongoing series that examines the ways in which journalists, politicians and leaders misuse or misinterpret education data and research. See our complete Fact-Check archive.
John Merrow, the retired PBS education reporter, does not like Michelle Rhee, Kaya Henderson, or Washington, D.C.’s education reforms.
Rhee was the controversial D.C. schools chancellor from 2007–2010, and Henderson is her successor, having led the district for the last five years. Both Rhee and Henderson implemented policies involving tougher teacher evaluations, expansion of charter schools, and test-based accountability.
Merrow was last seen engaged in a public feud with Success Academy founder Eva Moskowitz over his reporting in a PBS segment on the New York City charter school’s disciplinary practices.
In a recent blog post directed at D.C., he argues, “The approach to ‘education reform’ begun by Michelle Rhee in 2007 and continuing under Kaya Henderson to this day is a failure and a fraud. Washington’s students and teachers deserve better.”
With such strong language Merrow must have some pretty compelling evidence.
As it turns out, not really.
Merrow misuses test data (Part I)
Merrow acknowledges that D.C.’s scores on the National Assessment of Educational Progress (NAEP) “have improved faster than any other urban district’s” but they are “at best, mixed” because such gains were “likely the result of more affluent families moving into Washington and enrolling their children in public schools.”1
It’s difficult, even impossible, to know for sure, but two recent examinations of this question suggest that D.C.’s test score gains were not largely a result of changing student demographics. One comprehensive study found that “overall, even when accounting for changes in student demographics, test scores in the District have improved substantially, especially in math.”
A more recent look at the data reported that only a fraction of the NAEP gains were likely explained by demographic changes: “NAEP results have increased not just across race, but across all subgroups such as gender, disability status, and ELL status.”
More importantly, though, reading the NAEP tea leaves to try to determine the efficacy of certain policies is entirely misguided, referred to by some as “misNAEPery” — in which sweeping declarations are made without being able to tie the test scores back to specific policies.
To be clear: Merrow is not the only one to do this. Many supporters of D.C. reforms have engaged in this unfortunate practice. But that is no justification for making unsupported claims about D.C. NAEP scores, as Merrow has done.
Merrow misuses test data (Part II)
Merrow next points to D.C.’s proficiency rates from the recently released PARCC standardized tests: “Barely 10 percent of District students who took the PARCC geometry test, and only 25 percent of those taking the English test, achieved ‘college and career ready’ status.”
To put it bluntly: using raw proficiency rates is entirely inappropriate for judging the impact of specific education policies. We simply don’t know what the scores would have been in the absence of Henderson’s reforms.
Like most places that have administered new Common Core-aligned tests, D.C. saw proficiency rates drop — not because students are learning less, but because the bar for proficiency was made more difficult. Since the test is new, there’s no baseline for comparison.
Once again, it must be said that Merrow is not alone in abusing data in this way. Some supporters of school reform regularly use raw proficiency data to definitively judge schools or policies. Anyone who does so is wrong. So is Merrow.
Research says “test and punish” improved student achievement
Merrow broadens his critique: “Nationally, many in education people [sic] are waking up to the failures of ‘test and punish,’ and the new [K-12 federal education law signed by President Obama Thursday] pulls back on testing.” He presumably is referring in part to the sanctions levied under the former law, No Child Left Behind, to schools that scored poorly on state exams.
Merrow is certainly entitled to this view, but the empirical research on No Child Left Behind shows that it improved student achievement in math. As Tom Dee, a Stanford professor who has studied the law, told me a while back, “The available research evidence suggests it led to meaningful — but not transformational — changes in school performance.”
The law also produced many unintended consequences, such as teaching to the testing, which may be why Merrow has deemed it a failure. But he ought to acknowledge the real academic gains that NCLB produced as well.
The cheating conundrum
Merrow refers to the evidence that cheating occurred in D.C. under Rhee, which he has reported on extensively. This is an important topic. But Merrow offers no discussion of whether there is reason to believe that such cheating is ongoing or whether test security policies have improved in the D.C. Public Schools.
A National Research Council report states that “from what we could determine … the alleged [cheating] violations were likely not widespread enough to have affected citywide scoring levels” and that “new test security measures that have been introduced in D.C. schools.”
I would welcome a thoughtful discussion of this issue, but the notion that several instances of cheating years ago invalidate the entire D.C. reform project is absurd on its face.
Rigorous research on D.C. reforms paints a nuanced picture
Although Merrow doesn’t cite any of it, there is a good deal of careful research on D.C. that can contribute to a debate on the pros and cons of the policies pursued by Rhee and Henderson.
I would start with the city’s teacher evaluation program, an aggressive system that has led to performance bonuses and firings. Research on the early implementation of the policy shows it pushed many teachers to improve and led less-effective educators to leave the classroom. On the other hand, there is concerning evidence that the evaluation system unfairly penalizes teachers with the hardest-to-serve kids. High-poverty schools in DC experience significant teacher turnover (around 40 percent in the 2010–11 school year), which, although not necessarily bad, does on average reduce student achievement.
Another study found that the city’s policy of replacing principals deemed ineffective boosted test scores.
On school choice, D.C. charters appear to be relatively successful at improving student achievement.
D.C.’s aggressive school closures produced temporary declines in achievement for students directly affected, though they rebounded fairly quickly. There is no evidence on whether students who would have attended closed schools benefited, though in New York City they did.
However, the number of African-American teachers in D.C. has plummeted in recent years, potentially because of closure and teacher evaluation policies. Research has found that students of color benefit from teacher diversity.
The National Research Council report raised important concerns about how D.C. collects, disseminates, and uses data. It’s also worth noting that enrollment in D.C. Public Schools has increased in recent years, perhaps a sign that families are more confident in the quality of that schools, though we can’t assume cause and effect.
All of this is to say that there is a serious, nuanced discussion to be had on the impact of D.C. education policies. But John Merrow’s tirade is not it.
(Disclosures: I worked as a summer intern at StudentsFirst, an organization founded by Michelle Rhee, a few years ago. The Seventy Four’s Editor-in-Chief Campbell Brown sits on Success Academy’s board of directors.)


Footnotes:

1. As his source, Merrow says, “Earlier this year a report by the National Research Council pointed out that most of the academic gain was likely the result of more affluent families moving into Washington and enrolling their children in public schools.” I took a look at this report and I’m not sure what Merrow is referring to. The study concludes: “The percentage of all students scoring proficient or above in reading and math on the DC CAS increased between 2007 and 2014. The increase is larger for math than it is for reading. The positive trends are also apparent on NAEP.” However the report warns, “Although we can document some of the changes that occurred over the past 7 years, we cannot determine the independent effects of [DC reform efforts] on achievement and attainment. Changes in the demographic composition of D.C.’s public school students, the growth of the charter sector, differences in the programmatic choices made in DCPS and the individual charter schools, and many other changes that have occurred are intertwined with the changes brought by [DC education policies] …. The signs of improvement are positive, but a more complete picture of student outcomes is needed.“ In other words, as far as I can tell, the National Research Council suggests that demographic changes may be an important factor, but certainly does not definitively claim that they led to the observed test score gains. (Return to story)

Submit a Letter to the Editor