In New York City, Teachers Cheated to Help Students Graduate — and it Worked

The study released by the National Bureau of Economic Research found that score manipulation, often called “grade scrubbing,” on the required Regents exams was common practice; that teachers didn’t seem to be driven by their own personal interests; and that white and Asian students on the threshold of passing were more likely to be nudged over the line than black or Hispanic students.

The researchers also revealed that students truly in jeopardy of not graduating, benefitted — were more likely to get a diploma — by having their scores elevated that little bit. But students who might have otherwise finished high school with an advanced degree if made to go back and repeat a course, were hurt when teachers fudged their exams.

“Small differences in getting over a bar have long-term implications for student outcomes — and that’s worth thinking about,” said Jonah Rockoff, a Columbia researcher and co-author of the study.

The analysis follows a 2011 Wall Street Journal story showing that New York City students were far more likely to score just above 65 — the score needed to pass the Regents exams — than just below it.

New York state requires almost all high school students to pass Regents exams in English, math, science, U.S. History and Government, Global History and geography in order to graduate¹ — among the toughest requirements in the country. Soon after the Wall Street Journal story broke, the state did away with a longstanding policy of re-grading Regents whose scores fell just below failing and centralized the scoring system so that teachers were no longer marking tests from their own school.

Using data between 2003 and 2013, researchers were able to take advantage of that before-and-after grading change to examine what happened to affected students. The researchers say they didn’t have the same rich data set for districts outside New York City but their evidence suggests that Regents scores were tampered with throughout the whole state.

Like prior research, the study documents that New York City students were far more likely to get a score of 65 than 64, or other scores just above the graduation cutoff as opposed to just below it.² This can’t be explained by the point structure of the exam, suggesting that the teachers doing the grading intentionally gave students a boost to help them over the graduation bar.

The authors refer to this as “manipulation,” rather than cheating.

A remarkable 40 percent of New York City Regents exams near the graduation bar — or 6 percent of all Regents tests — were scrubbed, the study estimates. Some schools were more likely to fudge the data, and some students were more likely to gain from it.

In total numbers, black and Hispanic students benefited the most from the practice because they were more likely to be on the edge of failing the exam. But, paradoxically, when focusing exclusively on those students whose scores hovered near 65, teachers gave extra points more frequently to white and Asian students.

This unequal advantage might be explained by other research suggesting that white teachers often have lower expectations of students of color— which may make some graders less likely to give them the benefit of the doubt.

Overall, the authors conclude, “The fraction of students in our sample graduating from high school would have decreased from 76.6 percent to 75.3 percent without test score manipulation.”

That amounts to tens of thousands of students who graduated because of the grade changing.

But once New York tightened up on its scoring system in 2013 — so that it was centralized and that borderline exams were no longer re-graded — “manipulation appears to have been completely eliminated.”

Students whose scores were changed were 21.9 percentage points more likely to graduate high school — a large, if not suprising, effect since the nature of the grade scrubbing was to get students over the graduation hurdle. These students also spent about an extra half-year in school as a result of a test score bump, as opposed to simply dropping out.

Some students, however, didn’t fare better. For them, altering their score made it less likely that they would graduate with New York’s more rigorous advanced Regents diploma.

“Students on the margin of dropping out are ‘helped’ by test score manipulation because they are not forced to retake a class that may lead them to leave high school,” the researchers said. “Conversely students on the margin of an Advanced Regents diploma may be hurt by test score manipulation because they are not pushed to re-learn the introductory material or re-take the introductory class that the more advanced coursework requires.”

In other words, failing a test and retaking a class may be harmful to more academically advanced students but helpful for those who might otherwise get no diploma at all.

These results stand apart from research showing the negative effects on elementary school students in Atlanta whose teachers cheated. The Georgia scandal may have been more damaging to students because those teachers fudged exam scores to boost their schools’ ratings and get bonuses — not to get their students through high school.

The researchers looked to see whether New York City high schools facing accountability pressure or those with a shot at performance bonuses were more likely to bump up students’ Regents scores. There was no evidence of that, suggesting teachers’ motivations were altruistic.

Grade scrubbing was also common on separate exams — in chemistry, physics, and advanced math — that weren’t necessary for graduation but led to an advanced diploma.

“It seems that the manipulation of test scores may have simply been a widespread ‘cultural norm’ among New York high schools, in which students were often spared any sanctions involved with failing exams, including retaking the test or being ineligible for a more advanced high school diploma,” researchers said.

This study doesn’t show whether graduation tests are a good or bad idea, but does contribute interesting data to the ongoing debate over how America should set standards for high school graduation.

“There are some things our study can’t tell us, which are very important,” said Rockoff, of Columbia.

The study does confirm that it can be quite easy to reduce or even eliminate test-score manipulation, suggesting it’s not an inevitable part of the system.

The paper also brings home that these exams really are high stakes — one or two points on a single test can determine whether a student graduates high school, which may have lifelong implications.³

“The downside of having these high-stakes cutoffs is that in the end, we treat students on either side of the cut-off very differently,” said Rockoff.

The impetus behind high school graduation tests was to raise standards, incentivize student effort, and make sure that diplomas “meant something” to employers and colleges. And maybe the do-or-die consequences inherent in these exit exams would be justified if they had produced the intended effect, but research to date has overwhelmingly found otherwise. The existence of graduation tests doesn’t seem to boost student achievement, increase motivation, or improve the earnings of high school graduates.

However, tough exit exams carry a number of negative ramifications — felt more by disadvantaged students — including more dropouts, greater achievement gaps, and even higher incarceration rates.

This suggests that teachers manipulating tests were reacting to a system that was harming their students. New York’s response was to change scoring procedures, but perhaps the exit exam system in New York and elsewhere needs a more fundamental rethinking.

---