Yes, Poor Students Get Worse Teachers, but That Doesn’t Explain Most of the Achievement Gap

By Matt Barnum | December 12, 2016

Two months ago, a study came out that should have shocked anyone paying attention to the education policy debate of the past few decades.

The research, released by the U.S. Department of Education, showed that in 26 districts across the country there was virtually no difference in the quality of teachers instructing low-income students and those teaching higher-income ones. The results do not suggest that all teachers are equal, but rather that differences in their quality are not closely tied to how well off their students are.

This remarkable finding cut at the heart of education reform, a movement designed in large part to close achievement and opportunity gaps between poor and affluent students. The study would suggest that a focus on teachers amounts to a fool’s errand because quality is already distributed fairly equally. If true, this would bolster the argument of many reform-movement skeptics that the achievement gap is caused mainly by out-of-school — rather than in-school — factors.

A Huffington Post article described the study as “Proof You Shouldn’t Blame Teachers for the Achievement Gap.”

Now, however, another study has come out arguing that there is a “teacher quality gap” between low-income students of color and their white, wealthier peers. This research examined the states of North Carolina and Washington and found that poor students of color are more likely to get less-effective teachers.

Readers may be getting research whiplash at this point. How to square the two studies? It’s complicated, but there likely is, in fact, a teacher quality gap — but one that the researchers say accounts for only a small fraction of the achievement gap.

Paper qualifications not always the best judge

Traditionally, researchers have focused on objective teacher qualifications — years of experience, certification status, test scores on teacher licensure exams, degrees held — to look at whether the poorest students got access to equally skilled teachers. By these measures, the answer is clearly no.

For instance, a 2002 study of New York state found that “Urban schools, in particular, have lesser-qualified teachers, and New York City stands out among urban areas. Low-income, low-achieving and nonwhite students, particularly those in urban areas, find themselves in classes with many of the least skilled teachers.” This was based on a number of factors, including experience, the type of college teachers attended, and their certification levels. Similarly, a 2005 report on North Carolina schools found that black students were significantly more likely to get novice teachers than white students.

However, researchers have also come out with a number of studies showing that observable characteristics, like experience and licensure scores, have only a moderate relationship to a teacher’s ability to raise scores, and other factors, like master’s degrees, have none.

What’s more, a number of charter schools have shown the ability to improve test scores with what might be described as “poorly qualified” teachers — that is, many who are inexperienced and lack conventional training. In New Orleans, a package of reforms that included a large expansion of charter schools led to big gains in student achievement, even as the teaching force became less “qualified” — that is, less experienced and less likely to be certified.

In other words, it isn’t clear whether paper qualifications provide good evidence that a teacher quality gap really exists.

Using students test scores to gauge effectiveness

The obvious solution, it would seem, is to use more direct measures, such as the complex statistical models (often called “value-added”) that attempt to isolate teachers’ contributions to student test scores.1

Indeed, that’s what a number of researchers have done, reaching the conclusion, by and large, that a noticeable gap in teacher quality exists, according to studies of New York City, Los Angeles, Florida and North Carolina, Washington, and an anonymous “large, urban school district” in the South.

The most recent study of North Carolina and Washington state found that teacher quality gaps exist and have generally persisted over time for students who qualified for subsidized lunch and for black, Hispanic, and Native American students. Teachers of those students were less experienced, scored lower on state licensure exams, and had lower value-added scores.

So why did the federal study from a few months ago, that took in 26 unnamed districts from across the country, come to such different results, showing basically no teacher quality gap?

Dan Goldhaber, co-author of the North Carolina and Washington study, released a brief trying to answer that very question.

The first answer is simple: Different studies focus on different places, and it’s surely true that some districts and states have bigger gaps than others. This is likely due to a number of policies, such as how equitably the schools are funded.

Another explanation is technical, based on how teachers’ value-added scores are calculated. In contrast to some past studies, the federal report attempted to control for “peer effects” — like having disruptive or low-achieving classmates — which other research has shown can have a big impact on test scores. The idea here is that teachers with large numbers of low-achieving students shouldn’t be penalized for that when their effectiveness is being measured, and teachers with high-achieving classes shouldn’t get an unfair advantage.

Goldhaber finds that using this method in Washington state has no effect on the teacher quality gap between advantaged and less-advantaged students in elementary schools — it’s still there, regardless of how you slice the data.

In Washington middle schools, though, this does make a difference: Controlling for peer effect in calculating teachers’ contribution to student test scores significantly shrinks the observed teacher quality gap. Goldhaber posits that this may be because tracking students based on academic achievement is more prevalent in middle school, meaning the peer effects may vary more from class to class.

Understating the gap by looking only within districts

Another explanation for the disparate findings is that the federal study looks for gaps solely within districts and not between them.

Take New York City as an example: One way of measuring disparities would be looking at schools in the city that serve wealthy students versus those that serve low-income kids; another method would be to compare schools in New York City to one in a wealthier suburban location, like Scarsdale. To really measure inequality, you would probably want to do both.

Goldhaber does just that in North Carolina and Washington, and finds in Washington the teacher quality gaps between districts are the main driver of inequity. This suggests that the federal study may understate the gap by just looking within specific districts.

Notably, even this leaves out one additional layer of potential inequity: teacher quality gaps between states, such as those between affluent Massachusetts and low-income Mississippi. Goldhaber says there is no research on this question, but since school funding disparities persist across states, it seems likely that teacher quality gaps do as well.

Teacher quality gaps probably exist, but it’s bigger than that

Phil Gleason, co-author of the federal study, and Jonah Rockoff, a researcher at Columbia University, both said they generally agreed with Goldhaber’s explanations, which make a strong case that when measured not just within districts, a teacher quality gap likely exists, at least in some states. Controlling for peer effects might reduce it, but adding variation across states might widen it.

Rockoff notes, “Even though there is variation across different geographic areas, almost invariably the data shows that less-advantaged students get less-effective teachers, even if the differences are not very large.”

In other words, the federal study should not, on its own, overturn decades of conventional wisdom on inequity in education.

Still, it should continue to spur conversation. Gleason, co-author of that study, argues — perhaps, surprisingly — that his results are not as far out of line with past research as they might seem at first glance.

"I don't think the differences in the estimates are all that meaningful,” he said.

His point is that even if you look at the studies showing a teacher quality gap, the disparity accounts for only a relatively small share of the achievement gap. For instance, one study found that “the allocation of teacher and school inputs at the high school level contributes only 4 percent to the achievement gap.”

That’s not nothing, but neither does it support some of the broad rhetoric that pervades the education debate. Achievement gaps are probably exacerbated — and could be reduced — by the quality of teachers, but they seem to be caused primarily by out-of-school factors, including poverty.

Goldhaber thinks disparities in teacher quality are important to address, but he said, “In no way do I think that closing teacher quality gaps would eliminate achievement gaps.”


1. There is also the issue, which is a whole other can of worms, of whether test scores truly measure teacher effectiveness — research suggests that the value-added is important but doesn’t capture the entirety of teacher quality. (return to story)

Submit a Letter to the Editor