New York Times Misuses Their Data When Linking School Quality to Home Prices, Researchers Say

The March 30 piece attempts to explore the link between home prices and school performance. It relies on extensive data compiled by Sean Reardon of Stanford University and Andrew Ho of Harvard University, both of whom say the newspaper’s approach is off base because it evaluates schools on absolute test score performance, which is influenced by many factors outside of the schools’ control.

“It’s written from a very naive … perspective of what school quality is,” Ho told The 74. Similarly, in a comment on the article, Reardon criticizes it as making “a common, but significant, error.”

The article, produced by the Times’ analysis and data visualization team, The Upshot, maps housing prices in five large metropolitan areas, including New York City and Chicago, comparing those figures to the achievement of students in the neighboring school districts. Places with higher test scores are described as having “better schools.” The title of the piece — “Good Schools, Affordable Homes: Finding Suburban Sweet Spots” — suggests that parents might be able use this graph to figure out how to get a good education at a bargain price.

“There are many factors in a home price, of course, but economists have estimated that within suburban neighborhoods, a 5 percent improvement in test scores can raise prices by 2.5 percent,” the article states. “But there are some districts that break this pattern: schools that deliver on quality with homes that are relatively cheap.”

To measure what it describes as “school quality,” the reporters rely on information from the Stanford Education Data Archive, which has created a national measure of comparable student performance for school districts across the country. Reardon is the director of the initiative and Ho is a member of the project team.

The issue with the Times report is with the use of the data: the article uses a raw measure of student test scores to pass judgment on schools. Researchers have long warned that this is simply not an accurate way to evaluate school quality.

“To attribute test scores solely to ‘school quality’ ignores the powerful role that family background plays in shaping opportunity,” Reardon writes in his comment on the Times story. Research has found that although schools are important, out-of-school factors, including poverty, have an even greater influence on student achievement levels.

A presentation by Reardon notes, “Average test scores differences are not solely the result of differences in schools; they are the total result of children’s home, neighborhood, pre-school, after-school, and K-12 schooling experiences.”

Reardon also says that although test scores offer important information, other metrics of quality are also important for evaluating schools.

Reardon said that he spoke at length with one of the Times’ reporters for their story, but he declined to discuss the contents of the conversation. A newspaper spokesperson did not immediately respond to a request seeking comment.

Reardon points to a Times story from a year ago highlighting his data that acknowledge the relationship between out-of-school factors and student achievement.

“Children in the school districts with the highest concentrations of poverty score an average of more than four grade levels below children in the richest districts,” the article said.

Some, but hardly all, of that gap can likely be explained by differences in school or teacher quality.

The misconception in the latest Times piece is common and extends to other journalists, policymakers, parents, real estate agents, and websites that rate schools. It can have profound consequences, the researchers point out.

For one, it can lead to skewed judgments about specific districts. The Times visual, for instance, suggests that because students perform below grade level on average, Chicago schools are worse than most of the surrounding suburban districts. Secretary of Education Betsy DeVos recently called out the city’s schools, saying, “We cannot pretend that Chicago’s education is ‘above average’ for the tens of thousands of students being left behind.”

But when looking at how much students grow academically from year to year — a measure Ho says does a better, although still imperfect, job of capturing school district quality — the same data show Chicago performing quite well, coming out ahead of a number of more affluent nearby areas.

Reardon also worries that this mismeasurement of schools can exacerbate segregation, as families perceive the “good” schools as being ones that have a lot of kids from rich families, although he says there is little research on this specific question.

“People are willing to pay for districts with higher test scores, and since not everyone can afford to pay the same amount, that would lead to some increase in residential segregation by income,” he told The 74, “so highlighting test score levels without acknowledging that they are poor measures of school quality might lead to increased segregation by income.”

There has been a significant increase in school segregation by socioeconomic status, much of it driven by affluent parents buying homes in neighborhoods zoned for schools considered desirable.

Moreover, Ho says there is an important lesson for designing school accountability systems. “We’d rather hold schools accountable for what they have control over,” he said.

Ho said that he is encouraged that under the new federal K-12 education law, the Every Student Succeeds Act, many schools are starting to incorporate measures of school performance other than proficiency rates, including test score growth. As The 74 previously reported, most states are using a combination of growth and proficiency measures to evaluate schools, suggesting performance will still be associated with student poverty — but likely to a lesser degree than under ESSA’s predecessor, No Child Left Behind.