74 Interview: First Teacher of the Year From a Charter School Talks Social Justice, Trauma, and Accountability

New Videos, Photos, and Reactions: DeVos and Weingarten Visit Ohio School Together

Weekend Education Reads: 11 Important Stories on Schools, Students, and Policy You May Have Missed This Week

Darryl Adams Got iPads, Wi-Fi for Every Student in High-Needs Calif. District. Here’s What He’s Up to Now

Arne Duncan Says Don’t Believe the Hype on SIG Failure; Obama Program Has Lessons for DeVos

Inside the Supreme Court: Are Justices Ready to Side With Church in Case About Playground & State Funding?

Bill Clinton Highlights Undocumented Students, Diversity in Speech at America’s Promise Summit

L.A. School Board Puts Money & Lobbying Muscle Behind State Bills Drafted to Shackle Charter Schools

Nevada Bill Seeks Cameras in Special Ed Classrooms, Just as Texas Reworks Its First-in-the-Nation Law

New Study: 3 Ways to Tell If a Charter School Will Struggle Before It Even Opens Its Doors

Trinity Lutheran v. Comer: 7 Things to Know About a SCOTUS Preschool Case With Big School Choice Implications

L.A. School Board to Vote Tuesday on Whether to Support State ‘Charter Killer’ Legislation

A Form of Professional Development That Research Shows Might Actually Help Teachers: Coaching

True to Their (Old) School: 2 Minnesota Farmers Just Paid $35,000 to Save Shuttered K-12 Building

Colin Powell on Education, Opportunity & Immigrant Students Ahead of America’s Promise 20th-Anniversary Gala

Opt Out at Florida’s Supreme Court: How a Fight Over a Third-Grade Test Is Pitting Parents vs. the State

Meet the 9 New Staffers Tapped to Fill Key Roles in Secretary Betsy DeVos’s Department of Education

Boston Schools Have Vowed to Combat ‘Racist’ Maps. Experts Want a Better Geography Curriculum

Weekend Education Reads: 14 Important Stories on Schools, Students, and Policy You May Have Missed This Week

‘Bathroom Bills’ Beyond North Carolina: Here’s Where 10 Other States Stand on Public School Legislation

New York Times Misuses Their Data When Linking School Quality to Home Prices, Researchers Say

April 4, 2017

Talking Points

.@nytimes story on housing prices/school quality misuses data, say researchers who compiled it

Sign Up for Our Newsletter

Judging schools is not as simple as a recent New York Times article suggests, say two researchers whose work was central to the Times’ analysis.
The March 30 piece attempts to explore the link between home prices and school performance. It relies on extensive data compiled by Sean Reardon of Stanford University and Andrew Ho of Harvard University, both of whom say the newspaper’s approach is off base because it evaluates schools on absolute test score performance, which is influenced by many factors outside of the schools’ control.
“It’s written from a very naive … perspective of what school quality is,” Ho told The 74. Similarly, in a comment on the article, Reardon criticizes it as making “a common, but significant, error.”
The article, produced by the Times’ analysis and data visualization team, The Upshot, maps housing prices in five large metropolitan areas, including New York City and Chicago, comparing those figures to the achievement of students in the neighboring school districts. Places with higher test scores are described as having “better schools.” The title of the piece — “Good Schools, Affordable Homes: Finding Suburban Sweet Spots” — suggests that parents might be able use this graph to figure out how to get a good education at a bargain price.
“There are many factors in a home price, of course, but economists have estimated that within suburban neighborhoods, a 5 percent improvement in test scores can raise prices by 2.5 percent,” the article states. “But there are some districts that break this pattern: schools that deliver on quality with homes that are relatively cheap.”
To measure what it describes as “school quality,” the reporters rely on information from the Stanford Education Data Archive, which has created a national measure of comparable student performance for school districts across the country. Reardon is the director of the initiative and Ho is a member of the project team.
The issue with the Times report is with the use of the data: the article uses a raw measure of student test scores to pass judgment on schools. Researchers have long warned that this is simply not an accurate way to evaluate school quality.
“To attribute test scores solely to ‘school quality’ ignores the powerful role that family background plays in shaping opportunity,” Reardon writes in his comment on the Times story. Research has found that although schools are important, out-of-school factors, including poverty, have an even greater influence on student achievement levels.
A presentation by Reardon notes, “Average test scores differences are not solely the result of differences in schools; they are the total result of children’s home, neighborhood, pre-school, after-school, and K-12 schooling experiences.”
Reardon also says that although test scores offer important information, other metrics of quality are also important for evaluating schools.
Reardon said that he spoke at length with one of the Times’ reporters for their story, but he declined to discuss the contents of the conversation. A newspaper spokesperson did not immediately respond to a request seeking comment.
Reardon points to a Times story from a year ago highlighting his data that acknowledge the relationship between out-of-school factors and student achievement.
“Children in the school districts with the highest concentrations of poverty score an average of more than four grade levels below children in the richest districts,” the article said.
Some, but hardly all, of that gap can likely be explained by differences in school or teacher quality.
The misconception in the latest Times piece is common and extends to other journalists, policymakers, parents, real estate agents, and websites that rate schools. It can have profound consequences, the researchers point out.
For one, it can lead to skewed judgments about specific districts. The Times visual, for instance, suggests that because students perform below grade level on average, Chicago schools are worse than most of the surrounding suburban districts. Secretary of Education Betsy DeVos recently called out the city’s schools, saying, “We cannot pretend that Chicago’s education is ‘above average’ for the tens of thousands of students being left behind.”
But when looking at how much students grow academically from year to year — a measure Ho says does a better, although still imperfect, job of capturing school district quality — the same data show Chicago performing quite well, coming out ahead of a number of more affluent nearby areas.
Reardon also worries that this mismeasurement of schools can exacerbate segregation, as families perceive the “good” schools as being ones that have a lot of kids from rich families, although he says there is little research on this specific question.
“People are willing to pay for districts with higher test scores, and since not everyone can afford to pay the same amount, that would lead to some increase in residential segregation by income,” he told The 74, “so highlighting test score levels without acknowledging that they are poor measures of school quality might lead to increased segregation by income.”
There has been a significant increase in school segregation by socioeconomic status, much of it driven by affluent parents buying homes in neighborhoods zoned for schools considered desirable.
Moreover, Ho says there is an important lesson for designing school accountability systems. “We’d rather hold schools accountable for what they have control over,” he said.
Ho said that he is encouraged that under the new federal K-12 education law, the Every Student Succeeds Act, many schools are starting to incorporate measures of school performance other than proficiency rates, including test score growth. As The 74 previously reported, most states are using a combination of growth and proficiency measures to evaluate schools, suggesting performance will still be associated with student poverty — but likely to a lesser degree than under ESSA’s predecessor, No Child Left Behind.