WATCH: 4,000 Kids Take Over NYC’s Radio City Music Hall With America’s Biggest Academic Pep Rally

Where Education Research, Politics & Policy Intersect: 3 States Reveal How Data Help Shape Their ESSA Plans

Tennessee Bets Big on Personalized Learning, Launching Pilot Program & Eyeing Big 2020 Goals

MUST-SEE: School Films Epic 12-Minute ‘Trolls’ Music Video to Lift Spirits of Sick 5-Year-Old Girl

In Uniting Students With Prospective Employers, the Whether Job Search App Wins SXSWedu Tech Competition

More HS Students Are Graduating, but These Key Indicators Prove Those Diplomas Are Worth Less Than Ever

Race & Class: Chicago Schools Sue State, Claim Minority Kids See 78 Cents Per Dollar Sent to White Schools

KIPP v. UFT: Charter Network Sues Union, Arguing It Doesn’t Represent School’s Teachers

Supreme Court Sets New Standard for Special Ed, Unanimously Rejects Minimal School Progress

D.C. Approves ESSA Accountability Plan That Emphasizes Testing Standards & Transparency

Q&A: Why Miami’s Superintendent, Once an Undocumented Immigrant, Is Banning ICE From His Schools

No Warrant, No Way In: NYC to Prevent ICE Agents From Entering Schools Without Warrants

In California, a New Push for Teacher Tenure Reform as State Bill Would Extend Probation to 3 Years

VIDEO REPLAY: The 74 Moderates a Conversation on America’s Misleading Charter School Narrative

In Pitching School Chiefs on ESSA Plans, 2 Congressmen See Wildly Different Futures for U.S. Education

Effective, Efficient — and Limited: DeVos Lays Out Her Vision of Federal Role to State Schools Chiefs

The Contender: Nick Melvoin’s Plan for Combating Misinformation — and Unseating LA’s School Board President

Is a Maryland Teachers Union Organizer Shepherding Weak ESSA Accountability Bill?

Principal of the Day: This North Carolina School Leader Makes House Calls to ALL 990 Students

Baltimore’s ‘Happy Teacher Revolution’: Educators Band Together to Help Each Other Thrive in the Classroom

Exclusive: Education Cities and GreatSchools to Admit Flaw in Statewide Rankings of School Inequality

March 29, 2016

Talking Points

Groups acknowledge state rankings off in just released measure of school inequality

Sign Up for Our Newsletter

The 74 has learned that Education Cities and GreatSchools will issue a clarification saying the index it released last week ranking the school inequality gap in 100 cities and 35 states was faulty in its state comparisons.
The groups’ Education Equality Index, described as “a first-of-its-kind tool measuring the achievement gap” painted a picture of yawning  and persistent inequality across the country — tempered by a few bright spots — and was covered by media outlets from the Huffington Post to Chalkbeat Colorado to local news stations. Those reports tended to focus on places where the index showed the widest achievement gap between affluent and low-income students or the states and cities making progress closing that divide, according to the index.
Scores were given for each state, based on comparing test scores of low-income students to all students, and states were ranked, but soon a puzzling anomaly was pointed out on social media: States with higher poverty rates scored surprisingly well. In fact, the higher the poverty, the better the equality score.

Education Cities and GreatSchools, two nonprofits dedicated to improving schools, will release a statement today saying the formula it used to rank states was imprecise and cautioning others from following suit.

“Education Cities and GreatSchools have identified limitations in the interpretation of state-level Education Equality Index (EEI) scores. ... We are confident that school-level and city-level EEI scores are highlighting success stories across the nation, but we have concluded that the state-level EEI scores are not the best way to compare states. Because states’ absolute EEI scores are highly correlated to the percentage of students in the state who qualify for free and reduced-price lunch, we have removed the rankings of states based on the EEI score and pace of change pending further review,” the groups will say in a joint press release.
Matt Chingos, a senior fellow at the Urban Institute, said the flaw in the index resulted from a faulty comparison. Specifically, the groups’ methodology contrasted students who received free and reduced-price lunch (a common measure of low income) to all students in the same state. That means states with more low-income students looked better because their students’ average proficiency on standardized tests was lower, since poverty and weaker academic achievement are linked.
The website will still feature state level scores, but will not explicitly rank them and will caution readers against doing so.
Ethan Gray, the president of Education Cities, said in an interview that the index has been developed over the course of about a year-and-a-half in order “to have a better way of understanding progress that’s being made across cities around the countries.”
Gray acknowledged limitations in the methodology, and said this afternoon’s clarification was designed to ensure the measures were used correctly and that the report’s main message — highlighting educational inequality, as well as schools and states closing achievement gaps — still held value.
“We don’t think that the Education Equality Index is a replacement for other measures of quality. It’s only looking at student proficiency data, so there are some limitations to what it can tell you,” Gray said. “But we think it’s important that all the data is in one place, that there’s a methodology that allows for school comparison and city comparison across states, and that it provides some really important, high-level direction indicators of progress.”
Three researchers consulted by The 74 were generally skeptical that the methodology could be used to contrast schools and cities across different states, but said it could still hold value when looking within a given state.
Chingos said that although the index might be valid for comparing equality of schools or cities in the same state, “comparisons across states will be wrong/misleading” for the same reasons the state-versus-state rankings were inaccurate.
Dan Goldhaber, a researcher at the University of Washington Bothell, said in an email, “I think [the methodology] is probably ok within state, but … not across states.”
Steve Glazerman, a senior fellow at the policy group Mathematica, said, “I truly admire what Education Cities is trying to do and want them to succeed.”  
Glazerman’s research has found that low-income students do not have equal access to high quality teachers, with significant variation across districts. He emphasized that comparing educational quality across different contexts is always a challenging endeavor for a variety of reasons, including varying proficiency standards and concerns with using free and reduced-price lunch status as a proxy for poverty, because it encompasses such a broad range of income levels.
“It’s great that they put the state level data online, but even a casual inspection of those data raise some red flags,” Glazerman said. “This puts the burden of proof on Education Cities to convince us that the index really means something for cities and schools. Maybe it does, but we don’t have the data to check at that level.”
Education Cities was not able to provide The 74 with an independent researcher who could vouch for their methodology, but Gray did address the question of whether the city and school level data held up.
He said that city scores were a comparison of a given city’s education equality relative to its state and “not a comparison of the absolute low-income student performance across states; but rather of how each is doing as compared to its state.” He said that although there was a positive but weak association between cities’ poverty rates and their index scores, it wasn’t nearly as high as the state correlation.
“We’re very confident in the data and we’re very confident in the methodology. In this particular instance, the implication of the methodology leaves you with a story that is not particularly clarifying when trying to compare states,” Gray said.
“This is not the only measure of quality that matters or an exhaustive measure of quality. But it does give us some important directional indications about schools and cities that are moving the needle.”
Disclosure: Education Cities, GreatSchools, and The 74 all receive funding from the Walton Family Foundation.