Q&A: New Orleans Education Expert Doug Harris Walks Through His Research Showing a Decade of Student Gains
This is one in a series of articles covering the 10th anniversary of Hurricane Katrina and the rebuilding of New Orleans’ schools. Read all our coverage and essays here – and be sure to watch our three-part documentary series about the past, present and future of New Orleans education.
The debate has raged for years about the effects of school reforms implemented in New Orleans in the wake of Hurricane Katrina, as the vast majority of traditional public schools were converted into charters independently operated by nonprofit organizations.
No one has studied these effects in greater depth than economist Doug Harris, who directs the Education Research Alliance of Tulane University. Harris has done pioneering research on the effects of New Orleans reforms on student outcomes, and I have previously summarized the top findings of his latest report, which was unveiled at a conference in May.
In mid-July, I talked with Professor Harris at length about his studies — including New Orleans’ gains in test scores, the change in the teacher workforce, and the challenges of someday replicating New Orleans’ reforms in other cities. We present the transcript below, lightly edited for length and clarity:
The Seventy Four: Let me start by asking — What were the New Orleans reforms exactly?
Professor Doug Harris: I view it as a package of changes. It wasn’t one thing. They were a combination of changes that all fit within a few basic principles. Really two, I’d say: accountability for schools and autonomy for schools. Let me flesh that out.
On the accountability side you have what economists would call “market accountability.” They got rid of the attendance zones that would normally determine which students would go to which schools, so that no schools are guaranteed students. The schools in a sense have to compete for students and so they’re accountable to the market in that way. Second thing is that the government, in allowing charter schools into the market, was holding them to contracts where if certain schools didn’t perform — if they didn’t generate high enough test scores — then the schools would be closed down. And a lot of them have been. There have been 46 schools in a city of only 90 schools that have either been shut down or taken over or merged with other schools. There’s a high degree of accountability there based on the scores.
The second principle is autonomy: letting the schools figure out how to reach those goals. The government was setting the bar in terms of test scores, but then letting the schools figure out how to get there. Schools had control over their budgets. The budgets were on a per pupil basis, so it’s more like a business — if you get more customers you get more money. They were then given the flexibility on how to spend that money. You also got rid of the union contract, which was a pretty substantial change. There was no longer any restriction for hiring, firing, or evaluation of teachers. That was another big part of the autonomy puzzle.
Are test scores the only metric that are used for accountability purposes?
Pretty close. At the high school level they also use high school graduation. But otherwise that’s pretty much it.
Did it “work”? That is, what evidence is there that New Orleans reforms worked or didn’t work in terms of improving student learning or other outcomes?
When I got down here the first thing that everybody talked about are the increases in outcomes. And if you look at pretty much any measure, there has been a substantial increase and even more importantly a substantial increase relative to the state as a whole. You worry with some of these measures that maybe the scales are changing over time or there are state policies that also have an influence separate from the reforms, so having some sort of a comparison group is helpful. No matter what measure you look at, New Orleans has improved compared to the state. Whether it’s the state test scores in grades 3–8, or the ACT, or high school graduation, or college entry — everything looks better in New Orleans. The problem is that there are a lot of different reasons why that might be other than the reforms.
I’ve spent most of the last year, year and a half, trying to untangle the reform effect from those trends. And I think we have, so I’ll give you a rundown of the things we’ve looked at.
One concern is that the population changed. The public housing projects were shut down for a while. The hardest-hit neighborhoods in terms of flooding were also the lowest-income neighborhoods and there’s more direct evidence that low-income populations did not come back as quickly. From that alone you’d expect the scores to go up because of the strong link between income and educational outcomes. It could be the increases are just reflecting that.
We did a couple different kinds of analyses to address that. One, we looked at the pre-Katrina characteristics of people who returned to see whether they seemed to be higher-performing in terms of scores or more advantaged in socioeconomic status. What we found was that at the beginning that argument holds up. At the very beginning, the pre-Katrina scores of the returnees were much higher than the non-returnees. But over time it actually came back quite close to the pre-Katrina average. So at the beginning you could make that argument that the populations change was playing a role. But we’re mostly interested in whether there was an effect of the reform as of, let’s say, 2012. By that point the population actually looked quite similar demographically.
The second thing we did was look at census data, and then we actually simulated using some other data the relationship between income and test scores. We could get that information from this other nationally representative federal data set and do some statistical analysis there. And then we can combine that with information about how income changed here relative to the surrounding districts. And when we do that we get almost exactly the same answer: basically it looks like New Orleans ends up slightly more advantaged socioeconomically or has a slight improvement in socioeconomics relative to the other districts. But it’s so small it couldn’t possibly explain the increases in outcomes that we see.
So that’s the first thing: the change in the populations.
The second thing is test-based accountability distortions. So here we mean things like teaching to the test, drilling test items, and doing things that will get the scores up without actually improving learning in any substantive way. This was a little harder to tease out, but there’s certainly good reason to think that it would be a more severe problem here than in other districts. One reason is that charter schools tend to be very test-driven to begin with — their ethos is built around generating results and having high expectations. The second reason is they were going to be shut down if they didn’t show results. It wasn’t just a belief in scores generically — it was that their livelihoods depended on the scores.
To get at that we did a few different things. One was to look at the scores based on the stakes. In Louisiana, during the time frame we were looking at, the science and social studies tests had lower stakes than math and English language arts tests. If it’s an incentive driven response, you’d expect then to see larger gains in math and language arts. But we don’t see that — the science and social studies effects are about the same as math and language arts. That was our first direct indication that it probably wasn’t teaching to the test.
We also did an analysis of what we call “bubble effects,” which refer to the idea that if students are right near proficiency then the schools might put all their chips in the pot for those students and give less attention to students who are a ways from the bar — who are either too far below to make it to the standard or so far above that schools don’t need to pay any attention to them anymore. And since the school performance score was, at least for a while, based more on the proficiency rating, you might expect that to happen. We also don’t see any evidence of that — at least it’s not any different in New Orleans than any other districts.
Those are the only two things we could do. It’s still possible — and I actually still suspect — that some of the effect is driven by test-based distortions. But one reason I’m still reasonably confident in what we’re saying is that we also looked at the state reports of things like high school graduation and college outcomes in New Orleans.
College outcomes are really the most important outcome for two reasons. One is: from the standpoint of long-term life outcomes for students, college is the best predictor — it’s a much better predictor than test scores. We use the test scores because it’s an immediate response and it’s something we can look at quickly.
When we look at college outcomes and high school graduation, those also look positive. The percentage of students who graduate from high school that are going on to college right away is increasing, at the same time the high school graduation rate is going up.
Are they both going up at the same magnitude as well?
Yeah — well, actually the fact that they’re even going in the same direction is remarkable. Normally if you have a higher graduation rate you expect a lower percentage of those students to then go on to college, because presumably the extra students graduating are lower performing students, who you might think wouldn’t go on to college. The magnitudes are similar, but it’s even more remarkable that they’re going in the same direction.
What about school discipline? One of the concerns, as you know, is that these schools are suspending, expelling more students, and it’s argued that they might be getting these positive outcomes but there’s some cost in how students are treated in schools.
We did do some analysis of that. We have the number of expulsions and the number of suspensions in particular and whether suspensions are in school or out of school. And we find that initially when the reforms were first put in place that there was an increase in the number of disciplinary incidents — which you could attribute to two different things. One is when people were coming back there was a lot of trauma and upheaval in their lives, and they may have just been more prone to misbehavior because of what was happening in life at the time. The second reason could be that the schools were being more strict. We can’t really disentangle those two things clearly. What we can say is that at this point in time, that the number of incidents is actually lower than it was pre-Katrina.
There are a couple possible reasons that might be. One is that the number of incidents is actually lower. And I think that might be true. But it might be true because of the strict discipline policies. The fact that there are fewer suspensions doesn’t mean that schools aren’t strict; it could mean that the strictness is leading to there being fewer incidents.
Has there been any work looking at whether there are correlations between schools in New Orleans that are strict and their achievement outcomes or their outcomes in terms of suspensions or anything like that?
We haven’t done anything like that in part because I don’t think you can really interpret those numbers very well. For one reason, the schools that are doing that, or that have particular discipline policies, also have particular ways of teaching and it becomes very difficult to separate out these factors.
Next question is about the money. Some people would say, “Well, there’s all this money put into New Orleans and the increase in money really is what explains the improved outcomes.”
It’s possible, and I would say it’s pretty difficult to isolate the role of any one of the changes, including what we think of as the reforms but also things like money.
I can describe how the money changed. At the beginning New Orleans was spending about $8,000 more per pupil relative to similar districts. In other words, spending didn’t quite double, but it came pretty close to doubling in the initial years. And then it converged back to the normal, or close to normal rate. Now they’re spending about $1,000 more per pupil than similar districts, whereas before the storm they were spending close to the same as those comparison districts. The numbers I just gave you exclude construction money.
So what do you make of the interpretation of this, that all New Orleans shows is that we need to invest more money into our urban school districts and that will spur the improvements that we’ve seen in New Orleans?
It’s pretty unlikely that that’s the explanation, only because when we look at evidence from other places we don’t see really strong relationships in other places that have rapidly increased spending without generating substantial increases in scores, which isn’t to say there’s no relationship. In fact, I think the increase in spending was almost certainly a necessary component of the reforms because to be able to attract people down here you needed to pay people well. And you needed to put in resources to get things going. I’m not sure that everybody would have come down and that they would have done as well with the money that was being spent before. I think probably every element of the reform package, including the change in spending, probably contributed in some fashion, but I think there’s not much reason to think that it was all about the money.
Last question on the money point. What do you make of the argument that well okay we know we can’t just throw money at the problem necessarily, but what New Orleans could have done was really invest in things like pre-K, lower class sizes, and higher salaries for veteran teachers who were still in the system and that doing that would have improved academic outcomes and would have been less disruptive to the community?
It would have less disruptive for sure. The existing system is quite different from what people were used to. It took a long time to create and get off the ground; it continues to involve disruption because you’re closing schools, and because there’s higher teacher turnover. But it’s not clear how much disruption should be a criteria. I think ultimately what we should be focusing on is whether the schools are better or not.
Again I would say it’s true that putting more money into the schools does have some positive effect, but again looking at past research there’s not any reason to think that just doing those things would have generated this kind of an effect. In fact, we compared the effects that we generated to things like early childhood education and smaller class sizes. I’ve also done work in the past comparing increases in teacher salaries to things like smaller class sizes.
There just isn’t evidence to support the idea that any of these policies would generate effects of this size. Put differently, I’ve never seen an effect of this size before, which isn’t to say it hasn’t happened. I’m always poking around, asking people, “Have you ever seen an effect of this size before?” And I’ve yet to find one. And that’s thinking about any reform or any change in policy, including when you look at the track record of spending more money and smaller classes. Pre-K definitely has more support, and I support early childhood education. But even that doesn’t appear like it would have generated this kind of effect.
Can you talk about the change in the teaching force and the demographics in the teaching force? Particularly the sort of paradox as I see it — that the teaching force has gotten whiter, less experienced, less credentialed, and that from past research we would have expected at least some marginal decreases in teacher effectiveness based on those facts.
It’s an important topic and one reason New Orleans is so important is that it touches on so many different parts of the school reform debate. It touches on finances, how we think of the teacher workforce, choice, test-based accountability — it’s all wrapped in one package.
Descriptively, you’re right. The percentage of teachers who are certified dropped by 20 percentage points; the percentage of teachers with 10 or more years of experience dropped by 20 percentage points; teacher turnover doubled. Certification, experience, low turnover are all typically thought of markers of a quality school and workforce.
I think there’s actually a fairly easy resolution to the paradox. One part of it is that you’re bringing in teachers from alternative certification programs and those programs have a pretty good track record, at least a decent track record. For example, you look at Teach For America [TFA] — the evidence seems to suggest that TFA teachers in terms of generating achievement are about as good as traditionally prepared teachers and in some cases maybe better. And that’s where a lot of these teachers were coming from. Not just TFA, but other programs like it.
Second part of the puzzle is that the school models here involve teachers working really long hours — 80 hour weeks are not uncommon for teachers down here. The traditional career path doesn’t allow that. Teachers start to have families in their twenties and thirties, and nobody’s going to work 80 hours a week in that situation unless they have absolutely no other choice. By bringing in teachers who were less experienced and who didn’t intend to make it a career, they can have this model of very long hours and that’s how they built it. If we can get these less experienced teachers working much longer hours than more experienced teachers, you’re kind of making up for the lack of experience in that other way.
Do you think that selection and deselection of teachers through the evaluation system — I don’t know if charter schools are using pay for performance models, etc. — contributed?
Yes. There were two other factors I was going to mention and that’s one of them. Let me give you the other one, which is that the city is small, with only 90 or so schools. This is not New York City or Miami or Los Angeles or Chicago. This is Memphis. It’s a small city. You had all these people — not just locally, but people from all over the country — who wanted to come to help rebuild New Orleans and help be part of this unprecedented reform movement. So you’ve got a lot of people interested in that and you had a very small number of openings and a very large supply of people going to fill those slots. Because the schools could be selective, it’s closer to where you were going — they could pick who they want, and they could fire people, and they’re going to have a long list of people waiting. That’s a condition that’s unlikely to hold in most other places. I think that is part of what’s going on here. I think it’s important to consider that when we think about what New Orleans means for the rest of the country.
REPLICATING NEW ORLEANS
So let’s talk about that! What does New Orleans mean for the rest of the country?
The way I look at it, first of all, the results I think are clearly positive almost no matter how you look at it. Even the side effects seem to be small or not there. It didn’t seem to increase segregation. It seemed to benefit all subgroups of students, including special education students, who some were especially worried about. It didn’t increase the number of disciplinary incidents; in fact the number of out-of-school suspensions is substantially lower now. A lot of good news here for New Orleans.
I think one thing that tells you is what’s possible with a reform like this. I tend to think what happened here is the ceiling of what’s possible with this kind of approach. If other places try it, probably they’re not going to see effects of this size, which doesn’t mean they shouldn’t do it. It just means they should view New Orleans as what might be possible.
Another thing that I think is relevant is the way it forces people to rethink what a school system can look like — not necessarily what it should look like. The first two years I was here I saw something every day where I just sort of twisted my head and puzzled over, because it was something I’d never seen before. You realize the whole system is just totally different from anything you’d ever seen before. Even separate from how well it’s worked, seeing something this different I think is healthy. It’s kind of like going to a foreign country and seeing that in China everybody rides a bike. So why doesn’t everybody ride a bike in the United States? You don’t really think about it that way until you go to another country and see how their system works.
On the more skeptical side, as I said earlier, I don’t think you’d see the same effects in other places because the conditions here were distinctive. You have a lot of people from around the country wanting to come here to help in a way that you’re not going to have in Detroit for example. It’s the center of school reform. So you have a lot of very talented people who want to be part of school reform who are going to come to New Orleans. If you want to build a tech company you go to Palo Alto. If you want to be involved in school reform you come to New Orleans, and that’s not going to be true in other places. There’s only one Silicon Valley.
I think those are the considerations. I don’t think you’d see the same effect, but it doesn’t mean there aren’t lessons here.
Imagine a hypothetical school district that’s an urban district that’s struggling — I understand that “hypothetical school district” might not be the best way to think about this — but a school district that’s struggling, does not have any charter schools or has a limited charter sector. And they come to you and they say, “We want to replicate the New Orleans reform. Should we do it?” What are the factors you would tell them to consider and how likely would you be to recommend the New Orleans reforms?
Let me answer that by adding one more thing to what I was saying earlier about the conditions of New Orleans. One of the conditions in New Orleans was it was a very low performing district. It was a disaster by almost any measure. When I first got here I thought some of that was just hyperbole because everyone kept saying it, but now I’ve seen more and more objective evidence of that. The FBI had an office within the school board because there was so much corruption to investigate that they needed to have offices within the school board offices. I just got my hands on a report after many years of trying that was the Council of Great City Schools evaluating the human resources management of the district. The litany of problems in the district was just appalling, the way it operated. The outcomes were also poor as a result. It was the second lowest performing district in the second lowest performing state in the country.
If you’re in a district that already functions pretty well, maybe this is not necessarily a good idea. Here you had a district that was not functioning well by any measure. I think people tend to assume that all urban districts are terrible because the scores are low but it really is true that a lot of that is driven by the demographics and the socioeconomics of the districts. That’s real. There are some very good schools in urban areas and there are some districts that probably do a pretty good job. You don’t want to throw that out just because of a probable misinterpretation, if they’re not taking into account where students are starting.
I would tell a district or somebody advocating it in a district to look very carefully at how the traditional district is operating before going down that path.
I would also tell them all the conditions, all the things they have to do to make it work. I think people tend to think, oh we just need more charter schools. But Detroit is a case study in how that goes wrong. Detroit has a lot of charter schools — I think it’s second or third in market share — but it’s terrible. The system has so little coordination.
One thing that’s different about New Orleans is you have two authorizers; in Michigan you’ve got dozens of authorizers. It’s a very uncoordinated system. It’s possible to have autonomy at the schools and still have coordination at the system level. One possible lesson of New Orleans is that just making it a free-for-all is probably not going to be the right approach. You do need to have an active government, but a government that’s active in different ways, in ways that are ensuring the system of schools works well, but not putting its thumb on every little thing the schools are doing.
What do you think the chief methodological critiques of your paper are going to be?
To my mind the biggest critique isn’t so much methodological. I don’t think the critique will really be about methodology, in the sense that I don’t think people will say we should have done something else. We’ve shown this paper to a lot of people and we’ve done everything that anyone’s suggested that made any sense, and we reached the same result.
It will be more about how we’re interpreting it. I think one critique will be the critique that we put out there ourselves, which is that we can’t totally account for all the different elements of test-based accountability and distortions that can come with that. We did our best with the data that we have, but unfortunately there’s no National Assessment of Education Progress [NAEP] test in New Orleans. It would have been wonderful if we had a pre-Katrina NAEP and a post-Katrina NAEP and this would all be a much simpler exercise. I think that’s probably the biggest question mark.
One other thing that will probably come up is that when we look in the very first few years, so just up to 2009 when the system was still getting off the ground, the effects are generally positive but they’re not statistically significant. Some have looked at that result and said “well, what about this result?” even though what we really care about is 2012, which is the most recent year that we’re studying. When you’re developing a new system from scratch you want to give it some time to develop and see what happens in the long run. What we see looking at the data is that schools get better year after year after year. The differences in 2009 are not statistically significant, but if you just wait to 2012, they’re always statistically significant. We’ve estimated hundreds of different ways and it’s very hard to make the 2012 effect go away.
Going back to the test-based accountability piece I would say in addition to what we can directly study in terms of high-stakes and low-stakes subjects the fact that we’re seeing in the state reports that high school graduation and college entry also going up at very high rates suggest that it can’t just be teaching to the test because that would not get students into college.
I still think the data are quite convincing. Honestly, I didn’t expect that at all. I thought I was going to end up with a really murky situation where there were population change and test-based accountability distortions, and all these other things that could have gone wrong, and that we would not be able to draw really clear conclusions. We sort of went down our checklist — and it was a very long checklist — and it was so consistently positive that I think it’s very hard to look at these results in a negative way.
Any other things you wanted to mention?
One other thing that’s worth bringing up is the process, as much as we are focusing on the results. There is a real concern in New Orleans about how the system was put in place and how it’s run. The fact that you have a much smaller share of teachers who are African-American now, and that came on the heels of taking the schools away from the local district, which was elected by a majority African-American city, and turning it over to the state. The state leaders are not anywhere near a majority African-American. There’s a lot of racial tension around that. I think people feel like, especially taking a longer view of history, that the African-American community worked for a century to have some control and authority over its schools, and that was taken away.
When you think of it that way it gives you a little bit of perspective. There is a bitterness, even among the families who have their kids in charter schools, who might even like their individual schools — they have concerns about how the system operates and how it was put in place to begin with.
Is there data about how parents or even just how members of the New Orleans community, view the schools, view the reforms, and view the effectiveness of the reforms?
There is survey evidence. The Cowen Institute has done surveys every couple of years. Generally those results have majority support and majority support both in the white community and the African-American community. There’s not universal support by any means.
But I don’t think the questions were getting at quite what I was describing either. They were along the lines of “Do you think the current system works?” If they had also asked, “Do you think it’s good that a handful of people, while everyone was evacuated, made this decision to run the schools this way?” Almost everybody would say, no, that was not a good thing.
I think the process matters. One of the sessions from our conference had Howard Fuller in the discussion. This is a guy who has been a strong supporter of this kind of reform for much of his adult life, but who is deeply concerned about the process and the way the reforms are perceived in the local community.
As much as I’m about evidence, it’s not all about evidence and we have to keep that in perspective.
A 74 Documentary | Part I: Reopening in the Flood Zone
Part II: The Class of 2015
Part III: The Next Generation
Get stories like these delivered straight to your inbox. Sign up for The 74 Newsletter