News

To Test or Not to Test: As Tax Credit Scholarships Expand, Questions About Accountability and Outcomes

By Matt Barnum | May 4, 2017

Updated May 5
Correction: A previous version of this article stated that Nevada's tax credit scholarship program does not require testing; in fact, the state's Education Choice Scholarship program mandates that participating schools administer a national exam.
 
“Parents need to have information on how their child is doing,” Secretary of Education Betsy DeVos recently said at the Brookings Institution. “That is, I think, the first form and the best form of accountability. That information needs to be more broadly shared with those who would consider a choice for their child.”
Yet this vision — of families empowered with information — might contrast with DeVos’s push to expand tax credit scholarship programs, which offer tax breaks to individuals or corporations that donate money to fund private school scholarships.
An analysis by The 74 found that the majority of school choice tax credit scholarship initiatives — 12 of 21 — do not require participating private schools to administer any sort of standardized exam, potentially leaving families in the dark about how their students are progressing academically. In the handful of programs that do require some form of test, it is rarely the state exam, making comparisons to public schools difficult or impossible.
That also means that research on tax credit programs is scarce, since studies require the ability to compare students at private schools against those at public schools. Despite their proliferating in a number of states in recent years, only one, Florida’s, appears to have been studied to determine the direct impact of the scholarship on participating students’ academic performance. In that state, public and many private school students took the same national exam for several years.
This stands in contrast to school voucher programs, which more often require schools to administer state exams and have been extensively studied by researchers. Some state voucher programs dictate the need for external evaluations.
“If you’re not going to require anything, we’re not going to know anything about whether these programs worked or not,” said Doug Harris, a Tulane University professor who has done extensive research on school choice in New Orleans and has been a critic of DeVos’s approach to education reform. “When you put it that way, even the people who are somewhat supportive of the idea get a little squeamish.”
But supporters of publicly funded private school choice argue that requiring a state exam may drive away effective schools, and that test scores are poor measures of school quality.
“Since there is strong evidence that state testing mandates tend to have negative unintended consequences, such as narrowing the curriculum and distorting how schools teach kids, we’d rather that schools have the freedom to use the tests that are more closely aligned to their curriculum and give parents the freedom to choose the schools that work best for their kids,” said Jason Bedrick, of EdChoice, an Indianapolis-based group that backs voucher and tax credit programs.
The Trump administration has pledged to expand private school choice by some $20 billion, but details for doing so remain unclear. However, Politico previously reported that the administration hopes to use tax credits and quoted sources saying Trump’s plan would resemble legislation proposed by Florida Sen. Marco Rubio and Indiana Rep. Todd Rokita in 2013 to create a federal school tax credit. Their bill requires participating private schools to administer a national or state exam.
This would provide some information to families, but it could make it challenging for researchers to study the program if the tests private schools choose do not mirror the ones given in public schools.
If states are granted discretion to use federal dollars to expand existing tax credit programs, many may continue to be test-free.
Tax credits less regulated than vouchers
Using state-by-state data compiled and regularly updated by EdChoice, The 74 examined testing requirements in private school choice programs across the country. Of 21 tax credit scholarship programs, only eight require any sort of testing, and only one, Louisiana, mandates the state test.
Florida’s Tax Credit Scholarship, by far the largest in the country, with nearly 100,000 students, does require participating students to take some sort of exam, but the next biggest state programs, in Pennsylvania and Arizona, do not.


 


Tax credit programs provide generous tax breaks to encourage donations to third-party groups that in turn award scholarships to eligible students, often those who are low-income or attend a poorly performing school.
Individual tax credit and deduction programs, which offer tax breaks directly to families for approved educational expenses, often including private school tuition, rarely require tests. Of the eight state programs that provide a tax deduction directly to families, only one, Alabama, imposes student testing rules.
Voucher programs, which use public money to directly pay private school tuition, tend to have significantly more regulation. Of 23 programs, 15 have testing mandates, including eight that require the state test.1 Voucher systems that target students with disabilities tend not to have such rules, while programs focusing on low-income students or those in struggling schools usually mandate testing.
Although vouchers and tax credits are functionally quite similar, policymakers tend to see vouchers as needing more regulation because they involve direct use of taxpayer dollars.
This likely explains why the lion’s share of research on private school choice has focused on vouchers rather than tax credits. Recent studies of voucher programs in Indianapolis, Louisiana, and Ohio all show that students attending private schools do worse on state standardized tests compared with those attending public schools. State voucher laws sometimes require independent evaluations, including those in Indiana, Louisiana, and Washington, D.C.
David Figlio, a Northwestern University economist and author of the Ohio study, said that the key to examining school choice programs is making an apples-to-apples comparison with public schools.
That’s fairly easy when scholarship students have to sit for the state test, but much trickier if private schools choose a different exam. It’s essentially impossible when they are not required to give any standardized assessment.
Figlio also produced a series of reports on Florida’s tax credit scholarship, which only requires a national test. His study was able to take advantage of the fact that, in addition to the state exam, all Florida public schools had to use the Stanford Achievement Test, which also happened to be the most common assessment administered by Florida private schools. “The two ways you can make comparisons are if there are public school students taking a nationally norm-referenced test or private school students taking a state test,” Figlio said.
But when the state eliminated the Stanford exam in 2009 because of budget cuts, Figlio couldn’t continue making direct comparisons.2
For the years where he could compare, Figlio generally found that private schools and public schools in Florida performed comparably, with perhaps a slight advantage for students in private schools.
A 2013 study on D.C.’s voucher program was also able to use the same Stanford test as a point of comparison, but only because the researchers — rather than the schools — administered the exam themselves to both private and public school students.3 A recent study of the same program also used another national exam, known as the TerraNova, that researchers gave to both voucher recipients and non-recipients. In both studies, between 70 percent and 80 percent of students sat for the exam.
In the older study, researchers found some evidence that voucher students made gains in reading, but in the newer analysis students who received private school scholarships lost ground in math, compared with those attending public schools. The 2013 research found that voucher recipients were significantly more likely to graduate from high school, according to parent surveys. The more recent research showed that parents of D.C. scholarship students perceived their schools as being safer.
Bedrick, of EdChoice, says he’s fine with requiring schools to administer a national test, but not the state exam, even if it means that studying the program then becomes difficult.
“As researchers, we’d like the apples-to-apples comparisons that a single test provides,” he wrote in an email. “However, as policymakers, we also have to do what’s in the best interests of kids.”
“The main point of choice policies is to empower parents to choose schools based on the criteria that are important to them, not just to raise [specific] test scores,” Bedrick said, citing factors such as student-teacher ratio, curriculum, and college acceptances.
Weighing test scores and what your neighbors say
In DeVos’s vision, laid out at Brookings, informed parents can choose among private schools, traditional public schools, and charters equipped with information to make that decision.
Harris, of Tulane, said that lack of access to test scores could make it “harder” for families to shop around for the right school. Research, including Harris’s, suggests that parents do generally place significant value on a school’s test scores, but it may not be the biggest factor driving school preferences.
As a parent himself considering private schools, Harris says he’s never asked any of them for their test scores.
“Most of these decisions, and this is true public, private, or otherwise, are based more on reputation, on what your neighbors say,” he said.
But some research shows that information on test scores can make a difference in how families choose schools, and potentially can result in better choices.
One study, conducted in North Carolina’s Charlotte-Mecklenburg school district, examined the impact of providing parents with academic rankings of schools. The district has an extensive public school choice system, and researchers randomly gave some families clear data about schools’ test score performance. The information seemed to make a difference: Families that received it were significantly more likely to choose higher-scoring schools, and in turn, their children subsequently performed better on state exams. An important limitation of the study is that it does not look at outcomes beyond test scores, such as college attendance.
Parents participating in Arizona’s scholarship program — which currently doesn’t require any sort of testing — will likely find it difficult to judge potential schools by academic achievement. A 74 review of several of the state’s third-party scholarship-granting organizations found that none of them appeared to provide any publicly available information about participating schools’ performance.
Individual schools in Arizona sometimes boast of high performance on unspecified exams that are not comparable across different schools. Parents might turn to popular third-party websites like GreatSchools, but the site has significantly less information about private schools than public ones.
Of course, public schools are sometimes criticized on similar grounds: School report cards are often difficult to access or understand; states like California released a rating system that does not allow comparisons across schools; and a common metric of school performance, student proficiency on state tests, is seen as a poor measure of quality by many researchers. Some say that schools should be rated with straightforward letter grades, but others argue that this simplifies the inherently complex task of measuring school quality, often placing too much weight on standardized tests.
Still, it’s clear there is more information available about public schools than private.
Bedrick, again, points out that parents might not place significant value on test scores. A survey by EdChoice of why parents in Georgia chose to send their children to private school found: “Student performance on standardized test scores is one of the least important pieces of information upon which parents base their decision regarding the private school to which they send their children. Only 10.2 percent of the parents who completed the survey listed higher standardized test scores as one of their top five reasons why they chose a particular private school for their child.”
“We don’t think it is necessary to impose standardized tests,” Bedrick said. “If parents demand that information, schools will provide it.”
Harris disagrees: “I think that’s hooey … Schools themselves, even if they wanted to provide that kind of information, can’t provide that in a way that’s comparable across schools — you have to have a coordinating mechanism for information.”
Going with her gut
Melissa Merritt, of Inverness, Fla., has two sons attending private school using the state’s tax credit scholarship. She looked extensively for a school that was a good fit for the older child, hoping for small class sizes and a strong reading program, she told The 74, which was connected to her through Step Up for Students, the group that administers the state’s tax credit program.
At the school she ultimately chose, Solid Rock Christian Academy, Merritt and her son, who is adopted, first met the principal and the teacher her son would have.
“The teacher was amazing,” Merritt said. “The thing that really helps is that she’s also adopted children, so she knew the special needs that he had.”
She also appreciated that the school had a combined first- and second-grade class, which was a good fit for her son, who was struggling in reading but advanced in math.
To her, test scores were not a major factor. “That wasn’t really something I was concerned about,” she said. “As standardized testing goes, I’m not too big of a fan of it.”
But even if it had been important — which, Merritt said, is true for many other parents — she wouldn’t have been able to compare the performance of schools.
“Each school is rated differently,” she said. “It would be nice if all schools were rated equally and you were able to check that.”
Still, Merritt said, “It wasn’t something I was too terribly concerned with. I’m more [about] let’s visit the school, let’s talk to the staff, and I go with my gut.”
Merritt adopted her second son, who is now in kindergarten at Solid Rock Christian Academy and is also doing well there, she said.
Merritt estimates that she spent approximately 20 hours researching and visiting schools before making up her mind. Other families may not have the same time, resources, and wherewithal to shop as extensively for the right school.


Delivering results, private vs. public
As states have debated whether to create or expand school choice programs, many have wrestled with the question of whether to mandate state exams or any exam at all. A proposed voucher program in Tennessee has featured significant debate on whether participating students should have to take the same exam given at public schools or a national exam of their choosing.
In Arizona — both heralded and condemned for its lightly regulated school choice environment — a recently expanded school choice program now requires private schools with 50 or more students enrolled through the initiative to administer and publicize standardized test results.
One Arizona senator who backed the expansion said time was needed to “let this program do its thing and prove to us that the private schools can deliver superior results from this experiment."
It’s not clear that will be possible, though: The new legislation does not require participating schools to take the same exam as public schools, and a recent investigation by The Arizona Republic found it difficult to obtain basic data — such as which schools are receiving public dollars or how students are doing academically — about the existing program.
It’s unknown where precisely DeVos stands on the topic, though her role at the American Federation for Children indicates she might tread a middle path. Model school choice legislation from the group requires participating schools to administer either a national or state test and report the results back to the state. AFC’s model laws also mandate reporting of parental satisfaction and students’ high school graduation, college enrollment, and college graduation.
Echoing DeVos’s message, AFC states, “Clear and consistent information about the academic performance of participating students will help empower parents and will also provide the public and policymakers with the information they need to evaluate the effectiveness of the program and participating schools.”
Still, the group has repeatedly promoted tax credit programs in Rhode Island, Pennsylvania, Iowa, and Oklahoma that don’t include any testing requirements.
A spokesperson for the U.S. Department of Education did not respond to a request for comment.
DeVos has frequently cited the test score performance of American students — the vast majority of whom attend public schools — as an argument for expanding school choice.
“I’m not sure how [American schools] could get a lot worse on a nationwide basis than they are today,” she said at the Brookings Institution. “I mean, the fact that our PISA [Programme for International Student Assessment] scores have continued to deteriorate as compared to the rest of the world and that we’ve seen stagnant at best results with the NAEP [National Assessment of Educational Progress] scores over the years. I’m not sure we can deteriorate a whole lot.”
Figlio, the Northwestern professor, says that criticizing the performance of public schools while not requiring private schools funded with public dollars to take any sort of exams smacks of hypocrisy.
“If you’re going to make the argument that we need a voucher program because the public schools are failing our kids and the argument is based on test scores, then it seems like the most honest thing to do would be to have public schools and private schools taking the same test.”
Disclosure: The Dick & Betsy DeVos Family Foundation previously provided financial support to The 74.


Footnotes:

1. Programs that have not yet been implemented, have been halted because of court challenge, or have no participant data are not counted. (Return to story)

2. Figlio was able to make estimates of the impacts of receiving a scholarship on student performance relative to similar students who didn’t get a scholarship in his school years 2009–10 and 2010–11 reports because past data allowed for a comparison of the Stanford test and the state exam, showing they were highly correlated. But by 2011–12 the state test had changed and this was no longer possible, Figlio said in that year’s report. (Return to story)

3. The study says in a footnote: “The research team administered the tests to both the treatment- and control-group students because many of the private schools participating in the program did not administer the SAT-9 themselves, and the team wanted to ensure that the conditions of test administration were consistent across the two comparison groups. This decision proved to be crucial when DCPS changed its accountability test from the SAT-9 to the DC CAS during the second year of the study.” (Return to story)

Submit a Letter to the Editor