The Purge: California Leaves Researchers (and Policymakers) in the Dark by Gutting Education Data
;)
Untangle Your Mind!
Sign up for our free newsletter and start your day with clear-headed reporting on the latest topics in education.
This is the second in a three-part series examining California's approach to education data and school accountability. Part One surveyed how the state's skepticism of test-based accountability starts at the top with Gov. Jerry Brown, who successfully took on the federal government; Part Three will consider what the next era of accountability in California might look like under the new federal K–12 education law.
Everyone in California education policy circles knows it.
“Our governor has not been a fan of data,” said Ryan Smith, executive director of Education Trust — West, a civil rights and education group that generally backs tough school accountability standards.
Morgan Polikoff, a University of Southern California professor, echoed this sentiment, “California has an embarrassing relationship with educational data. … The governor has affirmatively stated that he doesn’t want to spend money on improving educational data systems.”
Indeed, it’s well understood in the education policy realm that California has bucked national trends by pausing school accountability and not requiring that teachers be evaluated by student test scores.
But what’s less discussed is how the removal of data systems in the state has made it hard for researchers to study California schools or for policymakers to understand and address educational challenges. Although it’s data for high-stakes accountability like teacher evaluation and school performance metrics that draws headlines, even data for low-stakes purposes, like academic research, is hard to come by in California.
California Gov. Jerry Brown’s philosophy can be best summed up by a line in a 2011 message after vetoing a school accountability system: “Adding more speedometers to a broken car won’t turn it into a high-performance machine. … The current fashion is to collect endless quantitative data to populate ever-changing indicators of performance to distinguish the educational ‘good’ from the education ‘bad.’”
He’s also criticized the federal government’s $300 million Race to the Top program as having “a pervasive technocratic bias and an uncritical faith in the power of social science.”
California hasn’t done away with data altogether — school level test scores are publicly reported and several large districts have worked together to create more robust data systems — but several researchers and advocates say they can’t fully judge the education policies of the most populous state in the country because of a lack of accessible data.
It’s a strange position for a state synonymous with being in the vanguard
“California is really behind compared to the rest of the nation when you think about data systems and the ability to use data to improve practice,” said Samantha Tran, of the nonprofit advocacy group Children Now.
Data becomes a four-letter word
The hostility to data runs deep. When Brown became governor for the second time in 2011 an array of education data systems have met the death knell of his veto pen.
In his first year back in office, Brown blocked a years-in-the-making teacher data system, forcing California to return $6 million in federal money. The extensive database would have given the state the ability to link individual teachers to student growth, as well as monitor trends in the teacher workforce. Brown said thanks, but no thanks to the free federal money, and the system is now defunct.
“I see education as a local responsibility. The data is there and the superintendents and the teachers and the principals and the school boards should make use of it,” Brown said at the time, apparently ignoring the state school boards association, which said otherwise.
Now, five years after Brown’s veto, the independent Legislative Analyst’s Office says such a database is crucial as the state faces a potential teacher shortage. The lack of a statewide system means on this issue, “Many questions legislators have cannot, in turn, be answered well or at all,” the office found.
“When we are facing a massive teacher shortage, which has been reported in California, you would think we would want the data on teachers, longitudinally over time, available to researchers to help California understand what might be good policy options,” said USC professor Katharine Strunk.
Also disappearing was the California Postsecondary Education Commission, an independent agency for higher education policy planning, research and analysis that had existed since 1973. Brown line-item vetoed funding for the program in 2011, saying he wanted to reduce costs and consolidate systems; the move saved $1.9 million from a $129 billion budget.
In 2015, Brown also stopped a bill to create a similar agency, with supporters saying there was a lack of data about and coordination within California’s sprawling higher education system that educates more than 238,000 students and employs more than 190,000 faculty and staff.
Brown has fought efforts to track chronic absenteeism in the state’s public schools prompted by multiple reports from the state Attorney General Kamala Harris finding that truancy was a widespread problem. In 2014, Brown vetoed two bills to require additional tracking of absenteeism by local schools and the state.
“These are missed opportunities to help keep California's youngest and most vulnerable students on track,” said Harris at the time. (State monitoring of chronic absenteeism finally started earlier this year after federal law began requiring it.)
One system that survived the governor’s cull is the California Longitudinal Pupil Achievement Data System — or CALPADS — though not for lack of trying on Brown’s part. Still, many say the database, which tracks a variety of student-level data including demographics and test scores, could be better.
“By statute, [CALPADs] comprises only the data that the federal government requires [California] to collect, so it’s not a comprehensive data system [and there are] a lot of weaknesses in it,” said David Plank, head of the research group Policy Analysis for California Education, or PACE.
In what might be the oddest instance of data skepticism, the California Department of Education, led by state Superintendent Tom Torlakson, deleted fifteen years’ worth of old test scores from an easily accessible part of its website just before the release of new Common Core-aligned assessment scores in August 2015.
A department spokesperson explained at the time that it was done “to avoid confusion because the two tests cannot be compared.” After a public uproar, the data was restored to the state website a few weeks later.
A request for comment from Brown’s office was directed to the state Board of Education. Julie White, the board’s director of the communications, wrote in an email, “Governor Brown’s comments about education data collection have been consistent over the years,” pointing to two veto messages from the governor.
“While well intentioned, the collection of data for the interests of faraway authorities would not get to the root of the issue — keeping kids in school and on track,” Brown wrote in 2014 when rejecting the bill to track truancy.
California: a data desert for researchers
Perhaps the most remarkable part of the state’s data purge is how difficult it has been for quantitative education researchers — those who use numbers to dissect the impacts of certain policies — to study the state.
In regards to Brown’s elimination of the teacher tracking database, Plank, of the Policy Analysis for California Education (PACE), said, “PACE works with a lot of academics, many of whom would love to have such a system.”
Speaking of the research community, Plank said, “We think we’ve got a lot to contribute here, but we don’t have the data. What we’ve found is that our academic colleagues are doing research in North Carolina or Florida or New York because they have data systems that work and we don’t.”
Plank said that a group of districts including Los Angeles and San Francisco with their own tracking systems have shared student-level test score data with researchers.
A 2012 report from the D.C.–based Data Quality Campaign found that California only met four of its 10 suggested “actions to ensure effective data use.” Since 2012, California has refused to respond to the Campaign’s survey on how different states use data.
Strunk of USC said the state’s decision to go a year without publicly reported state tests made it difficult to examine the state’s education policies. “I had a lot of studies looking at interventions in different districts, especially L.A., and we wanted to see longer-term outcomes and we’re cut [off] at two or three years because we just don’t have [2013–14] data,” she said.
She said even getting the most recent data — the first year of the new Common Core–aligned tests from the 2014–15 school year — has proved challenging. “Many researchers have not been able to access those data from different districts and from the state.”
Robert Oakes, a spokesperson for the California Department of Education, (CDE), originally said in an email, “CDE doesn’t release student-level data for privacy reasons.” However, Plank said that it is common practice to share data with researchers while protecting student privacy and that many states do so.
Oakes later said the department did, in fact, share student-level data with certain researchers.
“The CDE has selectively entered into data-sharing agreements for Smarter Balanced results with highly qualified researchers from different institutions over the past two years. … The CDE is not funded to review research proposals and data requests and therefore only accommodates these requests as time permits or does so as part of an already established work assignment or project.”
When asked how CDE could provide the student-level data to some in light of the privacy concerns mentioned earlier, Oakes said, “Specific situations vary, of course, but in general CDE removes any personally identifiable data that could violate privacy protections.”
Oakes said the department does make efforts to support the use of data: “We have a pretty sophisticated bank of data available through DataQuest, and researchers can fill out specific applications if they want it mined at different levels. … We’re really big on data; we have a lot available.”
Strunk agreed the state has great information accessible at the school level but that the key for researchers is to be able to track individual students and teachers over time.
“To get the best estimates of the impact of an intervention on student achievement, you'd like to have as fine-grained data as possible," she said.
For instance, Strunk described how she once attempted to estimate the rate of teacher turnover at different schools — but she wasn’t able to do so because the available data wasn’t detailed enough.
This isn’t necessarily the fault of the state education department, since some of the systems necessary were defunded. Strunk says that she’s worked with many state CDE staff members who are eager to help researchers but, on balance, “I've never found CDE to be particularly transparent or helpful in providing access to student-level data,” she said.
Like Plank, Strunk pointed out that those handful of states, such as North Carolina and New York, with strong data systems dominate the research scene.
“There is no [statewide] student-level data set available to researchers in California, at all,” she said.
It’s not just an academic point when dealing with a state as large, diverse and significant as California.
“There’s no reason to think that we can be able to apply what we learn from North Carolina to California,” said Strunk. “You can’t do much research on California.”
Polikoff of USC, said, “If you look at how many [research] publications there are that use student-level longitudinal data from California, there’s almost none.”
Michal Kurlaender, a professor at University of California, Davis, said she had been able to access some data by building strong working relationships with staff at the California Department of Education.
But, she said, “There is a lot of work being done and important research questions being explored in states that have rich, more easily accessible data systems … There hasn’t been enough [California] support to build rich data systems and to make them more publicly available for research purposes. … As a general rule, I do believe California is a tougher state for researchers to do work in because of the data accessibility issues.”
With growing calls for improved state data systems, but a popular governor who philosophically objects to quantifying education still in office, it’s unclear how long California’s data blackout will last. Regardless, it’s unlikely that the voice of researchers — and their desire for data — will be heard that loudly in Sacramento.
“The research community has no political clout,” Plank said.
Get stories like these delivered straight to your inbox. Sign up for The 74 Newsletter
