A number of high school math textbooks claiming to be aligned with the Common Core standards aren’t, according to an analysis by EdReports, a non-profit that bills itself as the “consumer reports” of instructional materials in schools.
In a review of five high school math textbook series — it previously issued reports on K-8 materials — the organization found only one that merited high marks on all criteria; another that partially met expectations on two standards, and three others that fell short of expectations.
The group says it plans to release additional findings about nine other high school math textbook series over the coming months.
EdReports was formed in 2014 in response to anecdotal and empirical evidence of a misalignment between many curricula and the new standards, despite textbook labels boldly proclaiming otherwise. Research has found that the quality of instructional materials can have significant effects on student achievement — found in one study “to rival those that are associated with differences in teacher effectiveness,” say researchers Matthew Chingos and Grover Whitehurst — but there is often little information about which materials are most effective or even which are being used schools and district. Surveys of teachers reveal that many spend significant time sorting through instructional material and creating their own.
With funding from the Gates Foundation and others, EdReports initially released reports last year on elementary and middle school math materials, finding mixed results. Its latest review focuses on five series of high school math textbooks.
EdReports enlists and trains educators to score instructional material, including textbooks. Scorers first rate “focus and coherence,” meaning the extent to which the material covers the standards and whether content is connected so that students can understand the relationship between different concepts. For example, coherent instructional materials frequently return to previously learned skills and explain their relationship to new content.
The group uses what it calls a “gateway” approach, meaning that textbook series scoring poorly on the focus and coherence indicators aren’t judged on any of the other criteria. Such was the fate for publishers of three of the recently reviewed series: Houghton Mifflin Harcourt, Pearson, and the College Board.
Eric Hirsch, executive director of EdReports, said the approach was based on feedback from teachers. “Our educators who designed the [evaluation] tool said, ‘If it’s not covering the right math, talking about how usable it is is not constructive.’”
Textbooks that meet or partially meet the first benchmark are then judged based on whether content is appropriately rigorous and based on certain mathematical practices. Finally, material meeting this standard is evaluated on the extent to which it is easy to use for teachers and students.
The one math series to make it to the usability gateway, receiving high marks on all three main criteria, was produced by the nonprofit publishing company CPM Educational Program.
EdReports has received criticism from textbook publishers, as well as the National Council of Teachers of Mathematics. A letter from the latter group in response to EdReports’ initial ratings said the use of gateways was inappropriate and that there were several errors in the analyses.
Hirsch said his group had made changes based on this and other feedback, including providing additional ratings for material that partially met certain gateways. The elementary and middle school curriculum ratings were updated after criticism was made.
According to Hirsch, EdReports works with publishers to obtain and fairly evaluate the material. In addition to their evaluation, EdReports includes responses from publishers on its website.
Four of the five publishers of the newly rated series offered such replies. Pearson called the report “incomplete, invalid, and unreliable,” but did not point to specifics issues. The College Board provided a technical point-by-point reply, saying its series was created “with a focus on developing conceptual understanding through rich problem solving opportunities and embedded student-centered learning strategies.” Carnegie Learning criticized EdReports as “provid[ing] contradictory comments and ratings and inconsistent use of the full scope of available resources.“
On the other hand, CPM, which received the most favorable review, wrote, “We are pleased that the EdReports review … validates that our courses meet all expectations for alignment to the [Common Core math standards] for high school!”