I can guarantee that the NAEP results — regardless of what the actual data are — will be used by commentators to reinforce their previously held policies positions. That people will use the same data to reach opposite conclusion is an indication that we shouldn’t read too much into said data.
Advocates will surely declare “[State X, which had ‘good’ results] did [Policy Y, which I already like]; therefore everyone should do [Policy Y].” If scores show improvement reformers will say, “This shows our policies are working — full speed ahead!” If there aren’t improvement reformers will say, “This shows why our schools are in desperate need of reform — full speed ahead!”
Similarly reform skeptics will gleefully point to disappointing results as evidence that reform policies are failing. But if scores rise, they will declare that NAEP scores shouldn’t be taken seriously and that tests don’t much matter.
People believe what they believe; NAEP scores won’t — and frankly shouldn’t — change this. But can we just drop the charade?
This is not to say that NAEP scores are useless. They are genuinely important indicators about whether students across the country are learning more math and reading than past students. And although raw data cannot be used to judge specific policies or policymakers, it is absolutely reasonable to make hypotheses about policy that can then be tested rigorously.
In turn, NAEP scores have been used by researchers with careful, statistically rigorous designs to test the efficacy of certain policies. (For example, much of the research on No Child Left Behind uses NAEP data, but does so by creating controls and applying careful statistical analyses.) The key words here are statistically rigorous — an eyeball test does not count.
So, yes, although some rumors suggest that they’ll be lower, I hope NAEP scores go up on Wednesday. It will be nice to see and a hopeful sign for education reform and our country. But no, I won’t be using raw NAEP scores to judge the success of policies or politicians or to support the things I already believe — however tempting it might be.