Mark Schneider: My Response to Essay Rebuttal — I’m Concerned About the PISA Exam’s Future and the Implications of Its Sponsor’s Global Ambitions

In her response to my Feb. 25 essay arguing against the expansion of the Program for International Student Assessment to 40 additional countries, Michele Bruniges documents the many admirable aspects of PISA. Indeed, many of her points are exactly why the U.S. has strongly supported the exam since its start. However, the past is not the future — and my concern is about the future implications of current trends for PISA.

My argument had nothing to do with pitting rich countries against poor ones. Rather, my point was to call for “a pause in PISA’s expansion [to] provide time for further development of testing methodology and governance structures to ensure that the benefits of the expansion are ultimately realized — for member and non-member countries alike.” This is not a call for a division of the world into rich versus poor — it’s a call to take a pause in the willy-nilly expansion of PISA to protect its future for all participants.

Countries that are not members of the Organisation for Economic Co-operation and Development (OECD), which administers the exam to 15-year-olds worldwide, have brought great value to PISA. Shanghai and Singapore, while not OECD members, are stellar educational systems from which we have learned much. However, the PISA design tells nothing about the why of Shanghai and Singapore, only the what. Moreover, it was Singapore’s participation in an early study by the International Association for the Evaluation of Educational Achievement, not PISA, that first shook that nation to its core and set off the reforms that have propelled its students to the top of international ratings.

Further, as I noted in my piece, PISA is testing radically different populations of 15-year-olds across non-OECD countries — a fact that is not made clear in Bruniges’s commentary or in PISA reports.

Empirically, the percentage of 15-year-olds participating in the exam across OECD countries is 92 percent and tightly clustered. Among non-OECD or partner countries, with the exception of Vietnam, no country reaches even the lowest level of participation of OECD countries; the average is only 79 percent, and the range among them is far greater.

It’s these types of omissions, obfuscations, and departures from the data that drive my call for global statistical standards for the OECD and all international testing organizations, which must be held to higher, more rigorous assessment standards given their international reach.

My concerns stand: that the OECD is pursuing global growth without considering the effects of continued expansion on the technical quality and validity of its data. Specifically, I pointed out that current questions about the validity of PISA’s comparisons “will grow worse as OECD pursues its global ambition of having about 100 non-member countries take part” in PISA. The methodological advances of PISA, as laudable as they are, have not sufficiently addressed these concerns.

Complacency about PISA’s future — and the unwillingness of the OECD to discuss the implications of its global ambitions — is exactly why I wrote my commentary in the first place.

Mark Schneider is director of the Institute of Education Sciences at the United States Department of Education.

Help fund stories like this. Donate now!