Produced by Mike Hally @ BBC Radio 4 Next week, governments around the world will see how where their country stacks up in the latest PISA survey, otherwise known as the international league table of schools. The edreformy contingent in United States is no doubt waiting to see how they can use the results to justify their agenda, while the opposition would like it if we all stopped putting so much emphasis on them. Tipped by Pasi Sahlberg, yesterday, this BBC 4 radio segment asks the question whether using the results and rankings to justify how children should be educated is even a good idea. Unfortunately, many in the United States and England seem to think so.
Cambridge University professor David Spiegelhalter investigates, talking to leading academics in the world of education including Svend Kreiner in Copenhagen, Harvey Goldstein at Bristol and Oxford’s Jenny Ozga. He discusses the issues with staff at the National Foundation for Educational Research who administer the PISA tests in the UK and puts the criticisms to the OECD’s Andreas Schleicher, the man that Education Secretary Michael Gove once called ‘the most important man in British education’ because his work on PISA.
With only five days to hear the segment, it is well done and quite worth the time to listen. Even more, it is great to hear a perspective outside the States, especially one from England where they are farther down the teach-and-test road with diminishing results. Included in the segment is an interrogation of the PISA method, which has been subject to its own mounting criticism. In fact, it reminded me of another resource tipped by the Finn Sahlberg during the summer, “Is Pisa fundamentally flawed?” In this piece from the British TES Connect, which could have served as a template fro the broadcast, many of the same questions are posed, including the deeper one that questions the very modeling and methodology of how the PISA results are obtained. There is no shortage of academics that challenge how the underlying conceptual framework is applied.
What if you learned that Pisa’s comparisons are not based on a common test, but on different students answering different questions? And what if switching these questions around leads to huge variations in the all- important Pisa rankings, with the UK finishing anywhere between 14th and 30th and Denmark between fifth and 37th? What if these rankings – that so many reputations and billions of pounds depend on, that have so much impact on students and teachers around the world – are in fact “useless”?
Chief among both pieces is Professor Svend Kreiner of the University of Copenhagen, Denmark. A student of Georg Rasch, a fellow Danish statistician, whose responsible for the complicated model used by PISA to determine results. While the argument can get pretty deep quickly here is a sample.
[Kreiner] says that for the Rasch model to work for Pisa, all the questions used in the study would have to function in exactly the same way – be equally difficult – in all participating countries. According to Kreiner, if the questions have “different degrees of difficulty in different countries” – if, in technical terms, there is differential item functioning (DIF) – Rasch should not be used.
This is a sentiment that the BBC 4 segment captures as well, in addition to a response from OECD’s Andreas Schleicher. It all makes for a very compelling, albeit wonky, story. Of course, it is precisely the kind of education story that is unlikely to get a lot of press in this country. Not only does it venture into extraordinary academic and statistical specialization, it cannot be easily reduce into a soundbite.
Nevertheless, the PISA survey and rankings is a keystone in the edreform agenda. If it is indeed invalid, as the possibility is suggested by these two pieces of journalism, it should torpedo the entire effort. Yet it won’t. Not only is this story likely to get little attention in this country, it is more likely to get buried by our largely lazy education reporting.
What’s more, those ardent edreform activists wouldn’t acknowledge any challenge to their agenda anyway. Politicians and policymakers long figured out that facts and evidence matter far less than emotion does. Those advancing that edreformy agenda know this and would only finely tune a fear message that overpowers any evidence. All the while, they will continue clamoring for more data and scientifically-based evidence, precisely the kind they will quickly dismiss if it challenges their message.