Beware! Flawed Research

Small School Closure in Wales: New Evidence - The Reynolds Report

Comments on methodology by Liz Spencer - eminent social science researcher, Emeritus Professor Ray Pahl FBA, was asked to undertake a peer review of this research because it had begun to surface in England as alleged evidence for closing and amalgamating small schools. Professor Pahl himself deplored the use of an IWA brochure as 'evidence.' He also noted the lack of qualification in educational and social science research of Meriel Jones. However, as he was in the area where the research had been cited and did not feel directly involved in the local debate, he asked Liz Spencer. Liz Spencer had produced for the Cabinet the seminal report on 'Qualitative Research Methodology.' She also directed the overall training of students in research methods. The Reynolds Report is being used now as an example of how NOT to do research. In 2009, along with the Vice-Chancellor of Cambridge University she was elected to the Academy of Social Sciences.

Conclusions - because of fundamental problems with the way the study was conducted - sampling, data collection and analysis - the reader can have no confidence in the conclusions. Unfortunately, in terms of the credibility of the report as a whole, there are some cases where we can see the conclusions are definitely not supported by the evidence presented.

This is one of the most amateur, ill-conceived, and thoroughly bad pieces of research I have seen in a long time. There is a worrying trend for consultants to carry out case studies, which seems to be used as an excuse for conducting poor quality quantitative and poor quality qualitative research. Of course, case studies don't usually allow the selection of large random samples, but there is no excuse for sloppy questions and analysis. Also, the strength of case studies is the detailed contextual information about each case, without which there is little scope for generalisation.

I think the person carrying out the empirical research has little grasp of good research practice, be it qualitative or quantitative. If this is the kind of evidence used in evidence-based policy, heaven help us!

NASS believes the Reynolds Research, which in the "Western Mail" and WALESONLINE he extends by openly endorsing the drift to closures and amalgamations in Welsh education policy, is already being used to shape policy and may be imported into the rest of the UK in similar closure proposals. We wish our members to be well aware of just how flawed this research is. The full peer review is available on request.