Matthew Taylor at the RSA has an insightful little post this week about Policy based evidence making, a neat reversal that exposes how often research is really only ideology skulking behind a mask of objective respectability. It coincided with my having to do some research of my own that exemplifies the problem Matthew was raising, so I thought it could be an informative thing to actually show readers what this looks like when it happens.

So the US education department employed SRI International, to carry out some research into e-learning and K-12 students (our primary and secondary pupils.) The study, published in May 2009, was a meta-analysis, so not original research, which looked at other people’s research over 12 years, comparing the effect of e-learning to face-to-face learning. It's even titled coincidentally, Evaluation of Evidence-Based Practices in Online Learning: A Meta-Analysis of Online Learning Studies. The researchers wanted to answer these four questions:

• How does the effectiveness of online learning compare with that of face-to-face instruction?
• Does supplementing face-to-face instruction with online instruction enhance learning?
• What practices are associated with more effective online learning?
• What conditions influence the effectiveness of online learning?

The researchers hit a bit of a problem which they openly acknowledge, An unexpected finding of the literature search, however, was the small number of published studies contrasting online and face-to-face learning conditions for K–12 students.

What of course this means is that because they couldn’t find much in the way of real research that met their strict criteria, (a) contrasted an online to a face-to-face condition, (b) measured student learning outcomes, (c) used a rigorous research design, and (d) provided adequate information to calculate an effect size for K-12 kids, they had to look outside that age range. They did and the more than 1000 papers they screened covered career technology, medical and higher education, as well as corporate and military training. That was the only way they could find enough studies with older learners to justify a quantitative meta-analysis.

Again this is something they openly acknowledge when they state: Thus, analytic findings with implications for K–12 learning are reported here, but caution is required in generalizing to the K–12 population because the results are derived for the most part from studies in other settings (e.g., medical training, higher education). My bold italics.

So how does that all translate into the real world? This is what that bastion of academic objectivity Becta, had to say about this research.

A thorough US Government report comparing results from e-learning and traditional learning, mainly in non-school based studies. Systematic search of the research literature from 1996 to July 2008 identified more than a thousand empirical studies of online learning. The meta-analysis found that, on average, students in online learning conditions performed better than those receiving face-to-face instruction.

So ignoring the illiterate, opening none-sentence, (no finite verb=no sentence I'm afraid guys!) my bold italics highlight the single sentence Becta quoted directly from the original research abstract. And they are far from the only ones who donned the selective spectacles. I suppose we should give them some credit for attempting to hint that the research might not be of much use, since it was in non-school based studies, but to leap from what the research actually dealt with, and the crystal clear reservations of the researchers themselves, to this huge claim that e-learning is best…well, as neat an example of policy based evidence making as Matthew could wish to find.

I guess the guys from Becta just never noticed that bit in the report (p. 51) which says, the findings of this meta-analysis...should not be construed as demonstrating that online learning is superior as a medium.