Blog post by Thomas H Bak
Yes, I admit it: I am a great fan of Peppa Pig. Unlike fairy-tales of magic castles and princesses it depicts in an entertaining way real every-day life and teaches useful skills like how to recycle rubbish, how to make peace with your best friend after falling out with her or how to understand the seemingly irrational behaviour of your younger brother. And it is good for languages too: not only is Peppa Pig highly multilingual, available in a large selection of languages. In several episodes, Peppa interacts with people speaking other languages, whether it’s her French friend or the friendly Italians she meets on holidays. I am sure Peppa, like me, would disagree with the recent article by Simon Jenkins in Guardian that for English speakers learning foreign languages is a waste of time (1).
However, a recent Guardian article about Australia pulling off the air Peppa Pig’s “Mister Skinny Legs” episode (2) made me realise how much Peppa Pig is ahead of some parts of the scientific community when it comes to the interpretation of data. Let me explain. In the “Mister skinny legs” episode, Peppa discovers a spider in the bath and is told that she does not have to fear it since spiders “are very small” and “can’t hurt you”. At the end, Peppa and her small brother George play happily with the spider. So, why are Australians unhappy with this? Were the British producers ruthlessly irresponsible? Or are the Australian authorities irrationally anxious? The answer is clearly neither nor. The episode is fine in Britain, where spiders are unlikely to cause children any harm. But it is less appropriate in Australia, where spiders can be large and dangerous and thousands of people are admitted to hospitals each year because of spider bites.
This answer will be quite obvious to anybody with some knowledge of Britain and Australia. But let’s imagine we want to find out in a systematic and scientific way whether spiders are dangerous or not. So we put together all information we find about casualties, accident and emergency admissions or any incidents related to spiders. We are not likely to find many in Britain, or in Europe, or indeed in many other countries of the world. But we will certainly find some in Australia. So, we count all the results and conclude that spiders are harmless and the Australian data are irrelevant, overridden by the overwhelming evidence from other places.
Well, Peppa Pig would probably laugh at such a simplification and the Australian authorities were well advised not to take this path when deciding to pull the spider episode off air. Yet, what I have just described is becoming increasingly commonplace in science as part of the current fashion for “systematic reviews”. The idea of a systematic review is appealing and deceptively simple: you put together in a systematic and accountable way (e.g. using specific keywords) all evidence you can find about a topic. Then you count it, weighting it for different factors such as sample size, randomisation methods used etc., and decide whether something is correct or wrong. I am all in favour of the first part: the systematic search. But, as illustrated by Peppa Pig’s spider episode, I also see the dangers of counting results coming from radically different environments against each other.
Of course, a systematic review can be at the same time systematic and scholarly, taking into account a whole range of factors which can influence the results. It can arrive at complex and differentiated conclusions. Unfortunately, this is often not the case. One of the reasons might be an expectation that a systematic review should produce a simple and definitive yes or no answer, valid across space and time. There is an effect or there is no effect: tertium non datur. This can tempt authors into giving simple answers to excruciatingly complex questions. The other problem is that systematic reviews, once you know their procedure and the rules of statistical evaluation, do not require in-depth knowledge of the field. They can be written by the best experts in the world, but also by authors who might not appreciate the whole complexity of the topic they are investigating.
All this can be clearly seen in the current debates about possible cognitive effects of bilingualism and language learning. There is indeed a lot of conflicting evidence, with some studies showing positive effects of bilingualism on attention, cognitive ageing, dementia etc and others not replicating these effects. So there is a big temptation to solve this problem once and for all by evaluating all the evidence in a systematic review. As mentioned above, I fully support the idea of a systematic search, but I have serious doubts about reducing all evidence to a simple yes or no answer (3). As pointed out by Bak & Alladi (4), the conflicting evidence in bilingualism and cognition research comes often from different countries and societies, with very different attitudes towards bilingualism. We need not only more studies, we need a much more diverse sample. We also need to study the interaction between the social and cognitive aspects of bilingualism (5). And we cannot assume that the answer will be the same in every place.
We should not abandon doing (and reading) systematic reviews. But we should read them critically, like any other piece of scientific evidence. As a general rule of thumb: it’s wise to be cautious about statements offering very simple answers to very complex questions.
Links:
- https://www.theguardian.com/education/2017/aug/28/just-speaking-english-wont-get-us-very-far-in-the-world
- https://www.theguardian.com/tv-and-radio/2017/sep/05/peppa-pig-spiders-cant-hurt-you-episode-pulled-off-air-in-australia-again?utm_source=esp&utm_medium=Email&utm_campaign=GU+Today+main+NEW+H+categories&utm_term=242330&subid=8128146&CMP=EMCNEWEML6619I2
- http://www.sciencedirect.com/science/article/pii/S0010945215002919?via%3Dihub
- http://www.sciencedirect.com/science/article/pii/S0010945215002117?via%3Dihub
- https://www.repository.cam.ac.uk/bitstream/handle/1810/264363/Bak%20MehmedbegovicJLSP%202017.pdf?sequence=1
