Genius Quiz! Dubious Research
By: Think Tank Team
How good an eye do you have for finding dubious research? Read these three studies, all real, and see if you can figure out why we might put them in the compost pile.
Study on EFL Writing
Back when this incident happened, TESOL Quarterly was the premier journal to publish your language teaching paper in. Some friends of mine had their paper accepted, and I still see it cited from time to time. To find out how different approaches to teaching writing worked, each of the three researchers, at two universities, used a different approach, and through pretests and post-tests measured language growth in their semester-long classes. They looked at error frequency, T-units (syntactic maturity), etc. Like all Japanese university courses, the classes met once a week for 90 minutes from April until October, with a two-month summer break in July and August. The students in the two universities were not at the same starting level of proficiency, but they accounted for that overall difference with statistical tools.
At least three other things, however, were not accounted for, and not even considered. Can you guess what was wrong?
Mediterranean-Style Diet Linked to Better Thinking Skills in Later Life
This study from Univ. Edinburgh popped up in Neuroscience News just this morning. Basically, 511 Scots over the age of 70, none of whom were smokers or suffering from dementia, were given global cognitive function and verbal ability tests. Childhood IQ scores were also factored in to account for starting differences. It was found that those who partook in a Mediterranean diet, or something similar, had better overall thinking skills. The difference was small, but statistically significant. What was wrong?
Setting Up Training Courses for Dentists
A questionnaire was developed for working dentists in order to determine what areas they were weak in, so that they could be steered towards appropriate training. The dentists were asked to list the dental topics they wanted to study. What was wrong?
Vaccines Cause Autism
Gemma Conroy wrote about this 2015 paper published in Law & Medicine. The graph is trying to establish a link between the measles-mumps-rubella (MMR) vaccine and autism in the UK and Scandinavia. What is wrong? (Hint, look at the percentages.)
“Japanese Students Prefer Online Classes”
From Curtis Kelly: “The title above is the same as in a paper one of my students wrote. The student interviewed ten students who took my online English class. All ten of the interviewees said they liked my online class more than any f2f classes they took in 2019.” Go to town Research Eagle Eyes.
Now the Answers: What’s Wrong
Writing Methods Study: Treatment Problem
In addition to the different methods being used by different teachers, which has a confounding effect, the students were English majors. That means that in addition to this one writing class, they were taking ten or so other English classes the same semester. That other 90% of their English training was an even more likely cause of gains in proficiency. In addition, going abroad to study English over the summer break was popular back then, which many students probably did, and came back semi-fluent. None of these factors affecting proficiency were accounted for, or even mentioned in the Limitations section.
Mediterranean Diet Study: Sample Problem
They tested elderly people in Scotland, not Italy or Spain, where the sample would be better balanced. Obviously, someone in Scotland adopting a Mediterranean diet is likely to read more, have a better education, and be more widely travelled (thus wealthier). It is no surprise that these kinds of people scored slightly higher on the things they measured: verbal ability and cognitive function. The only control they used to prevent sampling difference was childhood IQ scores, which, of course, were calculated long before these people found their life trajectories. In other words, this study just identified that better educated and brain healthy people were more likely to adopt a Mediterranean diet rather than others. Come on, Edinburgh! You are supposed to be a world leader in research!
Setting Up Courses for Dentists
As Chong-Ho wrote: “The reliability of self-report data is an Achilles’ heel of survey research” (2020). Indeed. Studies in psychology often wrestle with data from respondents that might be unintentionally exaggerated (I go to the gym eight times a month), embarrassing (I go once a year), or just plain misremembered. In this case, however, the problem was different. The dentists ended up listing the training areas they were most interested in and, thus, the areas they already knew the most about. A performance-based follow-up evaluation found they completely missed the areas they were weakest in, because, of course, they did not know about these areas.
Vaccines Cause Autism
It is a little hard to see without looking carefully, but the scales on the left and right are drastically different in size. Each 0.1% difference between ticks for autism is set against a whopping 2.0% for MMR coverage. Adjusting autism to the same 1% interval as MMR would show virtually no change in autism levels, while scaling MMR to the same 0.1% interval as autism would show a massive drop in MMRs from 2005-2008 while autism nearly doubled. Manipulating graphs lets you tell almost any story you want.
“Japanese Students Prefer Online Classes”
Also from Curtis: “The results are indisputable. The student got an ‘A’.”