Perspectives on Reading Research
By: Jason Walters, Stephen M. Ryan,
Curtis Kelly, and Pauline Bunce
We asked some of our contributors to give us quick thoughts on some key issues in being a consumer of research. Here is what they had to say in four short pieces.
Hacks for Reading Research
Stephen M. Ryan, Curtis Kelly, and Jason Walters
Stephen: I used to have a lot of trouble reading research. It took so long and a lot of it was just boring. Not having a research degree, I kind of had to teach myself how to make sense of it. Now I am teaching students to do it and trying to pass on what I learned myself for simplifying the process. Let’s call these ideas for taking the pain out of reading research “hacks.”
Curtis: I hear ya, buddy. I’ve picked up some hacks myself and the first hack helps relieve the problem that you just mentioned, that reading research is so time consuming. One of my friends said that the most important thing he learned in his doctoral program was how to read research. He used to read the whole paper. But now he just reads the abstract first and skips on down to the discussion (or conclusion), where he can get the ramifications of the study.
Jason: Exactly. Or if your purpose is to find out where science stands on the topic, you might skip to the literature review instead or, to find out what the sample was, to the procedure. But I agree, you don’t have to read the whole thing.
Stephen: I have a hack related to that. Once you start to know the area you are studying pretty well, look at an author’s reference list first. Are they citing people who are prominent in the field? Is there a bias towards those with a certain view on the issue? Oh, and looking at the publication dates can tell you how up-to-date the research is. I recently came across a new paper that cited nothing after 1990. In neuroscience that is ancient. I didn’t spend much time on that one.
Jason: No doubt. Even research in reputable, high-impact publications can be misleading if we don’t make sure the findings have stood the test of time. A prime example of this is Wakefield’s 1998 article linking autism to MMR vaccines, which was published in The Lancet, one of world’s oldest and most renowned peer-reviewed journals. This research was retracted in 2010, but continues to contribute to faulty research even today.
Curtis: So, here we get to another problem, deciding how much rigor to use in choosing your sources. I think we can call that a hack, too. When you are writing a grad school paper 0r publishing a research paper in a journal, you need to make sure all your sources are highly reputable. I’ve heard more than one professor say most of what is on the internet is bunk, or that students should never use Wikipedia, but I disagree. While I would never cite Wikipedia or a blog article in a journal paper, I find these less formal sources huge time savers in educating myself.
Wikipedia, for example. Here is a hack. For a general topic I am not familiar with, such as “Semiotics,” even before I start looking at papers, I go to Wikipedia to get the basic picture. Then if something catches my interest, I use the Wikipedia reference section to find the related paper and go to that.
Stephen: And references can work like a daisy-chain: the reference section in one paper (or website) can lead you to other publications on the same subject, and so on.
Jason: I have run into difficulties recommending this strategy to learners new to academic reading. Wikipedia’s standards for sourcing are loosely enforced; pop science and outdated research abounds, as well as inconsistencies between articles written in multiple languages.
Curtis: Really? I see it as the most peer-reviewed source on the planet, at least for the big topics. But I agree the pictures it presents can be limited. So, I also go to blogs and online magazines.
Jason: Even then, you’ll want to be careful. These kinds of secondary sources don’t generally have the safeguards against editorializing or embellishing findings you might find in a reputable journal. The principal motivator for most blogs and magazines is ad revenue, and there’s plenty of money in propagating old myths—”we only use ten percent of our brains,” for instance—in slick, accessible write-ups.
Curtis: Jason, give me some credit! I take care in what I look at. I have a lot of trust for online education-oriented publications like those produced by APA and Nature. And, of course, if I want to learn more, I can pull up the related journal paper.
Oh, and it just occurred to me. There is a super easy way to check out those theories that sound too good to be true. Just google the idea, like “Brain Gym” and add the words “criticism controversy dubious.” It’s amazing what comes up.
Stephen: Okay, it comes down to avoiding “disreputable” sources, and that can save you time and distress. After all, we know that once an idea gets lodged in our minds, it can become self-reinforcing—leading us to give more weight to evidence that supports it and less to counter-evidence (aka, confirmation bias). So, the first thing I tell my students is to check how reputable their sources are: journals rather than blogs, publications they have heard of rather than ones they haven’t, authors with academic affiliations rather than YouTubers.
Jason: For that reason, I’ve been showing my students how to use Google Scholar.
Curtis: Now we are cookin! It is amazing how few people really know how to use this wonderful tool. Google Scholar is a great source for dependable, peer-reviewed papers, and you can set the search time frame, such as everything after 2017, find out instantly if a PDF is available (or what other versions are), save your favorites in your library, and even get a list of related papers.
Stephen: You can also see how many times the paper has been cited elsewhere, which can be an indication of its respectability, and, best of all, if you click on the little blue quotation marks, you can get the APA reference of the paper and copy it. What a time saver!
“Research Says…”
Pauline Bunce
“Research” is such a huge topic for a Think Tank!
There’s the “doing,” there’s the “findings,” and there’s the “uses” to which educational research is put—all of which are fraught with misuse by non-experts (and experts!). So, when someone tries to tell me, “Research says …,” I freak out. To me, that brings about an immediate cancellation of almost anything that follows. Unless the speaker pauses and goes on to specify the details and the data from an actual piece of research. Then, I’m back in the room.
As teachers (and consumers), we frequently hear this expression when someone is trying to convince us of something that they are offering to us, be it a product, a method, or even an idea. How dare such people cheapen “research” in this way, by using other people’s years of work for their own, often unrelated, ends. How dare they offer this one word as “evidence” for whatever ideas they may be floating.
Frequently, the speaker is several steps removed from whatever “research” they may be referring to, and they don’t like being pushed to elaborate on it. But push we must. I’m not rude enough to interrupt an invited speaker in full flight, but I have no hesitation in taking them to task after their presentation. Even if the only effect is a private, personal decision on their part to avoid the expression “research says” in future, then that would achieve a small victory for all those tireless researchers whose work is often misrepresented or misused to promote something that they might not actually agree with.
Why do presenters use the unqualified expression “Research says …”? Not “My research says …” or “XYZ’s research says ….” No. Just something that someone might have said or published that seems to back up (shore up?) something that needs a modicum of evidence to convince the audience that it is legitimate. I’m not commenting on academic presentations in academic arenas here, because research attribution would be automatic and obligatory in that zone. No. My criticism is reserved for the pseudo-academic and commercially-oriented presentations that are often made in educational environments, such as schools and childcare settings with teachers and/or parents as the audience.
Unfortunately, the word “research,” is one of those “motherhood” terms that tends to create an immediate, almost unconscious, aura of trustworthiness, scientific verification, and validity around the topic under discussion. It’s the salesperson’s best friend. There’s no challenging “research.” The chink in the armour of this sales pitch, however, is a probing as to what research, in what settings, with how many participants, with what variables at play, with what results, whether it’s been replicated in different settings, etc., etc.
Please challenge such speakers. Tell them that research is great and that you really want to know more about the research they mentioned. If they can’t recall the details, excitedly exchange email addresses. Push them! Follow up when you receive the details, acknowledging any good research and bagging the bad. Encourage them to add some details of the best research into their next presentation. Suggest other supportive research.
It is particularly important for administrators to put on their “teacher hat” and press would-be presenters for details of the research that backs up their project/approach/ideas—before they unleash them on a room full of teachers or parents. School leaders must act as gatekeepers and not allow themselves to be deluded by the powerful potion of confirmation bias. Just because they see a certain problem in their school and they share this with their visitor, does not mean that their prospective speaker necessarily has the “cure.” Too often, school administrators are on a blind hunt for any new-look innovation that will carry their school through the year ahead.
It’s worth remembering that research also says that … “that which can be asserted without evidence, can be dismissed without evidence” (Bennett, 2013, p. 39). (This truism is known as Hitchens’s razor).
Interesting Online Bits
All of Us & Wikipedia
1) This bogus educational research paper was published, and is still available online!
Copied from Wikipedia: In 2020, Bradley Allf, a researcher at North Carolina State University, was invited to submit a paper to the journal US-China Education Reviews A&B, one of many journals run by David Publishing Company. Suspecting the journal was predatory, Allf submitted a nonsense paper espousing the educational benefits of high school students manufacturing drugs in the New Mexico desert, loosely following the plot of the television series Breaking Bad. The paper was authored by Allf as well as fictional Breaking Bad characters Walter White and Jesse Pinkman. In it, Allf claims to have demonstrated that “at-risk” high school students in a chemistry course can benefit from field trips into the desert to make methamphetamine. The paper makes a number of obviously untrue claims, including that Albuquerque is part of the Galápagos Islands, that craniotomy is an effective means of assessing student learning, and that humans did not appear in the New Mexico “fossil record” until 108 years ago. Additionally, the paper’s methodology utilizes invented statistical techniques named after Pokémon and, according to the paper, its figures were created in Microsoft Paint. Despite the obvious issues with the paper, Allf’s submission was accepted by the journal two weeks after undergoing a supposedly “rigorous” two-person peer review. Allf later wrote an article for Undark Magazine about the fake paper and how predatory publishing can be used to sow disinformation.[2]
The original paper The ‘Undark’ article
2) The Conceptual Penis as a Social Construct
Here is another bogus article that got published, with links to a site where you can make your own.
3) Three Online Videos We Love
Chapter 1.3: Where reasoning goes wrong
Victor Gijsbers from Leiden University has an excellent series of videos on logic, reasoning, and the scientific method. Curtis and Stephen have grown so fond of Victor that they started an email exchange with him. Watch the series, if for no other reason than to check out his fashion choices.
Scientific studies: Last Week Tonight with John Oliver
Bogus science has even made primetime TV. And if anyone can bring a point home, it is John Oliver!
How I hacked online dating | Amy Webb
Make research work for you, like Amy Webb did! This video is pure fun! And we guarantee it will touch your heart.
4) Correlations can be misleading!
Most of the research in language teaching is correlative research but, as we have all heard a million times, correlation is not causation. This site drives the point home!
5) Yet, this Is why we can still trust research!
Ben de Haas is our hero. Just a month ago, he wrote this article for Nature.com in which he tells the story of a neuroimaging paper he published in 2014, and asked to retract three months later: no small thing for a PhD. student. But as you read the story you’ll notice something fine and positive in his experience. Ben’s own attitude about trying to find the truth, the gentle and encouraging way Susanne Stoll challenged his findings, and the way everyone involved saw this incident as residing at the heart of pure scientific inquiry, will bring back your trust in the many fine people trying to figure out the world for us.
In this issue, we point out many problems with research, but stories like Ben’s will let you appreciate something important: that the kind of folks doing research are, more often than not, people with a high degree of integrity.
Jason R. Walters is Assistant Director of the Core English program at Nagoya University of Foreign Studies. His primary research interests include self-access learning, native speaker-ism in Asian EFL education, and practical applications of positive psychology in the classroom. <[email protected]>
Curtis Kelly (EdD.) is a professor at Kansai University, a founder of the JALT Mind, Brain, and Education SIG, and producer of the MindBrainEd Think Tanks. He has written over 30 books and given over 500 presentations. His life mission is “to relieve the suffering of the classroom.”
Pauline Bunce is an Australian teacher of English as an Additional Language to young adults. She has taught in Malaysia, Brunei Darussalam, Sri Lanka, and Hong Kong. She has published and presented widely on the specific needs of biscriptal English language learners and her handbook, According to the Script (2016), has been widely used in Australia.
Stephen M. Ryan Stephen M. Ryan teaches English at Sanyo Gakuen University, in Okayama, and tries to keep current.