A Consumer Guide for Reading Research

March 2021

In our longest issue yet, we take the time to examine the problems of junk science and the Replication Crisis, and we explore what it really means to read and understand research. Our authors share their tips for understanding research and how to avoid falling for fake statistics.

Our cover: “Flowers don’t tell, they show.” – Stephanie Skeem

Cover Photo by Edward Howell on unsplash.com, others from pixabay.com

Watch before you read...

In an issue rather different from our usual, and one of our longest, we will explore how research can mislead us, and suggest tools for reading it more intelligently. We start with a video by Kevin Lieber and a podcast by RadioLab that show two typical problems. Julia Daley will discuss why those issues are important for educators. Then we go to the Think Tank, where Stephen M. Ryan will write about his lifelong struggle with research (even though he now edits three journals), and a team of contributors will take some pot shots at research reading. Matt Ehlers will dazzle us with examples from two useful sources on how to sniff out poor research, and Curtis Kelly will tell us about the jaw-dropping Replication Crisis. You should be in the groove by now, so the Think Tank Team will give you a Genius Quiz to see if you can find where the bad smell is coming from in five studies. After that, Amanda Gillis-Furutaka will examine problems with studies based on fMRI, and Caroline Handley will pierce the veil of research stats to give us an insider’s view of what can go wrong in that mysterious corner.

Finally, in the PLUS section, Pauline Bunce will entertain us with one of her favorite books.

(In the magazine) We are slowly shifting over to a new layout design from this month, thanks to Rishma Hansil. Let us know what you think!

RadioLab Podcast

Stereothreat (2017)

Our Thoughts on Reading Research

When Research Goes Wrong Julia Daley

In recent years, the problem of “fake news” has become widely known, not just in the USA but around the world. Sensationalist headlines from dubious sources having little basis in reality are virulently shared in social media, spreading misinformation far and wide resulting in sometimes very real consequences. With this month’s issue we’re tackling a related issue, “junk science,” which often exhibits the same fundamental flaws as “fake news,” but also, fortunately, the same solutions: adopting a stance of healthy skepticism, examining the evidence, verifying the credibility of the sources, etc. In short: we need to wear our critical thinking caps when reading about science.

Think Tank Articles

My Social Science Crises Stephen M. Ryan

“Following the science,” has become the politician’s mantra of choice in these virus-ridden times. I am not a scientist, but, as a teacher, I have long been a consumer of social science. Following the science usually seems like a good idea at the time, but following it blindly is a terrible idea that has led me fairly regularly to false turns and pratfalls. I offer my cautionary autobiographical sketch to encourage others to handle science with care.

Perspectives on Reading Research Jason Walters, Stephen M. Ryan, Curtis Kelly, and Pauline Bunce

We asked some of our contributors to give us quick thoughts on some key issues in being a consumer of research. Here is what they had to say in four short pieces.

These Two Resources Teach You How to Spot Misinformation Mat Ehlers

It was early in 2007 when I first heard about a 1954 book called How to Lie with Statistics, by Darrell Huff. Though I knew next to nothing about it aside from its title, that alone made me want to read it, since it seemed like it promised to be an entertaining work that I’d learn something from. While I was unable to get a copy of it then, I vowed that I would find and read it someday.

The Replication Crisis: Research Fraud? Or Something Worse? Curtis Kelly

Brian Nosek’s Reproducibility Project touched on a major problem in research: the publish or perish dilemma. To get hired by a university, to get tenure, and to get research grants—all the gateways associated with a successful university career—you have to publish in good journals and publish a lot. “Publish or perish” they say, and that incentivizes researchers to get published no matter what.

Genius Quiz! Dubious Research Think Tank Team

How good an eye do you have for finding dubious research? Read these three studies, all real, and see if you can figure out why we might put them in the compost pile.

Be(a)ware of What You See in an fMRI Image Amanda Gilis-Furutaka

Functional magnetic resonance imaging (fMRI) generated great excitement when it was introduced in the 1990s, because it allowed brain researchers to see what was happening, and where, inside a living person’s head. Before the twentieth century, researchers could only deduce the functions of brain regions after they had been damaged, and could only look directly at the brains of people who had died. However, fMRI images have also been labelled as “brain porn” (Helding, 2020, p. 48) when they are overused to add credibility to what writers and public speakers wish to say. I count myself among the gullible guilty who have scoured the internet for an image that will illustrate my point, with too little investigation into the original source. Let me share what I now know better!

Lessons from a Journey into Quantitative Data Analysis: Understanding Research by Doing It Caroline Handley

The theme of this Think Tank has been to discuss why we believe it is important to be aware of various issues with psychology and applied linguistics research methods and analyses. I decided to write my story about how I became aware of these issues and what I understand about them. In focusing on how I learned about conducting quantitative research and interpreting the results of published studies, my aim is to give an example of how a deeper understanding of other people’s research begins with doing your own. My journey is just beginning and is one I have undertaken with the help of textbooks, articles, and online resources, but few taught classes or discussions. I do not pretend to be an expert, but I will share some of the important lessons I’ve learned over the last five years as I continue to work on getting a PhD.

Think Tank Plus

A Quick Review of Tom Bennett’s Teacher Proof (2013, Routledge) Pauline Bunce

In 2019, I attended an international conference that was headlined by Stanislas Dehaene (of Reading in the Brain fame). A tough act to follow, that’s for sure, but one of his supporting speakers gave an address that I’ve never forgotten, because it went straight to my teacher-core. Here was a teacher-researcher, still wrangling adolescents in an inner London classroom, taking a swipe at a range of unrealistic “educational innovations” that his school had attempted to implement. This guy was speaking my language, albeit with a Scottish accent.

Call for Contributions: Ideas & Articles Think Tank Staff

Become a Think Tank star! Here are some of the future issue topics we are thinking about. Would you, or anyone you know, like to write about any of these? Or is there another topic you’d like to recommend? Do you have any suggestions for lead-in, or just plain interesting, videos? How about writing a book review? Or sending us a story about your experiences? Contact us.

The MindBrained Think Tanks+

is produced by the Japan Association for Language Teaching (JALT) Mind, Brain, and Education Special Interest Group (BRAIN SIG). Kyoto, Japan. (ISSN 2434-1002)

Editorial Staff

Stephen M. Ryan Julia Daley Marc Helgesen

Curtis H. Kelly Skye Playsted Heather McCulloch

Jason Walters Rishma Hansil

Get issues delivered to your inbox at the start of each month!

Unsubscribe

Stop receiving issues in your inbox.

Reader's Survey

Take a short survey to help us learn more about our readers!