The Movements of Language: Learning from Embodied Semantics
Brian J. Birdsell
Among the Best of 2020
Readers chose this article as one of the five best from all of 2020! Check out the othe nominees with the button below.
From Computers to Bodies: Looking for the Architecture of Language
The brain and computer are two everyday words for concepts that widely capture the imagination of so many people by their complexity and intelligent design and have been bed partners in an enduring metaphor since the 1940s. These two concepts are so intricately interconnected in language and thought that it is hard to imagine one without imagining the other. For instance, the electrical currents of a computer are similar to the action potentials of neurons firing in the brain. When we talk about language, this metaphor gets further extended whereby language becomes a computational process involving algorithmic codes (the syntax) manipulating abstract symbols (the words) into strings. These abstract symbols (such as “orange”) are hierarchically connected to higher level symbols (“fruit”) and to an array of sematic features [sweet, grows, round]. Indeed, this metaphor sparks the imagination and some have even asserted that one day we might just be able to upload the brain (mind) into some computer, since the brain and the computer are essentially analogous. In fact, the New York Times covered this topic in 2015 with a piece called “The Neuroscience of Immortality” by Amy Harmon, who detailed the high-tech efforts required to accomplish this feat. This kind of thinking requires one to believe that the mind resides in the brain and that the brain is separate from the body, as in the Cartesian dualistic view of the mind and body.
In contrast, it is more difficult to view the brain as a biological organ with cell tissue, cerebral arteries, and veins, and able to grow (by way of neurotrophic factors) or decay (as in neurodegeneration). Moreover, it is difficult to think of the brain as being intricately linked to the body like any other organ. However, that the brain is an organ linked to the body is the perspective taken by theorists who follow an embodied view of cognition. This embodiment view gathered momentum in the 1980s and really became prominent in the early 2000s. This view rejects the computational view of the mind and instead argues that many processes, including language, are essentially grounded in the sensorimotor systems. That is to say, to use a term proposed by Michael Anderson, language developed as one form of “neural reuse,” a kind of cognitive piggy-backing. In this view, there is no dedicated language module in the brain (in the way Fodor or Chomsky imagined), but instead language processing is grounded in the sensory, motor, and affective systems. Take for example Broca’s region, which for a long time was viewed as being a dedicated language area of the brain for motor speech-production, but researchers now contend that this region is also responsible for a number of other cognitive functions like active understanding and imitation. In short, an embodied view of cognition proposes that language comprehension involves widely distributed networks in the brain and the words themselves provide cues to run a complex simulation (for more details on what a simulation means, refer to the articles by Gillis-Furutaka and Kelly in this edition) by recruiting the sensory, motor, and affective systems.
Movement, Odors, and Tastes: The Coupling of Language to the Body
Reading, viewed as a “high” form of cognition, compared to “lower” cognitive processes like perceptual or motor skills, is often assumed to be detached from the motor cortices (unless, of course, you like to walk while reading). Thus, reading is commonly thought of as a purely mental activity. Yet, an embodied view of cognition has questioned this assumption. Take for example one of the earlier behavioral studies that cleverly showed how reading involves running mental simulations that recruit the motor regions of the brain. In this study, Glenberg and Kaschak (2002) had participants judge the sensibility of sensible (e.g., Open the drawer) and nonsense (e.g., Boil the air) sentences. The sensible sentences were written in two ways; movement away from the body (e.g., I gave the pizza to Andy) and towards the body (e.g., Andy gave me the pizza). These participants then sat with a device on their laps. This device had three vertically placed buttons for yes, no, and start. The start button was always in the middle. The independent variable in this study consisted of having the “yes” button sometimes away from the participants (see Figure 1, Condition 1) and sometimes closer towards the participants (see Figure 1, Condition 2). Using reaction time as the measurement criterion, they found when the sentence and the action were congruent (I gave the pizza to Andy in Condition 1), reaction times were faster, compared to when they were incongruent (I gave the pizza to Andy in Condition 2). They interpreted these results as suggesting that when we read a sentence like I gave the pizza to Andy, we are actually running a mental simulation that recruits the motor areas of the brain for moving the arm away from the body. Therefore, in Condition 2, the physical action of moving the arm towards the body in order to press the “yes” button closer to us contradicts this motor simulation and thus causes the slower reaction time. If the mind was processing language as an algorithmic computation, where language is completely disembodied or decoupled from the somatosensory and motor systems, reaction times would be the same.
Condition 1
Condition 2
Figure 1: An illustration of the two conditions in the Glenberg and Kaschak (2002) study
Researchers working within the framework of embodied cognition conduct not only behavioral studies, but also neuroimaging ones that use functional magnetic resonance imagining (fMRI) to better understand structural areas of the brain that become active during a specific task. Watching a YouTube video of someone doing an extreme sport like mountain biking along some crevice in the Canyonlands, in Utah, is probably enough to make the average person cringe, squint, and try to cling to something in order to maintain a sense of balance. Performing an action yourself and watching another person perform the same action are more intricately connected than once believed. Similarly, reading about kicking a ball and actually kicking the ball has been shown to activate the same motor control areas in the brain responsible for kicking. This motor cortex has a somatotopic map of the human body and this map-like representation corresponds to different body parts. For example, at the dorsal part of the motor cortex near the midline there is the representation of the foot, knee, and leg, and then extending to the lateral part there is the arm and hand, followed by the face, mouth and tongue. Why is this important, you may ask? Well, this brings us back to the kicking ball example. In one early neuroimaging study that provided important evidence for an embodied view of language processing, the researchers had participants in an fMRI read a set of action-related words. These action verbs are associated with certain effectors or body parts that are used to enact that specific verb (e.g., kick – leg; lick – mouth; grab – arm/hand). When the participants read these action verbs, the area of the motor cortex associated with that effector showed activation, which they interpreted as evidence for an embodied view of language processing (at least for action-related verbs).
Further research has extended connections like these to the sensory systems. In everyday life, we take in endless amount of input from our environment. For instance, when I cut garlic for my pasta sauce, I am taking in sensory and motor information about this concept, garlic, which likely includes information about its shape, color, and, most importantly for anyone who cooks with garlic, its odor. Later, when I come across the word garlic in writing or perhaps in a conversation where the thing itself is not in my immediate environment, what kind of representational content provides meaning for it? Again, in a computational view, the meaning comes from the connections it has to other abstract symbols, divorced from the sensorimotor systems. Yet, in an embodied view, the meaning arises from recruiting those same sensorimotor systems that initially provided the input by way of running a simulation. In an interesting neuroimaging study by some researchers in Spain, they found that when people simply read odor-related words like garlic (lavender, cinnamon, etc., and even fart) compared to control words (e.g., coat, bell, letter), they found spreading activation in the olfactory system. They interpreted these results as suggesting that language processing recruits associated sensory information as well as traditional language areas. In a similar study, a group of researchers extended this idea and examined whether they could find the same sensory neural-reuse for gustatory words (e.g., chocolate, ham, pizza, etc.). Again, they found that words with gustatory associations activate gustatory regions of the brain. These studies and many more similar ones provide evidence that when we read, hear, or even think about a word, we run a mental simulation. This simulation reactivates experiential traces in corresponding brain regions that were active during the initial interaction with that specific object, action, or event. This might include the sensory, motor, or affective systems and this is really the heart of embodied semantics.
Controversies: The Problem of Abstract Concepts
The research mentioned so far primarily focused on concrete words that are tightly coupled to sensory modalities, or action-related words, but as we all know, a large part of language is abstract. That is to say, abstract words like democracy or honesty are not connected to any physical object or action in the external world, as garlic or kick are. How then do we understand abstract language and how is it embodied? Take for example the following sentence from BYU’s Corpus of Contemporary American English:
The Kellehers of Massachusetts wouldn’t begin to know how to pack lightly: They’re dragging around too much emotional baggage. And every summer, they haul it all up to Maine to their vacation home on three acres on Cape Neddick. [Christian Science Monitor, June 28, 2011]
We all know that the sentence above is not talking about actual “baggage” that is being packed, dragged around, and hauled up to Maine for a summer vacation, but rather psychologically unpleasant past experiences. This complex and abstract psychological concept gets mapped onto this now entrenched idiom “emotional baggage.” Why baggage and why not a bookshelf? This is probably due to the fact that baggage has the affordance of something we carry or drag around similar to these negative experiences in life. Therefore, one avenue of research in the field of embodied cognition argues that abstract language is grounded in the concrete by way of metaphors. For instance, when we think of intimacy, we understand this abstract concept by mapping it onto our understanding of a physical source concept such as, closeness. Thus, we can say something like we are not very close despite the fact that we may be in close proximity to each other, but, in this metaphorical expression, it refers to not having a social, friendly, and supporting relationship. Such metaphors often go unnoticed in everyday conversation since they are highly conventional and entrenched in the language. Nevertheless, they are metaphorical and provide conceptual structure to our understanding of abstract ideas.
In expanding earlier research on the involvement of the motor cortex in the processing of literal language, a group of researchers investigated whether similar involvement would also become active when those same action words were used metaphorically. For instance, we can say “grasp an object,” but also metaphorically “grasp an idea.” In their study, they used arm-related action word sentences along with leg-related action word sentences (e.g., kick the ball; kicked the habit). They showed semantic activation in the corresponding motor regions (dorsal motor cortex for leg and lateral motor cortex for arm) when participants read silently both literal and figurative uses of these action-related words, which they argue illustrates that abstract language is grounded in the sensorimotor systems by way of metaphor. Raymond Gibbs has also extensively studied the embodied effects of metaphor comprehension. Just to take one study by him and his colleague: they had participants perform, in the way of a pantomime, certain body movements (e.g., push; stamp; swallow) and then read metaphorical expressions that contained such movements (e.g., push the argument; stamp out a fear; swallow your pride) and measured their reaction time for comprehending the sentence. They had three conditions: 1) conducting an action matching the metaphorical sentence, 2) an action not matching the metaphorical sentence, and 3) no action. Results indicate that people are faster at comprehending the metaphorical sentences when the sentences are preceded by performing the related action. A second experiment extended this by showing that one does not need to actually perform the action, but simply imagine doing so to get similar results. That said, the main problem of a metaphorical view of explaining abstract words within an embodied framework is that it seems rather unlikely that it can explain all abstract words since many are simply not metaphorical.
Still, there is much controversy about abstract concepts within an embodied framework and many suggest that, instead of viewing language comprehension as fully embodied or purely computational and disembodied, it is more fruitful to view it in the light of gradations of sensorimotor engagement. This pluralistic or hybrid view of embodied cognition seeks to incorporate some form of symbolic representation, especially through the use of a semantic “hub” within the architectural system of language. This hub, most likely in the anterior temporal lobes, acts as an integration area for semantic content binding and organizing the diverse semantic features of words. In short, there is still much debate concerning the semantic representation of abstract words.
Grounding Foreign Language Education: Bringing the Body back into the Classroom
An embodied view of cognition and particularly language processing has had an enormous effect on how we understand language. Yet, those working in the field of foreign language instruction might question whether or not the semantic representations in a foreign or second language is rich enough for embodied activations within the motor and somatosensory areas. This is especially relevant since most research has been conducted using the participants’ L1. In one study, a group of researchers investigated the associations between the L2 and the sensorimotor systems for German-speaking (L1) participants who were English (L2) learners. In Experiment 3 of their study, they examined whether or not simply viewing L2 words that either had a spatially upward metaphorical mapping (e.g., happy is up) or downward mapping (e.g., sad is down) would facilitate hand movement in an upward or downward way. To accomplish this, they had participants respond with an upward or downward hand movement to a set of colored words that had emotional valence, either positive (e.g., happy, joyful) or negative (e.g., sad, depressed). Participants were told to ignore the meaning of the word and only consider the color (see figure 2). The participants began with their hands pressing down two middle buttons. Then they would see a word on the screen and depending on its color, they were to lift the right hand and move it in an upward direction to press the top button or move the left hand in a downward direction to press the bottom button. Despite the meaning of the word being irrelevant to the task, positive words facilitated faster upward movement and negative words facilitated faster downward movement. Thus, the researchers concluded that the metaphorical meaning of L2 words is processed automatically, even in an L2, and has a faciliatory effect on the motor system. That is to say, when we read words with positive emotional valence, this cues the motor system in a spatially upward direction, and such effects also appear in a second language.
Figure 2: An illustration of the experiment setup
How the L2 semantic system is actually organized is still not very well understood and likely highly dependent on the age of acquistion, learning environment, and overall competency with the language. That said, from an embodied perspective, L2 learners likely acquire the language in an impoverished setting, as in a classroom, compared to in a natural environment. For instance, in one study, Vukovic and Shtyrov (2014) compared L1 and L2 motor involvement during passive reading comprehension and found that motor involvement was involved in both languages, but the L1 showed stronger activation and involvement, which might reveal a shortcoming in the way we teach L2. They interpreted these differences as coming down to the reality that the L1 is learned in diverse real-world contexts compared to the artificial context commonly found in a classroom, and thus the L2 semantic representation is weaker and less rich. The question for language instructors is, could this difference be affecting acquisition and retention as well?
In foreign language instruction, the coupling of action, perception, and emotion with language is often neglected in the classroom due to: 1) the belief that language is computational and disembodied and thus learning vocabulary is like inputting codes into a mind’s database and body movement is unnecessary, and perhaps even disruptive, in the classroom; and/or 2) the artificial environment of the classroom makes it extremely difficult to naturally couple language to the body. Yet, as pointed out in this article, there is accumulating evidence to suggest that language is highly dependent on the body for semantic representation. The question then is what can instructors do to bring the body into the language classroom? The premise of an embodied learning approach is that greater sensorimotor learning engagement will lead to greater learning of the language, as a consequence, the following are some practical considerations:
- Provide learners with a rich multisensory learning environment that has the potential to leave behind experiential traces in the perceptual systems that can provide deeper encoding of the target language–this includes visual, auditory, tactile, and, if possible, even olfactory and gustatory stimuli.
- Utilize an enactment approach in the classroom, especially when teaching highly action-related words such as manner of movement verbs (e.g., stumble, skip, etc.). Such activities can help learners to retain the vocabulary. That is to say, instead of teaching these action-related words by linguistically explaining their meanings, have students enact them in the classroom.
- Use gestures in the classroom for enhancing vocabulary retention, especially having learners themselves perform the gestures. Gestures, like enactment, have a faciliatory effect on learning vocabulary, as it provides learners a richer representation of the word by leaving behind experiential traces in the motor system.
- Teach abstract language through metaphors by raising learners’ awareness of the grounding of the metaphor to a concrete and embodied concept. This could be done through the use of visual metaphors, especially prominent in advertising and social awareness campaigns. For instance, the ecosystem is relatively abstract and difficult to grasp, so the World Wildlife Fund for Nature recently ran a social awareness campaign that blended the ecosystem with a game of Jenga. A game of Jenga is something that is experienced physically and that most people know and through playing it have developed a deeper understanding of balance. This idea of balance then gets projected onto our understanding of ecosystems and how removing each block (one life form on the planet) may result in the whole structure collapsing, so the distantly related concepts (the ecosystem and Jenga) share some semantic features and thus make it a good metaphor.
In sum, becoming more aware of the interdisciplinary research being done in the area of embodied cognition provides language instructors new perspectives on semantic representation in both the L1 and L2. Such research can inform instructors about the importance of linking language to the body and through enactment, gesture, and multimodal learning, learners can deepen their knowledge of the target language.
Brian J. Birdsell, Ph.D. is an associate professor at Hirosaki University, Japan. His current area of research is on embodied cognition and ways to integrate movement-based approaches to foreign language teaching. He has also published on metaphorical creativity in a foreign language and visual metaphors in advertising as a creative tool to engage the viewer.