Building a Better Generative Model

Building a Better Generative Model

By: Jason Lowes

Graded readers and other forms of controlled linguistic input have been widely accepted tools to facilitate language learning; however, this controlled input is only part of the equation. Language learners also need clear and accurate internal models of correct linguistic forms to allow them to parse the language they encounter. This seems obvious enough, and I do not believe anyone would consider it a controversial statement. However, if asked why this is true, I’m not sure that many could offer a theory to support their gut feelings. Once again, we turn to the theory of predictive processing to answer this question. In this article, I hope to explain the interplay of top-down (from your brain to the world) and bottom-up (from the world to your brain) connections as considered through the theory of predictive processing. Additionally, I would like to highlight how precise models of the world are essential for learners to recognize the meanings of the situations they encounter. Furthermore, I’d like to show how a better understanding of the role that accurate internal models have in improving comprehension can allow teachers to expand their teaching and materials beyond controlled input. They may adjust their teaching to include ways to help learners develop their accurate models of the world, leading to an improvement in learners’ ability to perceive sensory input.

(Gregory, 1970)

As has been well laid out in the Mindcoolness article “The Bayesian Brain: Predictive Processing,” (2018), the brain’s primary function is to form predictions about the world around it. These predictions help it to determine what is the most adaptive course of action. These predictions are not guesses about the future, such as, “I think that the next person to walk through the door will be wearing a yellow hat.” The predictions being discussed here are predictions of the present. For example, take a look at Figure 1. The moment you see it, it looks like a mess of black and white blobs. However, as the sensory input is received by the eyes, the top-down processes begin to predict what you are encountering. Can you see what is in Figure 1?

One of the many strengths of predictive processing models is the efficiency with which they process the sensory data they encounter. Note that in the previous sentence, I used the word encounter. The brain does not just passively receive sensory data signals; rather, it actively attempts to predict the form of the world around it. It does this by comparing the input with a myriad of models that it has formed of the world. These models are known as generative models, for the predictions generated are based upon the models that the brain has previously stored. This efficiency is realized through many levels of processing. Sensory information does not pass further up the system than the level at which they matched the prediction thus lowering the total load on the system. Information that does not fit the prediction passes to the next level in the hierarchy, where they are met with a new prediction. In this way, input data are matched to the existing generative models in the brain. If some things are not accurately predicted at any of the existing levels of prediction, then they reach the top of the hierarchy and will be used to update the generative model. 

As shown in the diagram of the predictive processing model (Figure 2.), there is a bidirectional flow of information. The bottom-up flow (i.e., sensory data) is not always easy to separate into its relevant and irrelevant parts. With any input, there is likely to be a degree of noise. The less noise there is, the greater the level of precision in the signal. In the Mindcoolness article, the example of seeing something on a foggy day was presented as an example of the input having a lower level of precision. Within the domain of second language learning and teaching, using fully natural texts, or texts presented under challenging circumstances could be considered input with a lower level of precision. The lack of precision may be a result of non-standard linguistic forms, such as slang or from overly formal, noun-heavy spoken forms which overload the listener’s working memory, or just of the use of unfamiliar vocabulary. Another factor that could alter the input’s relative precision is the level of environmental (including cultural) information included in the generative model. For example, the meaning of an utterance can shift depending on the context. The sentence, “It’s raining” when spoken by a farmer to harvesters does not mean the same thing as when it is spoken by a man named Noah to his wife. Additionally, it is important to consider that the generative model is not limited to information and knowledge that is in one’s head. Instead, it can be comprised of neural, bodily, and environmental components. That is, the generative models are not limited to what you have in some portion of your brain. They are, like the mind itself, created through an integration of the neural structures of the brain, the embodied cognition of how one’s body is orientated in the world, and the context of the environment in which that body and mind is located. This cognitive assembly is a soft system that forms as needed to create the generative model. Kirchoff and Kiverstein (2019) described it as a “coalition of processing resources that will best get the job at hand [making accurate predictions] done” (p. 91). 

The external inputs (e.g., written and spoken texts) cannot be controlled by the language learner (that is unless they say, “Hey, slow down or speak more simply!” to their interlocutor). What they can control is the content of their generative model. They can form models to suit the contexts that they will encounter. For a language learner to devote years of study to create linguistic generative models that do not prepare them for the language they will face is inefficient. For example, the simple sentence, “I don’t know what you want,” is easy to understand when written or spoken very carefully; however, if it is spoken by a fifteen-year old east coast Canadian boy, it would be a little more challenging to understand. In IPA, it would look something like “aɪ doʊn noʊ wʌ ʧə wɔn.” In normal letters, it would be, “I dunno whacha wan.” The blended sounds and the lost sounds render the language, when spoken and heard, very different than what the written form would suggest. It would be much better for students to study the language they need to form the appropriate generative models. This leaves us with the question: how can this be done?

Extensive exposure to the language, as it is spoken, is an effective, if time-consuming, approach to developing generative models. Native speakers, through their vast experience with the language and culture, develop the models that they need; however, this is not always a feasible option for second language learners. They need a means of developing the generative models that is more efficient than simple trial and error. Therefore, they need instruction and support from their teachers that allows them quickly to see what is correct and what is not. 

All teachers want their students to produce as much language (both written and spoken) as they can. For this reason, there are times when teachers will opt to “just let the students talk,” without giving them the guidance that they need. However, teachers should not forget some direct teaching methods that can be helpful. One effective method is for teachers to explicitly highlight the target language features that the learner may not have even perceived. Concerning listening, pointing out sounds that are usually elided or reduced can help a learner recognize these situations (i.e., add them to their generative model of English sounds) to assist with future similar encounters.

Another form of direct instruction from the teacher is using something called elaborative interrogation. As the name suggests, this is using questions to expand an idea and to form connections between concepts which have, until that point, been disparate. Elaborative interrogation activates prior knowledge and the generative models associated with that knowledge. Using an almost Socratic series of questions, a teacher would be able to add to a learner’s generative models to include linguistic associations and/or cultural understanding. There are two aspects of the expansion that the teacher should aim to achieve. One is the richness of the knowledge. This is the number of interconnections between the prior knowledge and the new knowledge. The second aspect of knowledge that elaborative interrogation seeks to plumb is the distinctiveness of the knowledge. That is, what are the differences among the items of information being considered. Some questions that a teacher might use could include, “What does X remind you of?” or “When you see X, what do you think of?” To explore the differences, questions such as “In X situation, why would you use Y and not Z? Why/Why not?” “In what situations would you use X?” and “Are there any situations where X would not suit? Why not?” Through the expansion of the generative models, these new aspects could then allow more accurate predictions to be made on future occasions.

A final method that teachers should not neglect is providing effective feedback. Clear and timely feedback will enable the learners’ generative models to develop much more quickly than would naturally happen through only trial and error.

Earlier, you looked at Figure 1, and I asked if you could see what was in the picture. I will tell you that it is a dog. It is a dalmatian, and it is looking away from you. Its nose is lowered toward the ground as if it is sniffing something in the center of the picture. Can you see it now?

What I am doing is trying to help you to add a generative model of this picture. Once you see the dog, you will never be able to unsee it. Similarly, once your students have the generative models that they need to understand something, they will always understand that something.

If teachers consciously develop their teaching materials with one eye on how they can expand their students’ generative models of the language and the culture associated with the language/ context, then they will be able to really help their students to improve. Once the models have been instilled through teacher-led examples, teacher- and student-developed inquiry, and activities that encourage noticing and explicit perception, then the students will be better able to comprehend and respond to the world around them.


  • Gregory, R. L. (1970). The intelligent eye. New York, NY: McGraw-Hill Paperbacks.

  • Kirchhoff, M. D., & Kiverstein, J. (2019). Extended consciousness and predictive processing: A third-wave view. Abingdon, UK: Routledge.

  • Thurston, J. B., & Carraher, R. G. (1966). Optical illusions and the visual arts. New York, NY: Van Nostrand Reinhold.

Jason Lowes is a lecturer at Fukuyama University and long-time member of the Brain SIG. His education-related interests include strategies for effective learning, understanding the role of attention in education, and predictive processing. His non-education related interests are exercise, his dog (which is not a dalmatian as in the picture used above), and beer.

Leave a Reply

Your email address will not be published. Required fields are marked *