Prediction and integration during language comprehension.

Friday, September 30th,   2011 [08:30 - 10:30]

SY_01. Prediction and integration during language comprehension

Martin, C. ¹ , Nieuwland, M. . ² & Costa, A. ^{1, 3}

1 Department of Technology -UPF
2 BCBL
3 ICREA

The purpose of this symposium is to present recent advances in psycholinguistics that deal with prediction and integration during language comprehension, two processes that are crucial for rapid and efficient language comprehension. Prediction allows comprehenders to probabilistically pre-activate words (or discourse topics) that are likely to appear in an unfolding sentence. Integration deals involves combining semantic information as contained by sentences with world knowledge and indexical information, such as the identity of the speaker. We will present 6 recent studies with the common aim of revealing the contributions of prediction and integration to how people arrive at the intended meaning of utterances. These studies all employ online measures of written and spoken language comprehension, such as eye-tracking and event-related potentials (ERPs). First, we will discuss how comprehenders build predictions and abandon them when faced with disfluencies in spoken language. Building on the fact that predictions during online language comprehension can lead to the activation of event representations, we will then present evidence that these representations contain general knowledge information associated with the linguistic input, beyond what is immediately relevant for processing the current linguistic input. We will also discuss how the presentation of a concurrent visual scene affects predictions. We will see how ERP responses are modulated by the emotional valence of predictable words, and by predictability from either a factual or a counterfactual context. Finally, we will focus on whether monolingual and bilingual speakers rely on predictive mechanisms in language comprehension in equal amounts.

 

TALKS

SY_01.1 - What the comprehension of spontaneous speech tells us about prediction

Corley, M. ¹ , MacGregor, L. ² & Donaldson, D. ³

1 University of Edinburgh
2 MRC Cognition and Brain Sciences Unit
3 University of Stirling

The human language comprehension system must be robust to the fact that spontaneous speech is riddled with false starts, repetitions, and disfluent hesitations, at an estimated rate of 6 per 100 words (Fox Tree, 1995). Where predictions are made, they must be updated or abandoned as necessary in the face of these disfluencies. Here, we contrast two ways in which prediction could be affected by disfluency. The first is algorithmic", in the sense that the listener makes use of full information about shared knowledge and the speaker's state (e.g., Arnold et al., 2004, 2007) in order to predict what is most likely to be mentioned following a disfluency. However, this view is largely based on evidence from Visual World experiments in which one of a small set of candidate images will ultimately be named. An alternate, heuristic", view derives from a series of ERP experiments (Corley et al., 2007; MacGregor et al., 2009, 2010) in which the magnitude of an N400 response to unpredictable vs. predictable words is attenuated when the target words follow a disfluency, and targets which occur post-disfluency are more likely to be accurately identifed in a later recognition test. Based on this evidence, we argue that listeners do not make specific predictions where the evidence is not good (e.g., where the speaker is disfluent). As a consequence, more attention is allocated to the speech signal, rendering it more memorable, and preventing subsequent acoustic anomalies from further engaging attention (Collard et al., 2008). Once fluent speech resumes, listeners must get `back on track', and we offer a tentative memory-control account of a distinct post-disfluency late positivity (MacGregor et al., 2009, 2010). By focusing on spoken language, we emphasize the interplay of predictive, attentional, and memory processes in robustly comprehending the speaker's intended message.

SY_01.2 - Generalized Event Knowledge is Activated during Online Language Comprehension

McRae, K. ¹ , Metusalem, R. ² , Kutas, M. . ² , Urbach, T. P. ² , Hare, M. ³ & Elman, J. L. ²

1 University of Western Ontario
2 University of California, San Diego
3 Bowling Green State University

People possess a wide range of knowledge regarding real-world events, and recent research has demonstrated that this knowledge plays an important role in guiding prediction and integration during incremental language comprehension. However, the nature of event knowledge activation during online language comprehension is not fully understood. The present study focused on the degree of generality of online event knowledge activation. One possibility is that activation is limited to elements of an event that meet the constraints imposed by the local linguistic context (i.e., the concept predicted to occur at a specific point in the linguistic stream). It is also possible that activation extends more generally to knowledge associated with the event, regardless of local contextual fit. The present study addressed this issue by analyzing event-related brain potentials (ERPs) recorded as participants read three-sentence scenarios describing common real-world events. The final sentence of each scenario contained a target word that was either expected, anomalous and unrelated to the event described, or anomalous but related to the event. Analyses of ERPs elicited by these target types showed differences among the three conditions. The N400 was smallest for the expected word, and largest for the anomalous, event-unrelated word, replicating previous effects of prediction during language comprehension. For the condition that is novel in the present study, a zero-cloze anomalous word that is consistent with the event/scenario, the N400 was both larger than for the expected word, and smaller than for the anomalous, event-unrelated word. These results demonstrate that during the course of comprehension, comprehenders activate general knowledge associated with the described event, even when the specific concept is not an appropriate continuation of the linguistic input at that precise point in time. Thus, generalized event knowledge is available to immediately influence language processing, and it drives predictive processing during language comprehension.

SY_01.3 - Prediction during Situated Language Processing

Joergensen, G. & Altmann, G. T.

University of York

It is generally accepted that prediction plays a fundamental role during language comprehension (as well as during language production and indeed, cognition more generally). Here, we discuss some of the constraints on the predictive process afforded by visual contexts (past and present). A range of studies suggests that when a sentence unfolds in the context of a concurrent scene, the predictions that are generated as the sentence unfolds reflect the structure of the event described by the sentence. Moreover, those predictions reflect an underlying assumption that the participants in the event will be drawn from the concurrent context, with the more plausible participants attracting more attention than the less plausible participants. In a series of eye-tracking studies we explored the role of plausibility in the generation of such predictions, and the manner in which the scene constrained those predictions. Surprisingly, we found that removing the visual scene immediately prior to the unfolding language appeared to remove the assumption that participants in the event would be drawn from the visual context (even though the critical manipulation was simply whether the scene was concurrent, or preceded the language by a few seconds). The data constrain accounts of sentence comprehension by placing limits on the nature of the predictive process in different contexts, and on the role of experience in modulating contextual constraint. The data have implications also for the relationship between the predictive processes that result in the activation of representations anticipating future language input, and the integrative processes that map actual language input onto those previously activated representations.

SY_01.4 - Expect the Worse and You Will Never Be Disappointed: a Language Comprehension Event-Related Potentials Study

Moreno, E. M.

Universidad Complutense de Madrid

Expectations are crucial in everyday life. People respond to good and bad outcomes not only based on the emotional valence of the outcome but also based on what they had `in mind´, what they expected. Language comprehension electrophysiological studies have shown that brains respond to words in context depending on allowed expectations that are made based on semantic memory and world-knowledge constraints. However, personality traits, cognitive bias, and defensive pessimism strategies might go beyond pure factual semantic and world-knowledge constraints and determine what might have been expected in a particular context at a particular point in time. Using the N400 component of event-related potentials as an index of word expectation, we explored how individuals reacted when processing some a priori highly expected emotionally negative and positive word endings in scenarios in which both types of expectations could be violated at random. Violations included switches to unexpected opposite emotional word endings and nonsense. Our results reveal that despite having a similarly high cloze probability, expected positive endings were processed as more 'surprising' than negative ones. Either the negative predictions were rather strongly made or individuals were somehow reluctant to make strong positive predictions in preparation for them to become untrue. By contrast, post-N400 frontal effects elicited by all types of emotionally opposite outcomes were similar regardless of the direction of the emotional switch. Thus, our brainwave study suggests that humans selectively adjust the strength of self-allowed positive and negative expectations before a verbal input comes in. During language comprehension tasks, just as in other life situations, individuals aim to encounter pleasant 'surprises' and minimize the consequences of emotional setbacks.

SY_01.5 - If the real world were irrelevant, so to speak: An ERP study on counterfactual comprehension

Nieuwland, M. S. & Martin, A. E.

Basque Center on Cognition, Brain and Language

Counterfactual comprehension provides an interesting test-case for studying the interaction between real-world knowledge and discourse context because counterfactuals may require keeping in mind both what is true and what happens to be false. Recent event-related potential (ERP) and eye-tracking results suggest that real-world knowledge briefly interferes with counterfactual comprehension, consistent with two-stage accounts of discourse comprehension. In an ERP experiment, we tested whether real-world interference upholds when incoming information is highly predictable from the counterfactual context. Participants read 120 counterfactually true/false statements (“If N.A.S.A. had not developed its Apollo Project, the first country to land on the moon would be Russia/America”) and real-world true/false statements (“Because N.A.S.A. developed its Apollo Project, the first country to land on the moon has been America/Russia"). Based on results from independent pre-tests, counterfactual and real-world statements were matched for critical word expectancy and average truth-value rating. Our hypothesis involved N400 ERP amplitude, which indexes early semantic processing and is sensitive to subtle variations in discourse-semantic fit. If real-world knowledge interferes with counterfactual comprehension despite this strong context, then critical words in counterfactually true statements should evoke larger N400s than counterfactual false statements and real-world true statements. In contrast, if incoming words are mapped onto the most relevant interpretive context without delay and without initial regard to real-world truth-value, then false statements should elicit an N400 effect compared to true statements, for counterfactual and real-world statements alike. Our results support this latter hypothesis: counterfactually false sentences elicited an N400 effect compared to counterfactually true sentences, identical to the N400 effect for real-world sentences. These results argue against interference from real-world knowledge during counterfactual comprehension. Instead, they suggest that incoming words are mapped onto discourse context without any delay if they are sufficiently plausible and predictable given this context, whether factual or counterfactual.

SY_01.6 - How do early and late bilinguals predict words during sentence reading?

Martin, C. ¹ , Thierry, G. ² , Kuipers, J. ² & Costa, A. ^{1, 3}

1 Universitat Pompeu Fabra
2 Bangor University
3 Institució Catalana de Recerca i Estudis Avançats

The goal of the present study was to investigate if being bilingual affects the way people predict words to come when they read sentences. Using Event-Related Potentials (ERPs), DeLong and collaborators have shown that reading a predicted word at the end of a sentence elicits smaller N400 amplitudes than words that are not predicted, albeit being semantically congruent with the sentence context. Interestingly, prediction effects were also observed on the article preceding the final word, i.e., the N400 was more negative for the article ‘an’ when the most expected final word started with a consonant, and inversely for the article ‘a’ (DeLong et al., 2005). In this study, we investigated prediction effects in early English-Welsh bilinguals, late Spanish-English bilinguals and English monolinguals. Participants were asked to read English sentences while undergoing 64-channel ERP recording. Sentences ended with a predicted noun starting with (a) a vowel or (b) a consonant; a non-predicted noun starting with (c) a vowel or (d) a consonant. In monolinguals, we found a significant prediction effect both on the final noun of the sentence and the preceding article. The N400 modulation elicited by the article was similar in early bilinguals and monolinguals. The final noun failed to modulate the N400 in early bilinguals but elicited a significant P600 effect for non-predicted final nouns. As for late bilinguals, no prediction effect was observed on the article while the final noun produced a typical N400 integration effect. We interpret these results as evidence that early bilinguals predict the noun ending a sentence as monolinguals do. This prediction effect has a cost as early bilinguals show a late integration effect on the final noun of the sentence. On the contrary, late bilinguals do not predict words to come during sentence reading.