The Influence of Globally Ungrammatical Local Syntactic Constraints on Real-Time Sentence Comprehension: Evidence From the Visual World Paradigm and Reading.

We investigated the influence of globally ungrammatical local syntactic constraints on sentence comprehension, as well as the corresponding activation of global and local representations. In Experiment 1, participants viewed visual scenes with objects like a carousel and motorbike while hearing sentences with noun phrase (NP) or verb phrase (VP) modifiers like "The girl who likes the man (from London/very much) will ride the carousel." In both cases, "girl" and "ride" predicted carousel as the direct object; however, the locally coherent combination "the man from London will ride…" in NP cases alternatively predicted motorbike. During "ride," local constraints, although ruled out by the global constraints, influenced prediction as strongly as global constraints: While motorbike was fixated less than carousel in VP cases, it was fixated as much as carousel in NP cases. In Experiment 2, these local constraints likewise slowed reading times. We discuss implications for theories of sentence processing.

Event processing in the visual world: Projected motion paths during spoken sentence comprehension.

Motion events in language describe the movement of an entity to another location along a path. In 2 eye-tracking experiments, we found that comprehension of motion events involves the online construction of a spatial mental model that integrates language with the visual world. In Experiment 1, participants listened to sentences describing the movement of an agent to a goal while viewing visual scenes depicting the agent, goal, and empty space in between. Crucially, verbs suggested either upward (e.g., jump) or downward (e.g., crawl) paths. We found that in the rare event of fixating the empty space between the agent and goal, visual attention was biased upward or downward in line with the verb. In Experiment 2, visual scenes depicted a central obstruction, which imposed further constraints on the paths and increased the likelihood of fixating the empty space between the agent and goal. The results from this experiment corroborated and refined the previous findings. Specifically, eye-movement effects started immediately after hearing the verb and were in line with data from an additional mouse-tracking task that encouraged a more explicit spatial reenactment of the motion event. In revealing how event comprehension operates in the visual world, these findings suggest a mental simulation process whereby spatial details of motion events are mapped onto the world through visual attention. The strength and detectability of such effects in overt eye-movements is constrained by the visual world and the fact that perceivers rarely fixate regions of empty space. (PsycINFO Database Record

Knowing what, where, and when: event comprehension in language processing.

We investigated the retrieval of location information, and the deployment of attention to these locations, following (described) event-related location changes. In two visual world experiments, listeners viewed arrays with containers like a bowl, jar, pan, and jug, while hearing sentences like "The boy will pour the sweetcorn from the bowl into the jar, and he will pour the gravy from the pan into the jug. And then, he will taste the sweetcorn". At the discourse-final "sweetcorn", listeners fixated context-relevant "Target" containers most (jar). Crucially, we also observed two forms of competition: listeners fixated containers that were not directly referred to but associated with "sweetcorn" (bowl), and containers that played the same role as Targets (goals of moving events; jug), more than distractors (pan). These results suggest that event-related location changes are encoded across representations that compete for comprehenders' attention, such that listeners retrieve, and fixate, locations that are not referred to in the unfolding language, but related to them via object or role information.

To Dash or to Dawdle: Verb-Associated Speed of Motion Influences Eye Movements during Spoken Sentence Comprehension.

In describing motion events verbs of manner provide information about the speed of agents or objects in those events. We used eye tracking to investigate how inferences about this verb-associated speed of motion would influence the time course of attention to a visual scene that matched an event described in language. Eye movements were recorded as participants heard spoken sentences with verbs that implied a fast ("dash") or slow ("dawdle") movement of an agent towards a goal. These sentences were heard whilst participants concurrently looked at scenes depicting the agent and a path which led to the goal object. Our results indicate a mapping of events onto the visual scene consistent with participants mentally simulating the movement of the agent along the path towards the goal: when the verb implies a slow manner of motion, participants look more often and longer along the path to the goal; when the verb implies a fast manner of motion, participants tend to look earlier at the goal and less on the path. These results reveal that event comprehension in the presence of a visual world involves establishing and dynamically updating the locations of entities in response to linguistic descriptions of events.

Learning individual talkers' structural preferences.

Listeners are often capable of adjusting to the variability contained in individual talkers' (speakers') speech. The vast majority of findings on talker adaptation are concerned with learning the contingency between phonological characteristics and talker identity. In contrast, the present study investigates representations at a more abstract level - the contingency between syntactic attachment style and talker identity. In a 'visual-world' experiment, participants were exposed to semi-realistic scenes depicting several objects (e.g., an adult man, a young girl, a motorbike, a carousel, and other objects) accompanied by a spoken sentence with a structurally ambiguous relative clause (e.g., 'The uncle of the girl who will ride the motorbike/carousel is from France.' In the context of the scene, 'motorbike' suggested the uncle as the agent of the riding, whereas 'carousel' suggested the girl as the agent). For half the experimental items, one version of the sentence was read by one talker, who always uttered sentences that resolved, pragmatically, to the high attachment (the uncle as the agent), and the other by another talker, who always uttered sentences resolving to the low attachment (the girl as the agent). For the other half of the experimental items, both versions were read by a third talker who produced both high and low attachments. It was found that, after exposure to these stimuli, and for new sentences not heard previously, participants learnt to anticipate the 'appropriate' attachment depending on talker identity (with no attachment preference for the talker who produced both attachment types). The data suggest that listeners can learn the relationship between talker identity and abstract, structural, properties of their speech, and that syntactic attachment decisions in comprehension can reflect sensitivity to talker-specific syntactic style.

Discourse-mediation of the mapping between language and the visual world: eye movements and mental representation.

Two experiments explored the mapping between language and mental representations of visual scenes. In both experiments, participants viewed, for example, a scene depicting a woman, a wine glass and bottle on the floor, an empty table, and various other objects. In Experiment 1, participants concurrently heard either 'The woman will put the glass on the table' or 'The woman is too lazy to put the glass on the table'. Subsequently, with the scene unchanged, participants heard that the woman 'will pick up the bottle, and pour the wine carefully into the glass.' Experiment 2 was identical except that the scene was removed before the onset of the spoken language. In both cases, eye movements after 'pour' (anticipating the glass) and at 'glass' reflected the language-determined position of the glass, as either on the floor, or moved onto the table, even though the concurrent (Experiment 1) or prior (Experiment 2) scene showed the glass in its unmoved position on the floor. Language-mediated eye movements thus reflect the real-time mapping of language onto dynamically updateable event-based representations of concurrently or previously seen objects (and their locations).

Integration of syntactic and semantic information in predictive processing: cross-linguistic evidence from German and English.

Two visual-world eyetracking experiments were conducted to investigate whether, how, and when syntactic and semantic constraints are integrated and used to predict properties of subsequent input. Experiment 1 contrasted auditory German constructions such as, "The hare-nominative eats ... (the cabbage-acc)" versus "The hare-accusative eats ... (the fox-nom)," presented with a picture containing a hare, fox, cabbage, and distractor. We found that the probabilities of the eye movements to the cabbage and fox before the onset of NP2 were modulated by the case-marking of NP1, indicating that the case-marking (syntactic) information and verbs' semantic constraints are integrated rapidly enough to predict the most plausible NP2 in the scene. Using English versions of the some stimuli in active/passive voice (Experiment 2), we replicated the same effect, but at a slightly earlier position in the sentence. We discuss the discrepancies in the two Germanic languages in terms of the ease of integrating information across, or within, constituents.