Investigating the interplay between morphosyntax and event comprehension from the perspective of intersecting object histories.

In a series of sentence-picture verification studies we contrasted, for example, "… choose the balloon with "… inflate the balloon" and "… the inflated balloon" to examine the degree to which different representational components of event representation (specifically, the different object states entailed by the inflating event; minimally, the balloon in its uninflated and inflated states) are jointly activated after state-change verbs and past participles derived from them. Experiments 1 and 2 showed that the initial and end states are both activated after state-change verbs, but that the initial state is considerably less accessible after participles. Experiment 3 showed that intensifier adverbs (e.g., completely) before both state-change verbs and participles further modulate the accessibility of the initial state. And in Experiment 4, we ruled out the possibility that the initial state is accessible only because of the semantic overlap. We conclude that although state-change verbs activate representations of both the initial and end states of their event participants, their accessibility is graded, modulated by the morphosyntactic devices used to describe the event. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

A novel approach to the assessment of higher-order rule learning in male mice.

Historically, the development of valid and reliable methods for assessing higher-order cognitive abilities (e.g., rule learning and transfer) has been difficult in rodent models. To date, limited evidence supports the existence of higher cognitive abilities such as rule generation and complex decision-making in mice, rats, and rabbits. To this end, we sought to develop a task that would require mice to learn and transfer a rule. We trained mice to visually discriminate a series of images (image set, six total) of increasing complexity following three stages: (1) learn a visual target, (2) learn a rule (ignore any new images around the target), and finally (3) apply this rule in abstract form to a comparable but new image set. To evaluate learning for each stage, we measured (1) days (and performance by day) to discriminate the original target at criterion, (2) days (and performance by day) to get back to criterion when images in the set were altered by the introduction of distractors (rule learning), and (3) overall days (and performance by day) to criterion when experienced versus naïve cohorts of mice were tested on the same image set (rule transfer). Twenty-seven wild-type male C57 mice were tested using Bussey-Saksida touchscreen operant conditioning boxes (Lafayette Instruments). Two comparable black-white image sets were delivered sequentially (counterbalanced for order) to two identical cohorts of mice. Results showed that all mice were able to effectively learn their initial target image and could recall it >80 d later. We also found that mice were able to quickly learn and apply a "rule" : Ignore new distractors and continue to identify their visual target embedded in more complex images. The presence of rule learning was supported because performance criterion thresholds were regained much faster than initial learning when distractors were introduced. On the other hand, mice appeared unable to transfer this rule to a new set of stimuli. This is supported because visual discrimination curves for a new image set were no better than an initial (naïve) learning by a matched cohort of mice. Overall results have important implications for phenotyping research and particularly for the modeling of complex disorders in mice.

Online eye tracking and real-time sentence processing: On opportunities and efficacy for capturing psycholinguistic effects of different magnitudes and diversity.

Online research methods have the potential to facilitate equitable accessibility to otherwise-expensive research resources, as well as to more diverse populations and language combinations than currently populate our studies. In psycholinguistics specifically, webcam-based eye tracking is emerging as a powerful online tool capable of capturing sentence processing effects in real time. The present paper asks whether webcam-based eye tracking provides the necessary granularity to replicate effects-crucially both large and small-that tracker-based eye tracking has shown. Using the Gorilla Experiment Builder platform, this study set out to replicate two psycholinguistic effects: a robust one, the verb semantic constraint effect, first reported in Altmann and Kamide,  Cognition 73(3), 247-264 (1999), and a smaller one, the lexical interference effect, first examined by Kukona et al. Journal of Experimental Psychology: Learning, Memory, and Cognition, 40(2), 326 (2014). Webcam-based eye tracking was able to replicate both effects, thus showing that its functionality is not limited to large effects. Moreover, the paper also reports two approaches to computing statistical power and discusses the differences in their outputs. Beyond discussing several important methodological, theoretical, and practical implications, we offer some further technical details and advice on how to implement webcam-based eye-tracking studies. We believe that the advent of webcam-based eye tracking, at least in respect of the visual world paradigm, will kickstart a new wave of more diverse studies with more diverse populations.

Encoding and inhibition of arbitrary episodic context with abstract concepts.

Context is critical for conceptual processing, but the mechanism underpinning its encoding and reinstantiation during abstract concept processing is unclear. Context may be especially important for abstract concepts-we investigated whether episodic context is recruited differently when processing abstract compared with concrete concepts. Experiments 1 and 2 presented abstract and concrete words in arbitrary contexts at encoding (Experiment 1: red/green colored frames; Experiment 2: male/female voices). Recognition memory for these contexts was worse for abstract concepts. Again using frame color and voice as arbitrary contexts, respectively, Experiments 3 and 4 presented words from encoding in the same or different context at test to determine whether there was a greater recognition memory benefit for abstract versus concrete concepts when the context was unchanged between encoding and test. Instead, abstract concepts were less likely to be remembered when context was retained. This suggests that at least some types of episodic context-when arbitrary-are attended less, and may even be inhibited, when processing abstract concepts. In Experiment 5, we utilized a context-spatial location-which (as we show) tends to be relevant during real-world processing of abstract concepts. We presented words in different locations, preserving or changing location at test. Location retention conferred a recognition memory advantage for abstract concepts. Thus, episodic context may be encoded with abstract concepts when context is relevant to real-world processing. The systematic contexts necessary for understanding abstract concepts may lead to arbitrary context inhibition, but greater attention to contexts that tend to be more relevant during real-world processing.

Finding event structure in time: What recurrent neural networks can tell us about event structure in mind.

Under a theory of event representations that defines events as dynamic changes in objects across both time and space, as in the proposal of Intersecting Object Histories (Altmann & Ekves, 2019), the encoding of changes in state is a fundamental first step in building richer representations of events. In other words, there is an inherent dynamic that is captured by our knowledge of events. In the present study, we evaluated the degree to which this dynamic was inferable from just the linguistic signal, without access to visual, sensory, and embodied experience, using recurrent neural networks (RNNs). Recent literature exploring RNNs has largely focused on syntactic and semantic knowledge. We extend this domain of investigation to representations of events within RNNs. In three studies, we find preliminary evidence that RNNs capture, in their internal representations, the extent to which objects change states; for example, that chopping an onion changes the onion by more than just peeling the onion. Moreover, the temporal relationship between state changes is encoded to some extent. We found RNNs are sensitive to how chopping an onion and then weighing it, or first weighing it, entails the onion that is being weighed being in a different state depending on the adverb. Our final study explored what factors influence the propagation of these rudimentary event representations forward into subsequent sentences. We conclude that while there is much still to be learned about the abilities of RNNs (especially in respect of the extent to which they encode objects as specific tokens), we still do not know what are the equivalent representational dynamics in humans. That is, we take the perspective that the exploration of computational models points us to important questions about the nature of the human mind.

Psychology of cleansing through the prism of intersecting object histories.

We link cleansing effects to contemporary cognitive theories via an account of event representation (intersecting object histories) that provides an explicit, neurally plausible mechanism for encoding objects (e.g., the self) and their associations (with other entities) across time. It explains separation as resulting from weakening associations between the self in the present and the self in the past.

The activation of object-state representations during online language comprehension.

Understanding the time-course of event knowledge activation is crucial for theories of language comprehension. We report two experiments using the 'visual world paradigm' (VWP) that investigated the dynamic mapping between object-state representations and real-time language processing. In Experiment 1, participants heard sentences that described events resulting in either a substantial change of state (e.g. The chef will chop the onion) or a minimal change of state (e.g. The chef will weigh the onion). Concurrently, they viewed pictures depicting two versions of the target object (e.g., an onion) corresponding to the intact and changed states, and two unrelated distractors. A second sentence referred to the object with either a backward or a forward shift in event time (e.g. But first/And then, he will smell the onion). In Experiment 2, Degree of Change was manipulated by using different nouns in the first sentence (e.g. The girl will stomp on thepenny/egg). The second sentence was similar to the ones used in Experiment 1 (e.g., But first/And then, she will look atthe penny/egg). The results from both experiments showed that participants looked more at the 'appropriate' state of the object that matched the language context, but the shift of visual attention emerged only when the object name was heard. Our findings suggest that situationally appropriate object representations do trigger eye movements to the corresponding states of the target object, but inappropriate representations are not necessarily eliminated from consideration until the language forces it.

Language as a mental travel guide.

Gilead et al.'s approach to human cognition places abstraction and prediction at the heart of "mental travel" under a "representational diversity" perspective that embraces foundational concepts in cognitive science. But, it gives insufficient credit to the possibility that the process of abstraction produces a gradient, and underestimates the importance of a highly influential domain in predictive cognition: language, and related, the emergence of experientially based structure through time.

Situational systematicity: A role for schema in understanding the differences between abstract and concrete concepts.

concepts differ from concrete concepts in several ways. Here, we focus on what we refer to as situational systematicity: The objects and relations that constitute an abstract concept (e.g., justice) are more dispersed through space and time than are those that typically constitute a concrete concept (e.g., chair); a larger set of objects and relations constitute an abstract concept than a concrete one; and exactly which objects and relations constitute a concept is more context-dependent for abstract concepts. We thus refer to abstract concepts as having low situational systematicity. We contend that situational systematicity, rather than abstractness per se, is a critical determinant of the cognitive, behavioural, and neural phenomena associated with concepts. Further, viewing concepts as schema provides insight into (i) the situation-based dynamics of concept learning and representation and (ii) the functional significance of the brain regions and their interactions that comprise the schema control network.

The influence of state change on object representations in language comprehension.

To understand language people form mental representations of described situations. Linguistic cues are known to influence these representations. In the present study, participants were asked to verify whether the object presented in a picture was mentioned in the preceding words. Crucially, the picture either showed an intact original state or a modified state of an object. Our results showed that the end state of the target object influenced verification responses. When no linguistic context was provided, participants responded faster to the original state of the object compared to the changed state (Experiment 1). However, when linguistic context was provided, participants responded faster to the modified state when it matched, rather than mismatched, the expected outcome of the described event (Experiment 2 and Experiment 3). Interestingly, as for the original state, the match/mismatch effects were only revealed after reading the past tense (Experiment 2) sentences but not the future-tense sentences (Experiment 3). Our findings highlight the need to take account of the dynamics of event representation in language comprehension that captures the interplay between general semantic knowledge about objects and the episodic knowledge introduced by the sentential context.

Events as intersecting object histories: A new theory of event representation.

We offer a new account of event representation based on those aspects of object representation that encode an object's history, and which convey the distinct states that an object has experienced across time-minimally reflecting the before and after of whatever changes the object undergoes as an event unfolds. Our intention is to account for the content of event representations. For an event that can be described as "the chef chopped the onion," the event as a whole is defined by the changes in state and location, across time, of the onion, the chef, and any instruments that (might have) mediated the interaction between the chef and the onion. Thus, we maintain that events are encoded as "ensembles of intersecting object histories" in which one or more objects change state. Our approach requires not just the distinction between object types and object tokens, but also between tokens and token-states (e.g., between that specific onion and its different states before, during, and after the chopping). These distinctions require an account of how object tokens are represented within the context of episodic and semantic memory, and how distinct object states are bound into a single object identity. We shall argue that the theoretical pieces, and their neural instantiation, are in place to develop a unified account of event representation in which such representation is simply a consequence of the mechanism for generating object tokens, their histories, and the binding of one to the other. (PsycINFO Database Record (c) 2019 APA, all rights reserved).

Unfolding meaning in context: The dynamics of conceptual similarity.

How are relationships between concepts affected by the interplay between short-term contextual constraints and long-term conceptual knowledge? Across two studies we investigate the consequence of changes in visual context for the dynamics of conceptual processing. Participants' eye movements were tracked as they viewed a visual depiction of e.g. a canary in a birdcage (Experiment 1), or a canary and three unrelated objects, each in its own quadrant (Experiment 2). In both studies participants heard either a semantically and contextually similar "robin" (a bird; similar size), an equally semantically similar but not contextually similar "stork" (a bird; bigger than a canary, incompatible with the birdcage), or unrelated "tent". The changing patterns of fixations across time indicated first, that the visual context strongly influenced the eye movements such that, in the context of a birdcage, early on (by word offset) hearing "robin" engendered more looks to the canary than hearing "stork" or "tent" (which engendered the same number of looks), unlike in the context of unrelated objects (in which case "robin" and "stork" engendered equivalent looks to the canary, and more than did "tent"). Second, within the 500 ms post-word-offset eye movements in both experiments converged onto a common pattern (more looks to the canary after "robin" than after "stork", and for both more than after "tent"). We interpret these findings as indicative of the dynamics of activation within semantic memory accessed via pictures and via words, and reflecting the complex interaction between systems representing context-independent and context-dependent conceptual knowledge driven by predictive processing.

Recalling what was where when seeing nothing there.

So-called "looks-at-nothing" have previously been used to show that recalling what also elicits the recall of where this was. Here, we present evidence from an eye-tracking study which shows that disrupting looks to "there" does not disrupt recalling what was there, nor do (anticipatory) looks to "there" facilitate recalling what was there. Therefore, our results suggest that recalling where does not recall what.

Abstraction and generalization in statistical learning: implications for the relationship between semantic types and episodic tokens.

Statistical approaches to emergent knowledge have tended to focus on the process by which experience of individual episodes accumulates into generalizable experience across episodes. However, there is a seemingly opposite, but equally critical, process that such experience affords: the process by which, from a space of types (e.g. onions-a semantic class that develops through exposure to individual episodes involving individual onions), we can perceive or create, on-the-fly, a specific token (a specific onion, perhaps one that is chopped) in the absence of any prior perceptual experience with that specific token. This article reviews a selection of statistical learning studies that lead to the speculation that this process-the generation, on the basis of semantic memory, of a novel episodic representation-is itself an instance of a statistical, in fact associative, process. The article concludes that the same processes that enable statistical abstraction across individual episodes to form semantic memories also enable the generation, from those semantic memories, of representations that correspond to individual tokens, and of novel episodic facts about those tokens. Statistical learning is a window onto these deeper processes that underpin cognition.This article is part of the themed issue 'New frontiers for statistical learning in the cognitive sciences'.

Competition between Mutually Exclusive Object States in Event Comprehension.

Successful language comprehension requires one to correctly match symbols in an utterance to referents in the world, but the rampant ambiguity present in that mapping poses a challenge. Sometimes the ambiguity lies in which of two (or more) types of things in the world are under discussion (i.e., lexical ambiguity); however, even a word with a single sense can have an ambiguous referent. This ambiguity occurs when an object can exist in multiple states. Here, we consider two cases in which the presence of multiple object states may render a single-sense word ambiguous. In the first case, one must disambiguate between two states of a single object token in a short discourse. In the second case, the discourse establishes two different tokens of the object category. Both cases involve multiple object states: These states are mutually exclusive in the first case, whereas in the second case, these states can logically exist at the same time. We use fMRI to contrast same-token and different-token discourses, using responses in left posterior ventrolateral prefrontal cortex (pVLPFC) as an indicator of conflict. Because the left pVLPFC is sensitive to competition between multiple, incompatible representations, we predicted that state ambiguity should engender conflict only when those states are mutually exclusive. Indeed, we find evidence of conflict in same-token, but not different-token, discourses. Our data support a theory of left pVLPFC function in which general conflict resolution mechanisms are engaged to select between multiple incompatible representations that arise in many kinds of ambiguity present in language.

A cortical network for the encoding of object change.

Understanding events often requires recognizing unique stimuli as alternative, mutually exclusive states of the same persisting object. Using fMRI, we examined the neural mechanisms underlying the representation of object states and object-state changes. We found that subjective ratings of visual dissimilarity between a depicted object and an unseen alternative state of that object predicted the corresponding multivoxel pattern dissimilarity in early visual cortex during an imagery task, while late visual cortex patterns tracked dissimilarity among distinct objects. Early visual cortex pattern dissimilarity for object states in turn predicted the level of activation in an area of left posterior ventrolateral prefrontal cortex (pVLPFC) most responsive to conflict in a separate Stroop color-word interference task, and an area of left ventral posterior parietal cortex (vPPC) implicated in the relational binding of semantic features. We suggest that when visualizing object states, representational content instantiated across early and late visual cortex is modulated by processes in left pVLPFC and left vPPC that support selection and binding, and ultimately event comprehension.

Knowing what, where, and when: event comprehension in language processing.

We investigated the retrieval of location information, and the deployment of attention to these locations, following (described) event-related location changes. In two visual world experiments, listeners viewed arrays with containers like a bowl, jar, pan, and jug, while hearing sentences like "The boy will pour the sweetcorn from the bowl into the jar, and he will pour the gravy from the pan into the jug. And then, he will taste the sweetcorn". At the discourse-final "sweetcorn", listeners fixated context-relevant "Target" containers most (jar). Crucially, we also observed two forms of competition: listeners fixated containers that were not directly referred to but associated with "sweetcorn" (bowl), and containers that played the same role as Targets (goals of moving events; jug), more than distractors (pan). These results suggest that event-related location changes are encoded across representations that compete for comprehenders' attention, such that listeners retrieve, and fixate, locations that are not referred to in the unfolding language, but related to them via object or role information.

Language-guided visual processing affects reasoning: the role of referential and spatial anchoring.

Language is more than a source of information for accessing higher-order conceptual knowledge. Indeed, language may determine how people perceive and interpret visual stimuli. Visual processing in linguistic contexts, for instance, mirrors language processing and happens incrementally, rather than through variously-oriented fixations over a particular scene. The consequences of this atypical visual processing are yet to be determined. Here, we investigated the integration of visual and linguistic input during a reasoning task. Participants listened to sentences containing conjunctions or disjunctions (Nancy examined an ant and/or a cloud) and looked at visual scenes containing two pictures that either matched or mismatched the nouns. Degree of match between nouns and pictures (referential anchoring) and between their expected and actual spatial positions (spatial anchoring) affected fixations as well as judgments. We conclude that language induces incremental processing of visual scenes, which in turn becomes susceptible to reasoning errors during the language-meaning verification process.

The effect of object state-changes on event processing: do objects compete with themselves?

When an object is described as changing state during an event, do the representations of those states compete? The distinct states they represent cannot coexist at any one moment in time, yet each representation must be retrievable at the cost of suppressing the other possible object states. We used functional magnetic resonance imaging of human participants to test whether such competition does occur, and whether this competition between object states recruits brain areas sensitive to other forms of conflict. In Experiment 1, the same object was changed either substantially or minimally by one of two actions. In Experiment 2, the same action either substantially or minimally changed one of two objects. On a subject-specific basis, we identified voxels most responsive to conflict in a Stroop color-word interference task. Voxels in left posterior ventrolateral prefrontal cortex most responsive to Stroop conflict were also responsive to our object state-change manipulation, and were not responsive to the imageability of the described action. In contrast, voxels in left middle frontal gyrus responsive to Stroop conflict were not responsive even to language, and voxels in left middle temporal gyrus that were responsive to language and imageability were not responsive to object state-change. Results suggest that, when representing object state-change, multiple incompatible representations of an object compete, and the greater the difference between the initial state and the end state of an object, the greater the conflict.