The electrophysiology of lexical prediction of emoji and text.

As emoji often appear naturally alongside text in utterances, they provide a way to study how prediction unfolds in multimodal sentences in direct comparison to unimodal sentences. In this experiment, participants (N = 40) read sentences in which the sentence-final noun appeared in either word form or emoji form, a between-subjects manipulation. The experiment featured both high constraint sentences and low constraint sentences to examine how the lexical processing of emoji interacts with prediction processes in sentence comprehension. Two well-established ERP components linked to lexical processing and prediction - the N400 and the Late Frontal Positivity - are investigated for sentence-final words and emoji to assess whether, to what extent, and in what linguistic contexts emoji are processed like words. Results indicate that the expected effects, namely an N400 effect to an implausible lexical item compared to a plausible one and an LFP effect to an unexpected lexical item compared to an expected one, emerged for both words and emoji. This paper discusses the similarities and differences between the stimulus types and constraint conditions, contextualized within theories of linguistic prediction, ERP components, and a multimodal lexicon.

Anaphoric distance dependencies in visual narrative structure and processing.

Linguistic syntax has often been claimed as uniquely complex due to features like anaphoric relations and distance dependencies. However, visual narratives of sequential images, like those in comics, have been argued to use sequencing mechanisms analogous to those in language. These narrative structures include "refiner" panels that "zoom in" on the contents of another panel. Similar to anaphora in language, refiners indexically connect inexplicit referential information in one unit (refiner, pronoun) to a more informative "antecedent" elsewhere in the discourse. Also like in language, refiners can follow their antecedents (anaphoric) or precede them (cataphoric), along with having either proximal or distant connections. We here explore the constraints on visual narrative refiners created by modulating these features of order and distance. Experiment 1 examined participants' preferences for where refiners are placed in a sequence using a force-choice test, which revealed that refiners are preferred to follow their antecedents and have proximal distances from them. Experiment 2 then showed that distance dependencies lead to slower self-paced viewing times. Finally, measurements of event-related brain potentials (ERPs) in Experiment 3 revealed that these patterns evoke similar brain responses as referential dependencies in language (i.e., N400, LAN, Nref). Across all three studies, the constraints and (neuro)cognitive responses to refiners parallel those shown to anaphora in language, suggesting domain-general constraints on the sequencing of referential dependencies.

Individual differences in the neural dynamics of visual narrative comprehension: The effects of proficiency and age of acquisition.

Understanding visual narrative sequences, as found in comics, is known to recruit similar cognitive mechanisms to verbal language. As measured by event-related potentials (ERPs), these manifest as initial negativities (N400, LAN) and subsequent positivities (P600). While these components are thought to index discrete processing stages, they differentially arise across participants for any given stimulus. In language contexts, proficiency modulates brain responses, with smaller N400 effects and larger P600 effects appearing with increasing proficiency. In visual narratives, recent work has also emphasized the role of proficiency in neural response patterns. We thus explored whether individual differences in proficiency modulate neural responses to visual narrative sequencing in similar ways as in language. We combined ERP data from 12 studies examining semantic and/or grammatical processing of visual narrative sequences. Using linear mixed effects modeling, we demonstrate differential effects of visual language proficiency and "age of acquisition" on N400 and P600 responses. Our results align with those reported in language contexts, providing further evidence for the similarity of linguistic and visual narrative processing, and emphasize the role of both proficiency and age of acquisition in visual narrative comprehension.

Background check: cross-cultural differences in the spatial context of comic scenes.

Cognitive research points towards cultural differences in the way people perceive and express scenes. Whereas people from Western cultures focus more on focal objects, those from East Asia have been shown to focus on the surrounding context. This paper examines whether these cultural differences are expressed in complex multimodal media such as comics. We compared annotated panels across comics from six countries to examine how backgrounds convey contextual information of scenes in explicit or implicit ways. Compared to Western comics from the United States and Spain, East Asian comics from Japan and China expressed the context of scenes more implicitly. In addition, Nigerian comics moderately emulated American comics in background use, while Russian comics emulated Japanese manga, consistent with their visual styles. The six countries grouped together based on whether they employed more explicit strategies such as detailed, depicted backgrounds, or implicit strategies such as leaving the background empty. These cultural differences in background use can be attributed to both cognitive patterns of attention and comics' graphic styles. Altogether, this study provides support for cultural differences in attention manifesting in visual narratives, and elucidates how spatial relationships are depicted in visual narratives across cultures.

The Meaning of Motion Lines?: A Review of Theoretical and Empirical Research on Static Depiction of Motion.

Static depiction of motion, particularly lines trailing behind a mover, has long been of interest in the psychology literature. Empirical research has demonstrated that these "motion lines" benefited motion comprehension in static images by disambiguating the direction of movement. Yet, there is no consensus on how those lines derive their meaning. In this article, we review three accounts suggesting different interpretations of what motion lines represent. While a perceptual account considers motion lines originating from motion streaks in the primary visual cortex, metaphorical and lexical accounts propose them as graphical conventions that should be learned-either through resemblance to sensory experiences (e.g., natural path marks) or directly being mapped to a conceptual category of paths. To contrast these three accounts, we integrate empirical research on motion lines and their understanding. Overall, developmental, proficiency, and cross-cultural variances indicate that the understanding of motion lines is neither innate nor universal, thus providing less support for lines having a purely perceptual origin. Rather, we argue the empirical findings suggest that motion lines are not iconic depictions of visual percepts but are graphical conventions indexing conceptual path information, which need to be learned and encoded in a visual lexicon.

Visual languages and the problems with ideographies: A commentary on Morin.

Morin argues that ideographies are limited because graphic codes lack a capacity for proliferating standardization. However, natural graphic systems display rich standardization and can be placed in sequences using complex combinatorial structures. In contrast, ideographies are not natural, and their limitations lie in their attempts to artificially force a graphic system to behave like a writing system.

Sequences in harmony: Cognitive interactions between musical and visual narrative structure.

From film and television to graphic storytelling, tonal music can accompany visual narratives in a variety of contexts. The apprehension of both musical and narrative sequences involves temporal categories in ordered patterning, which raises an interesting question: Do musical progressions and visual narratives rely on shared sequence processing mechanisms? If this is the case, then cues from music and sequential static images, when presented simultaneously, should interact during audiovisual online processing. We tested this question by measuring reaction times to target picture panels appearing in visual narrative (comic strip) sequences, which were presented panel by panel and synchronized with musical chord progressions. Image sequences were either coherent narratives or incoherent (random) panels, and they were aligned with musical accompaniment consisting of coherent tonal chord progressions or non-tonal (unrelated) chords. Reaction times were faster for target images in coherent sequences than incoherent sequences, and even faster for coherent images with tonal accompaniment than non-tonal chords. This indicated an interaction between sequential structures across domains. We take these results as evidence for a shared, domain-general sequence processing mechanism operating across music and visual narrative.

The framing of subjectivity: Point-of-view in a cross-cultural analysis of comics.

In visual narratives like comics, the most overt form of perspective-taking comes in panels that directly depict the viewpoints of characters in the scene. We therefore examined these subjective viewpoint panels (also known as point-of-view panels) in a corpus of over 300 annotated comics from Asia, Europe, and the United States. In line with predictions that Japanese manga use a more 'subjective' storytelling style than other comics, we found that more manga use subjective panels than other comics, with high proportions of subjective panels also found in Chinese, French, and American comics. In addition, panels with more 'focal' framing, i.e. micro panels showing close ups and/or amorphic panels showing views of the environment, had higher proportions of subjective panels than panels showing wider views of scenes. These findings further show that empirical corpus analyses provide evidence of cross-cultural variation and reveal relationships across structures in the visual languages of comics.

The Lexicon of Emoji? Conventionality Modulates Processing of Emoji.

Emoji have been ubiquitous in communication for over a decade, yet how they derive meaning remains underexplored. Here, we examine an aspect fundamental to linguistic meaning-making: the degree to which emoji have conventional lexicalized meanings and whether that conventionalization affects processing in real-time. Experiment 1 establishes a range of meaning agreement levels across emoji within a population; Experiment 2 measures accuracy and response times to word-emoji pairings in a match/mismatch task. In this experiment, we found that accuracy and response time both correlated significantly with the level of population-wide meaning agreement from Experiment 1, suggesting that lexical access of single emoji may be comparable to that of words, even out of context. This is consistent with theories of a multimodal lexicon that stores links between meaning, structure, and modality in long-term memory. Altogether, these findings suggest that emoji can allow a range of entrenched, lexicalized representations.

Semantic processing of verbal narratives compared to semantic processing of visual narratives: an ERP study of school-aged children.

There is a misconception that pictures are easy to comprehend, which is problematic in pedagogical practices that include pictures. For example, if a child has difficulties with verbal narration to picture sequences, it may be interpreted as specific to spoken language even though the child may have additional difficulties with comprehension of visual narratives in the form of picture sequences. The purpose of the present study was therefore to increase our understanding of semantic processing in the pictorial domain in relation to semantic processing in the verbal domain, focusing on 9-13 years-old children with typical language development. To this end, we measured electrical brain responses (event related potentials, ERPs) in 17 children to (i) pictures (panels) that were predicted versus unpredicted in sequences of panels that conveyed visual narratives and (ii) words that were predicted versus unpredicted in sentences that conveyed verbal narratives. Results demonstrated similarities as there were no significant difference in the magnitude of the N400 effect across domains. The only difference between domains was the predicted difference in distribution, that is, a more posterior N400 effect in the verbal domain than in the pictorial domain. The study contributes to an increased understanding of the complexity of processing of visual narratives and its shared features with processing of verbal narratives, which should be considered in pedagogical practices.

Reimagining Language.

Since its inception, the study of language has been a central pillar to Cognitive Science. Despite an "amodal view," where language is thought to "flow into" modalities indiscriminately, speech has always been considered the prototypical form of the linguistic system. However, this view does not hold up to the evidence about language and expressive modalities. While acknowledgment of both the nonvocal modalities and multimodality has grown over the last 40 years in linguistics and psycholinguistics, this has not yet led to a necessary shift in the mainstream linguistic paradigm. Such a shift requires reconfiguring models of language to account for multimodality, and demands a different view on what the linguistic system is and how it works, necessitating a Cognitive Science sensitive to the full richness of human communication.

An electrophysiological investigation of co-referential processes in visual narrative comprehension.

Visual narratives make use of various means to convey referential and co-referential meaning, so comprehenders must recognize that different depictions across sequential images represent the same character(s). In this study, we investigated how the order in which different types of panels in visual sequences are presented affects how the unfolding narrative is comprehended. Participants viewed short comic strips while their electroencephalogram (EEG) was recorded. We analyzed evoked and induced EEG activity elicited by both full panels (showing a full character) and refiner panels (showing only a zoom of that full panel), and took into account whether they preceded or followed the panel to which they were co-referentially related (i.e., were cataphoric or anaphoric). We found that full panels elicited both larger N300 amplitude and increased gamma-band power compared to refiner panels. Anaphoric panels elicited a sustained negativity compared to cataphoric panels, which appeared to be sensitive to the referential status of the anaphoric panel. In the time-frequency domain, anaphoric panels elicited reduced 8-12 Hz alpha power and increased 45-65 Hz gamma-band power compared to cataphoric panels. These findings are consistent with models in which the processes involved in visual narrative comprehension partially overlap with those in language comprehension.

Meaning above (and in) the head: Combinatorial visual morphology from comics and emoji.

Compositionality is a primary feature of language, but graphics can also create combinatorial meaning, like with items above faces (e.g., lightbulbs to mean inspiration). We posit that these "upfixes" (i.e., upwards affixes) involve a productive schema enabling both stored and novel face-upfix dyads. In two experiments, participants viewed either conventional (e.g., lightbulb) or unconventional (e.g., clover-leaves) upfixes with faces which either matched (e.g., lightbulb/smile) or mismatched (e.g., lightbulb/frown). In Experiment 1, matching dyads sponsored higher comprehensibility ratings and faster response times, modulated by conventionality. In Experiment 2, event-related brain potentials (ERPs) revealed conventional upfixes, regardless of matching, evoked larger N250s, indicating perceptual expertise, but mismatching and unconventional dyads elicited larger semantic processing costs (N400) than conventional-matching dyads. Yet mismatches evoked a late negativity, suggesting congruent novel dyads remained construable compared with violations. These results support that combinatorial graphics involve a constrained productive schema, similar to the lexicon of language.

Linguistic typology of motion events in visual narratives.

Languages use different strategies to encode motion. Some use particles or "satellites" to describe a path of motion (Satellite-framed or S-languages like English), while others typically use the main verb to convey the path information (Verb-framed or V-languages like French). We here ask: might this linguistic variation lead to differences in the way paths are depicted in visual narratives like comics? We analyzed a corpus of 85 comics originally created by speakers of S-languages (comics from the United States, China, Germany) and V-languages (France, Japan, Korea) for both their depictions of path segments (source, route, and goal) and the visual cues signaling these paths and manner information (e.g., motion lines and postures). Panels from S-languages depicted more path segments overall, especially routes, than those from V-languages, but panels from V-languages more often isolated path segments into their own panels. Additionally, comics from S-languages depicted more motion cues than those from V-languages, and this linguistic typology also interacted with panel framing. Despite these differences across typological groups, analysis of individual countries' comics showed more nuanced variation than a simple S-V dichotomy. These findings suggest a possible influence of spoken language structure on depicting motion events in visual narratives and their sequencing.

The electrophysiology of audiovisual processing in visual narratives in adolescents with autism spectrum disorder.

We investigated the semantic processing of the multimodal audiovisual combination of visual narratives with auditory descriptive words and auditory sounds in individuals with ASD. To this aim, we recorded ERPs to critical auditory words and sounds associated with events in visual narrative that were either semantically congruent or incongruent with the climactic visual event. A similar N400 effect was found both in adolescents with ASD and neurotypical adolescents (ages 9-16) when accessing different types of auditory information (i.e. words and sounds) into a visual narrative. This result might suggest that verbal information processing in ASD adolescents could be facilitated by direct association with meaningful visual information. In addition, we observed differences in scalp distribution of later brain responses between ASD and neurotypical adolescents. This finding might suggest ASD adolescents differ from neurotypical adolescents during the processing of the multimodal combination of visual narratives with auditory information at later stages of the process. In conclusion, the semantic processing of verbal information, typically impaired in individuals with ASD, can be facilitated when embedded into a meaningful visual information.

A starring role for inference in the neurocognition of visual narratives.

Research in verbal and visual narratives has often emphasized backward-looking inferences, where absent information is subsequently inferred. However, comics use conventions like star-shaped "action stars" where a reader knows events are undepicted at that moment, rather than omitted entirely. We contrasted the event-related brain potentials (ERPs) to visual narratives depicting an explicit event, an action star, or a "noise" panel of scrambled lines. Both action stars and noise panels evoked large N400s compared to explicit-events (300-500 ms), but action stars and noise panels then differed in their later effects (500-900 ms). Action stars elicited sustained negativities and P600s, which could indicate further interpretive processes and integration of meaning into a mental model, while noise panels evoked late frontal positivities possibly indexing that they were improbable narrative units. Nevertheless, panels following action stars and noise panels both evoked late sustained negativities, implying further inferential processing. Inference in visual narratives thus uses cascading mechanisms resembling those in language processing that differ based on the inferential techniques.

Remarks on Multimodality: Grammatical Interactions in the Parallel Architecture.

Language is typically embedded in multimodal communication, yet models of linguistic competence do not often incorporate this complexity. Meanwhile, speech, gesture, and/or pictures are each considered as indivisible components of multimodal messages. Here, we argue that multimodality should not be characterized by whole interacting behaviors, but by interactions of similar substructures which permeate across expressive behaviors. These structures comprise a unified architecture and align within Jackendoff's Parallel Architecture: a modality, meaning, and grammar. Because this tripartite architecture persists across modalities, interactions can manifest within each of these substructures. Interactions between modalities alone create correspondences in time (ex. speech with gesture) or space (ex. writing with pictures) of the sensory signals, while multimodal meaning-making balances how modalities carry "semantic weight" for the gist of the whole expression. Here we focus primarily on interactions between grammars, which contrast across two variables: symmetry, related to the complexity of the grammars, and allocation, related to the relative independence of interacting grammars. While independent allocations keep grammars separate, substitutive allocation inserts expressions from one grammar into those of another. We show that substitution operates in interactions between all three natural modalities (vocal, bodily, graphic), and also in unimodal contexts within and between languages, as in codeswitching. Altogether, we argue that unimodal and multimodal expressions arise as emergent interactive states from a unified cognitive architecture, heralding a reconsideration of the "language faculty" itself.

Zooming in on visual narrative comprehension.

The comprehension of visual narratives requires paying attention to certain elements and integrating them across a sequence of images. To study this process, we developed a new approach that modified comic strips according to where observers looked while viewing each sequence. Across three self-paced experiments, we presented sequences of six panels that were sometimes automatically "zoomed-in" or re-framed in order to highlight parts of the image that had been fixated by another group of observers. Fixation zoom panels were rated as easier to understand and produced viewing times more similar to the original comic than panels modified to contain non-fixated or incongruous regions. When a single panel depicting the start of an action was cropped to show only the most fixated region, viewing times were similar to the original narrative despite the reduced information. Modifying such panels also had an impact on the viewing time on subsequent panels, both when zoomed in and when regions were highlighted through an "inset" panel. These findings demonstrate that fixations in a visual narrative are guided to informative elements, and that these elements influence both the current panel and the processing of the sequence.

Zooming in on the cognitive neuroscience of visual narrative.

Visual narratives like comics and films often shift between showing full scenes and close, zoomed-in viewpoints. These zooms are similar to the "spotlight of attention" cast across a visual scene in perception. We here measured ERPs to visual narratives (comic strips) that used zoomed-in and full-scene panels either throughout the whole sequence context or at specific critical panels. Zoomed-in panels were automatically generated on the basis of fixations from prior participants' eye movements to the crucial content of panels (Foulsham & Cohn, 2020). We found that these fixation panels evoked a smaller N300 than full-scenes, indicative of reduced cost for object identification, but that they also evoked a slightly larger amplitude N400 response, suggesting a greater cost for accessing semantic memory with constrained content. Panels in sequences where fixation panels persisted across all positions of the sequence also evoked larger posterior P600s, implying that constrained views required more updating or revision processes throughout the sequence. Altogether, these findings suggest that constraining a visual scene to its crucial parts triggers various processes related not only to the density of its information but also to its integration into a sequential context.

Predictability modulates neurocognitive semantic processing of non-verbal narratives.

Predictability is known to modulate semantic processing in language, but it is unclear to what extent this applies for other modalities. Here we ask whether similar cognitive processes are at play in predicting upcoming events in a non-verbal visual narrative. Typically developing adults viewed comics sequences in which a target panel was highly predictable ("high cloze"), less predictable ("low cloze"), or incongruent with the preceding narrative context ("anomalous") during EEG recording. High and low predictable sequences were determined by a pretest where participants assessed "what happened next?", resulting in cloze probability scores for sequence outcomes comparable to those used to measure predictability in sentence processing. Through both factorial and correlational analyses, we show a significant modulation of neural responses by cloze such that N400 effects are diminished as a target panel in a comic sequence becomes more predictable. Predictability thus appears to play a similar role in non-verbal comprehension of sequential images as in language comprehension, providing further evidence for the domain generality of semantic processing in the brain.