Anaphoric distance dependencies in visual narrative structure and processing.

Linguistic syntax has often been claimed as uniquely complex due to features like anaphoric relations and distance dependencies. However, visual narratives of sequential images, like those in comics, have been argued to use sequencing mechanisms analogous to those in language. These narrative structures include "refiner" panels that "zoom in" on the contents of another panel. Similar to anaphora in language, refiners indexically connect inexplicit referential information in one unit (refiner, pronoun) to a more informative "antecedent" elsewhere in the discourse. Also like in language, refiners can follow their antecedents (anaphoric) or precede them (cataphoric), along with having either proximal or distant connections. We here explore the constraints on visual narrative refiners created by modulating these features of order and distance. Experiment 1 examined participants' preferences for where refiners are placed in a sequence using a force-choice test, which revealed that refiners are preferred to follow their antecedents and have proximal distances from them. Experiment 2 then showed that distance dependencies lead to slower self-paced viewing times. Finally, measurements of event-related brain potentials (ERPs) in Experiment 3 revealed that these patterns evoke similar brain responses as referential dependencies in language (i.e., N400, LAN, Nref). Across all three studies, the constraints and (neuro)cognitive responses to refiners parallel those shown to anaphora in language, suggesting domain-general constraints on the sequencing of referential dependencies.

Visual languages and the problems with ideographies: A commentary on Morin.

Morin argues that ideographies are limited because graphic codes lack a capacity for proliferating standardization. However, natural graphic systems display rich standardization and can be placed in sequences using complex combinatorial structures. In contrast, ideographies are not natural, and their limitations lie in their attempts to artificially force a graphic system to behave like a writing system.

Reimagining Language.

Since its inception, the study of language has been a central pillar to Cognitive Science. Despite an "amodal view," where language is thought to "flow into" modalities indiscriminately, speech has always been considered the prototypical form of the linguistic system. However, this view does not hold up to the evidence about language and expressive modalities. While acknowledgment of both the nonvocal modalities and multimodality has grown over the last 40 years in linguistics and psycholinguistics, this has not yet led to a necessary shift in the mainstream linguistic paradigm. Such a shift requires reconfiguring models of language to account for multimodality, and demands a different view on what the linguistic system is and how it works, necessitating a Cognitive Science sensitive to the full richness of human communication.

Remarks on Multimodality: Grammatical Interactions in the Parallel Architecture.

Language is typically embedded in multimodal communication, yet models of linguistic competence do not often incorporate this complexity. Meanwhile, speech, gesture, and/or pictures are each considered as indivisible components of multimodal messages. Here, we argue that multimodality should not be characterized by whole interacting behaviors, but by interactions of similar substructures which permeate across expressive behaviors. These structures comprise a unified architecture and align within Jackendoff's Parallel Architecture: a modality, meaning, and grammar. Because this tripartite architecture persists across modalities, interactions can manifest within each of these substructures. Interactions between modalities alone create correspondences in time (ex. speech with gesture) or space (ex. writing with pictures) of the sensory signals, while multimodal meaning-making balances how modalities carry "semantic weight" for the gist of the whole expression. Here we focus primarily on interactions between grammars, which contrast across two variables: symmetry, related to the complexity of the grammars, and allocation, related to the relative independence of interacting grammars. While independent allocations keep grammars separate, substitutive allocation inserts expressions from one grammar into those of another. We show that substitution operates in interactions between all three natural modalities (vocal, bodily, graphic), and also in unimodal contexts within and between languages, as in codeswitching. Altogether, we argue that unimodal and multimodal expressions arise as emergent interactive states from a unified cognitive architecture, heralding a reconsideration of the "language faculty" itself.

The grammar of emoji? Constraints on communicative pictorial sequencing.

Emoji have become a prominent part of interactive digital communication. Here, we ask the questions: does a grammatical system govern the way people use emoji; and how do emoji interact with the grammar of written text? We conducted two experiments that asked participants to have a digital conversation with each other using only emoji (Experiment 1) or to substitute at least one emoji for a word in the sentences (Experiment 2). First, we found that the emoji-only utterances of participants remained at simplistic levels of patterning, primarily appearing as one-unit utterances (as formulaic expressions or responsive emotions) or as linear sequencing (for example, repeating the same emoji or providing an unordered list of semantically related emoji). Emoji playing grammatical roles (i.e., 'parts-of-speech') were minimal, and showed little consistency in 'word order'. Second, emoji were substituted more for nouns and adjectives than verbs, while also typically conveying nonredundant information to the sentences. These findings suggest that, while emoji may follow tendencies in their interactions with grammatical structure in multimodal text-emoji productions, they lack grammatical structure on their own.

The Role of Shape in Semantic Memory Organization of Objects: An Experimental Study Using PI-Release.

Visual information contributes fundamentally to the process of object categorization. The present study investigated whether the degree of activation of visual information in this process is dependent on the contextual relevance of this information. We used the Proactive Interference (PI-release) paradigm. In four experiments, we manipulated the information by which objects could be categorized and subsequently be retrieved from memory. The pattern of PI-release showed that if objects could be stored and retrieved both by (non-perceptual) semantic and (perceptual) shape information, then shape information was overruled by semantic information. If, however, semantic information could not be (satisfactorily) used to store and retrieve objects, then objects were stored in memory in terms of their shape. The latter effect was found to be strongest for objects from identical semantic categories.

Evidence for the role of shape in mental representations of similes.

People mentally represent the shapes of objects. For instance, the mental representation of an eagle is different when one thinks about a flying or resting eagle. This study examined the role of shape in mental representations of similes (i.e., metaphoric comparisons). We tested the prediction that when people process a simile they will mentally represent the entities of the comparison as having a similar shape. We conducted two experiments in which participants read sentences that either did (experimental sentences) or did not (control sentences) invite comparing two entities. For the experimental sentences, the ground of the comparison was explicit in Experiment 1 ("X has the ability to Z, just like Y") and implicit in Experiment 2 ("X is like Y"). After having read the sentence, participants were presented with line drawings of the two objects, which were either similarly or dissimilarly shaped. They judged whether both objects were mentioned in the preceding sentence. For the experimental sentences, recognition latencies were shorter for similarly shaped objects than for dissimilarly shaped objects. For the control sentences, we did not find such an effect of similarity in shape. These findings suggest that a perceptual symbol of shape is activated when processing similes.

How object shape affects visual metaphor processing.

In order to interpret novel metaphoric relations, we have to construct ad hoc categories under which the metaphorically related concepts can be subsumed. Shape is considered to be one of the primary vehicles of object categorization. Accordingly, shape might play a prominent role in interpreting visual metaphors (i.e., two metaphorically related objects depicted in one visual array). This study explores the role of object shape in visual metaphor interpretation of 10- to 12-year-olds. The experiment shows that participants can produce more correspondences between similarly shaped objects as compared to dissimilarly shaped objects and that they need less thinking time to do so. These findings suggest that similarity in shape facilitates the process of interpreting visual metaphors.

Nonverbatim captioning in Dutch television programs: a text linguistic approach.

In the Netherlands, as in most other European countries, closed captions for the deaf summarize texts rather than render them verbatim. Caption editors argue that in this way television viewers have enough time to both read the text and watch the program. They also claim that the meaning of the original message is properly conveyed. However, many deaf people demand verbatim subtitles so that they have full access to all original information. They claim that vital information is withheld from them as a result of the summarizing process. Linguistic research was conducted in order (a) to identify the type of information that is left out of captioned texts and (b) to determine the effects of nonverbatim captioning on the meaning of the text. The differences between spoken and captioned texts were analyzed on the basis of on a model of coherence relations in discourse. One prominent finding is that summarizing affects coherence relations, making them less explicit and altering the implied meaning.