Kinect-ing the Dots: Using Motion-Capture Technology to Distinguish Sign Language Linguistic From Gestural Expressions.

Just as vocalization proceeds in a continuous stream in speech, so too do movements of the hands, face, and body in sign languages. Here, we use motion-capture technology to distinguish lexical signs in sign language from other common types of expression in the signing stream. One type of expression is constructed action, the enactment of (aspects of) referents and events by (parts of) the body. Another is classifier constructions, the manual representation of analogue and gradient motions and locations simultaneously with specified referent morphemes. The term signing is commonly used for all of these, but we show that not all visual signals in sign languages are of the same type. In this study of Israeli Sign Language, we use motion capture to show that the motion of lexical signs differs significantly along several kinematic parameters from that of the two other modes of expression: constructed action and the classifier forms. In so doing, we show how motion-capture technology can help to define the universal linguistic category "word," and to distinguish it from the expressive gestural elements that are commonly found across sign languages.

A stepping stone to compositionality in chimpanzee communication.

Compositionality refers to a structural property of human language, according to which the meaning of a complex expression is a function of the meaning of its parts and the way they are combined. Compositionality is a defining characteristic of all human language, spoken and signed. Comparative research into the emergence of human language aims at identifying precursors to such key features of human language in the communication of other primates. While it is known that chimpanzees, our closest relatives, produce a variety of gestures, facial expressions and vocalizations in interactions with their group members, little is known about how these signals combine simultaneously. Therefore, the aim of the current study is to investigate whether there is evidence for compositional structures in the communication of chimpanzees. We investigated two semi-wild groups of chimpanzees, with focus on their manual gestures and their combinations with facial expressions across different social contexts. If there are compositional structures in chimpanzee communication, adding a facial expression to a gesture should convey a different message than the gesture alone, a difference that we expect to be measurable by the recipient's response. Furthermore, we expect context-dependent usage of these combinations. Based on a form-based coding procedure of the collected video footage, we identified two frequently used manual gestures (stretched arm gesture and bent arm gesture) and two facial expression (bared teeth face and funneled lip face). We analyzed whether the recipients' response varied depending on the signaler's usage of a given gesture + face combination and the context in which these were used. Overall, our results suggest that, in positive contexts, such as play or grooming, specific combinations had an impact on the likelihood of the occurrence of particular responses. Specifically, adding a bared teeth face to a gesture either increased the likelihood of affiliative behavior (for stretched arm gesture) or eliminated the bias toward an affiliative response (for bent arm gesture). We show for the first time that the components under study are recombinable, and that different combinations elicit different responses, a property that we refer to as componentiality. Yet our data do not suggest that the components have consistent meanings in each combination-a defining property of compositionality. We propose that the componentiality exhibited in this study represents a necessary stepping stone toward a fully evolved compositional system.

The Body as Evidence for the Nature of Language.

Taking its cue from sign languages, this paper proposes that the recruitment and composition of body actions provide evidence for key properties of language and its emergence. Adopting the view that compositionality is the fundamental organizing property of language, we show first that actions of the hands, face, head, and torso in sign languages directly reflect linguistic components, and illuminate certain aspects of compositional organization among them that are relevant for all languages, signed and spoken. Studies of emerging sign languages strengthen the approach by showing that the gradual recruitment of bodily articulators for linguistic functions directly maps the way in which a new language increases in complexity and efficiency over time. While compositional communication is almost exclusively restricted to humans, it is not restricted to language. In the spontaneous, intense emotional displays of athletes, different emotional states are correlated with actions of particular face and body features and feature groupings. These findings indicate a much more ancient communicative compositional capacity, and support a paradigm that includes visible body actions in the quest for core linguistic properties and their origins.

Compositionality in the language of emotion.

Emotions are signaled by complex arrays of face and body actions. The main point of contention in contemporary treatments is whether these arrays are discrete, holistic constellations reflecting emotion categories, or whether they are compositional-comprised of smaller components, each of which contributes some aspect of emotion to the complex whole. We address this question by investigating spontaneous face and body displays of athletes and place it in the wider context of human communicative signals and, in particular, of language. A defining property of human language is compositionality-the ability to combine and recombine a relatively small number of elements to create a vast number of complex meaningful expressions, and to interpret them. We ask whether this property of language can be discerned in a more ancient communicative system: intense emotional displays. In an experiment, participants interpreted a range of emotions and their strengths in pictures of athletes who had just won or lost a competition. By matching participants' judgements with minutely coded features of face and body, we find evidence for compositionality. The distribution of participants' responses indicates that most of the athletes' face and body features contribute to displays of dominance or submission. More particular emotional components related, for example, to positive valence (e.g. happy) or goal obstruction (e.g. frustrated), were also found to significantly correlate with certain face and body features. We propose that the combination of features linked to broader components (i.e, dominant or submissive) and to more particular emotions (e.g, happy or frustrated) reflects more complex emotional states. In sum, we find that the corporeal expression of intense, unfiltered emotion has compositional properties, potentially providing an ancient scaffolding upon which, millions of years later, the abstract and constrained compositional system of human language could build.

Constructing Complexity in a Young Sign Language.

A universally acknowledged, core property of language is its complexity, at each level of structure - sounds, words, phrases, clauses, utterances, and higher levels of discourse. How does this complexity originate and develop in a language? We cannot fully answer this question from spoken languages, since they are all thousands of years old or descended from old languages. However, sign languages of deaf communities can arise at any time and provide empirical data for testing hypotheses related to the emergence of language complexity. An added advantage of the signed modality is a correspondence between visible physical articulations and linguistic structures, providing a more transparent view of linguistic complexity and its emergence (Sandler, 2012). These essential characteristics of sign languages allow us to address the issue of emerging complexity by documenting the use of the body for linguistic purposes. We look at three types of discourse relations of increasing complexity motivated by research on spoken languages - additive, symmetric, and asymmetric (Mann and Thompson, 1988; Sanders et al., 1992). Each relation type can connect units at two different levels: within propositions (simpler) and across propositions (more complex). We hypothesized that these relations provide a measure for charting the time course of emergence of complexity, from simplest to most complex, in a new sign language. We test this hypothesis on Israeli Sign Language (ISL), a young language, some of whose earliest users are still available for recording. Taking advantage of the unique relation in sign languages between bodily articulations and linguistic form, we study fifteen ISL signers from three generations, and demonstrate that the predictions indeed hold. We also find that younger signers tend to converge on more systematic marking of relations, that they use fewer articulators for a given linguistic function than older signers, and that the form of articulations becomes reduced, as the language matures. Mapping discourse relations to the bodily expression of linguistic components across age groups reveals how simpler, less constrained, and more gesture-like expressions, become language.

The effect of being human and the basis of grammatical word order: Insights from novel communication systems and young sign languages.

This study identifies a central factor that gives rise to the different word orders found in the world's languages. In the last decade, a new window on this long-standing question has been provided by data from young sign languages and invented gesture systems. Previous work has assumed that word order in both invented gesture systems and young sign languages is driven by the need to encode the semantic/syntactic roles of the verb's arguments. Based on the responses of six groups of participants, three groups of hearing participants who invented a gestural system on the spot, and three groups of signers of relatively young sign languages, we identify a major factor in determining word order in the production of utterances in novel and young communication systems, not suggested by previous accounts, namely the salience of the arguments in terms of their human/animacy properties: human arguments are introduced before inanimate arguments ('human first'). This conclusion is based on the difference in word order patterns found between responses to depicted simple events that vary as to whether both subject and object are human or whether the subject is human and the object inanimate. We argue that these differential patterns can be accounted for uniformly by the 'human first' principle. Our analysis accounts for the prevalence of SOV order in clauses with an inanimate object in all groups (replicating results of previous separate studies of deaf signers and hearing gesturers) and the prevalence of both SOV and OSV in clauses with a human object elicited from the three groups of participants who have the least interference from another linguistic system (nonliterate deaf signers who have had little or no exposure to another language). It also provides an explanation for the basic status of SOV order suggested by other studies, as well as the scarcity of the OSV order in languages of the world, despite its appearance in novel communication systems. The broadest implication of this study is that the basic cognitive distinction between humans and inanimate entities is a crucial factor in setting the wheels of word ordering in motion.

The emergence of embedded structure: insights from Kafr Qasem Sign Language.

This paper introduces data from Kafr Qasem Sign Language (KQSL), an as-yet undescribed sign language, and identifies the earliest indications of embedding in this young language. Using semantic and prosodic criteria, we identify predicates that form a constituent with a noun, functionally modifying it. We analyze these structures as instances of embedded predicates, exhibiting what can be regarded as very early stages in the development of subordinate constructions, and argue that these structures may bear directly on questions about the development of embedding and subordination in language in general. Deutscher (2009) argues persuasively that nominalization of a verb is the first step-and the crucial step-toward syntactic embedding. It has also been suggested that prosodic marking may precede syntactic marking of embedding (Mithun, 2009). However, the relevant data from the stage at which embedding first emerges have not previously been available. KQSL might be the missing piece of the puzzle: a language in which a noun can be modified by an additional predicate, forming a proposition within a proposition, sustained entirely by prosodic means.

Competing iconicities in the structure of languages.

The paper examines the role that iconicity plays in the structuring of grammars. Two main points are argued for: (a) Grammar does not necessarily suppress iconicity; rather, iconicity and grammar can enjoy a congenial relation in that iconicity can play an active role in the structuring of grammars. (b) Iconicity is not monolithic. There are different types of iconicity and languages take advantage of the possibilities afforded by them. We examine the interaction between iconicity and grammar by focusing on the ways in which sign languages employ the physical body of the signer as a rich iconic resource for encoding a variety of grammatical notions. We show that the body can play three different roles in iconic forms in sign languages: it can be used as a naming device where body parts represent body parts; it can represent the subject argument of verbal signs, and it can stand for first person. These strategies interact and sometimes compete in the languages under study. Each language resolves these competitions differently, which results in different grammars and grammatical structures. The investigation of the ways in which grammar and iconicity interact in these languages provides insight into the nature of both systems.

THE PHONOLOGICAL ORGANIZATION OF SIGN LANGUAGES.

Visually perceivable and movable parts of the body - the hands, facial features, head, and upper body - are the articulators of sign language. It is through these articulators that that words are formed, constrained, and contrasted with one another, and that prosody is conveyed. This article provides an overview of the way in which phonology is organized in the alternative modality of sign language.

The gradual emergence of phonological form in a new language.

The division of linguistic structure into a meaningless (phonological) level and a meaningful level of morphemes and words is considered a basic design feature of human language. Although established sign languages, like spoken languages, have been shown to be characterized by this bifurcation, no information has been available about the way in which such structure arises. We report here on a newly emerging sign language, Al-Sayyid Bedouin Sign Language, which functions as a full language but in which a phonological level of structure has not yet emerged. Early indications of formal regularities provide clues to the way in which phonological structure may develop over time.

The emergence of complexity in prosody and syntax.

The relation between prosody and syntax is investigated here by tracing the emergence of each in a new language, Al-Sayyid Bedouin Sign Language. We analyze the structure of narratives of four signers of this language: two older second generation signers, and two about 15 years younger. We find that younger signers produce prosodic cues to dependency between semantically related constituents, e.g., the two clauses of conditionals, revealing a type and degree of complexity in their language that is not frequent in that of the older pair. In these younger signers, several rhythmic and (facial) intonational cues are aligned at constituent boundaries, indicating the emergence of a grammatical system. There are no overt syntactic markers (such as complementizers) to relate clauses; prosody is the only clue. But this prosodic complexity is matched by syntactic complexity inside propositions in the younger signers, who are more likely to use pronouns as abstract grammatical markers of arguments, and to combine predicates with their arguments within in a constituent. As the prosodic means emerge for identifying constituent types and signaling dependency relations between them, the constituents themselves become increasingly complex. Finally, our study shows that the emergence of grammatical complexity is gradual.

The Uniformity and Diversity of Language: Evidence from Sign Language.

Evidence from sign language strongly supports three positions: (1) language is a coherent system with universal properties; (2) sign languages diverge from spoken languages in some aspects of their structure; and (3) domain-external factors can be identified that account for some crucial aspects of language structure -- uniform and diverse -- in both modalities. Assuming that any of these positions excludes the others defeats the purpose of the enterprise.

Visual intonation in the prosody of a sign language.

While visual signals that accompany spoken language serve to augment the communicative message, the same visual ingredients form the substance of the linguistic system in sign languages. This article provides an analysis of visual signals that comprise part of the intonational system of a sign language. The system is conveyed mainly by particular actions of the upper face, and is shown to pattern linguistically and predictably in Israeli Sign Language. Its components, aligned with prosodic constituents, are associated with particular but general meanings and may be combined to derive complex meanings. The Brow Raise component is functionally comparable to H tones, signaling continuation and dependency, and characterizing yes/no questions and the if-clause of conditionals, for example. The component Squint instructs the addressee to retrieve information that is not readily accessible, and characterizes relative clauses, topics, and other structures. The details of the componential analysis proposed here explain why the two components together co-occur on such seemingly diverse structures as yes/no questions about mutually retrievable information and counterfactual conditionals. Like auditorily perceived intonational melodies, the visual intonational arrays in sign language provide a subtle, intricately structured, and meaningful accompaniment to the words and sentences of language.

Symbiotic symbolization by hand and mouth in sign language.

Current conceptions of human language include a gestural component in the communicative event. However, determining how the linguistic and gestural signals are distinguished, how each is structured, and how they interact still poses a challenge for the construction of a comprehensive model of language. This study attempts to advance our understanding of these issues with evidence from sign language. The study adopts McNeill's criteria for distinguishing gestures from the linguistically organized signal, and provides a brief description of the linguistic organization of sign languages. Focusing on the subcategory of iconic gestures, the paper shows that signers create iconic gestures with the mouth, an articulator that acts symbiotically with the hands to complement the linguistic description of objects and events. A new distinction between the mimetic replica and the iconic symbol accounts for the nature and distribution of iconic mouth gestures and distinguishes them from mimetic uses of the mouth. Symbiotic symbolization by hand and mouth is a salient feature of human language, regardless of whether the primary linguistic modality is oral or manual. Speakers gesture with their hands, and signers gesture with their mouths.

The Roots of Linguistic Organization in a New Language.

It is possible for a language to emerge with no direct linguistic history or outside linguistic influence. Al-Sayyid Bedouin Sign Language (ABSL) arose about 70 years ago in a small, insular community with a high incidence of profound prelingual neurosensory deafness. In ABSL, we have been able to identify the beginnings of phonology, morphology, syntax, and prosody. The linguistic elements we find in ABSL are not exclusively holistic, nor are they all compositional, but a combination of both. We do not, however, find in ABSL certain features that have been posited as essential even for a proto-language. ABSL has a highly regular syntax as well as word-internal compounding, also highly regular but quite distinct from syntax in its patterns. ABSL, however, has no discernable word-internal structure of the kind observed in more mature sign languages: no spatially organized morphology and no evident duality of phonological patterning.

Body as subject().

The notion of subject in human language has a privileged status relative to other arguments. This special status is manifested in the behavior of subjects at the morphological, syntactic, semantic and discourse levels. Here we bring evidence that subjects have privileged status at the lexical level as well, by analyzing lexicalization patterns of verbs in three different sign languages. Our analysis shows that the sublexical structure of iconic signs denoting state of affairs in these languages manifests an inherent pattern of form-meaning correspondence: the signer's body consistently represents one argument of the verb, the subject. The hands, moving in relation to the body, represent all other components of the event - including all other arguments. This analysis shows that sign languages provide novel evidence in support of the centrality of the notion of subject in human language. It also solves a typological puzzle about the apparent primacy of object in sign language verb agreement, a primacy not usually found in spoken languages, in which subject agreement ranks higher. Our analysis suggests that the subject argument is represented by the body and is part of the lexical structure of the verb. Because it is always inherently represented in the structure of the sign, the subject is more basic than the object, and tolerates the omission of agreement morphology.

The emergence of grammar: systematic structure in a new language.

This report contains a linguistic description of a language created spontaneously without any apparent external influence in a stable existing community. We describe the syntactic structure of Al-Sayyid Bedouin Sign Language, a language that has arisen in the last 70 years in an isolated endogamous community with a high incidence of nonsyndromic, genetically recessive, profound prelingual neurosensory deafness. In the space of one generation from its inception, systematic grammatical structure has emerged in the language. Going beyond a conventionalized list of words for actions, objects, people, characteristics, and so on, a systematic way of marking the grammatical relations among those elements has appeared in the form of highly regular word order. These systematic structures cannot be attributed to influence from other languages, because the particular word orders that appear in Al-Sayyid Bedouin Sign Language differ from those found both in the ambient spoken languages in the community and in the other sign language found predominantly in the surrounding area. Therefore, the emerging grammatical structures should be regarded as an independent development within the language.