Seeking a reading machine for the blind and discovering the speech code.

A machine that can read printed material to the blind became a priority at the end of World War II with the appointment of a U.S. Government committee to instigate research on sensory aids to improve the lot of blinded veterans. The committee chose Haskins Laboratories to lead a multisite research program. Initially, Haskins researchers overestimated the capacities of users to learn an acoustic code based on the letters of a text, resulting in unsuitable designs. Progress was slow because the researchers clung to a mistaken view that speech is a sound alphabet and because of persisting gaps in man-machine technology. The tortuous route to a practical reading machine transformed the scientific understanding of speech perception and reading at Haskins Labs and elsewhere, leading to novel lines of basic research and new technologies. Research at Haskins Laboratories made valuable contributions in clarifying the physical basis of speech. Researchers recognized that coarticulatory overlap eliminated the possibility of alphabet-like discrete acoustic segments in speech. This work advanced the study of speech perception and contributed to our understanding of the relation of speech perception to production. Basic findings on speech enabled the development of speech synthesis, part science and part technology, essential for development of a reading machine, which has found many applications. Findings on the nature of speech further stimulated a new understanding of word recognition in reading across languages and scripts and contributed to our understanding of reading development and reading disabilities.

Talking as doing: Language forms and public language.

I discuss language forms as the primary means that language communities provide to enable public language use. As such, they are adapted to public use most notably in being linguistically significant vocal tract actions, not the categories in the mind as proposed in phonological theories. Their primary function is to serve as vehicles for production of syntactically structured sequences of words. However, more than that, phonological actions themselves do work in public language use. In particular, they foster interpersonal coordination in social activities. An intriguing property of language forms that likely reflects their emergence in social communicative activities is that phonological forms that should be meaningless (in order to serve their role in the openness of language at the level of the lexicon) are not wholly meaningless. In fact, the form-meaning "rift" is bridged bidirectionally: The smallest language forms are meaningful, and the meanings of lexical language forms generally inhere, in part, in their embodiment by understanders.

An ecological alternative to a "sad response": public language use transcends the boundaries of the skin.

Embedding theories of language production and comprehension in theories of action-perception is realistic and highlights that production and comprehension processes are interleaved. However, layers of internal models that repeatedly predict future linguistic actions and perceptions are implausible. I sketch an ecological alternative whereby perceiver/actors are modeled as dynamical systems coupled to one another and to the environment.

Functional activation for imitation of seen and heard speech.

This study examined fMRI activation when perceivers either passively observed or observed and imitated matched or mismatched audiovisual ("McGurk") speech stimuli. Greater activation was observed in the inferior frontal gyrus (IFG) overall for imitation than for perception of audiovisual speech and for imitation of the McGurk-type mismatched stimuli than matched audiovisual stimuli. This unique activation in the IFG during imitation of incongruent audiovisual speech may reflect activation associated with direct matching of incongruent auditory and visual stimuli or conflict between category responses. This study provides novel data about the underlying neurobiology of imitation and integration of AV speech.

Transposition Effects in an Aksharic Writing System: The Case of Hindi.

We tested for transposition effects (TEs) in Hindi (a Modern Indo-Aryan language) using unprimed lexical decision. TEs are defined as less accurate and slower responses to transposed-nonwords (e.g., , formed from base-word ) than corresponding replaced-nonwords (e.g., ). In Hindi's orthography, letters map transparently to phonemes (except schwa), but the letters are arranged into "akshars," (n]V>) which encode open syllables. This formal characteristic makes Hindi's orthography typologically "aksharic." We used TEs to determine whether the orthography's typological units, letters and akshars, are also functional units for readers. We conducted three visual word recognition experiments with adult readers whose native language was Hindi. In Experiment 1, we found TEs for consonant () and matra (, a vowel diacritic) letters, using different stimulus sets for each type of transposition. In the next two experiments, we used the same base words to form all of the transposed and replaced items. In Experiment 2, we replicated the findings of Experiment 1 in a different stimulus set; additionally, we found TEs for transpositions between a letter and a akshar. In Experiment 3, we replicated results of the first two experiments by finding TEs for both consonants and matras in another stimulus set; additionally, we found similar TEs for akshars. These results show that and letters are functional units for Hindi readers; the transposition results for akshars are tentative. TEs for letters show that the aksharic grouping of letters does not prevent readers from decoding the constituent letters of akshars. Hindi is read alphabetically.

Locus equations are an acoustic expression of articulator synergy.

The study investigated the articulatory basis of locus equations, regression lines relating F2 at the start of a Consonant-Vowel (CV) transition to F2 at the middle of the vowel, with C fixed and V varying. Several studies have shown that consonants of different places of articulation have locus equation slopes that descend from labial to velar to alveolar, and intercept magnitudes that increase in the opposite order. Using formulas from the theory of bivariate regression that express regression slopes and intercepts in terms of standard deviations and averages of the variables, it is shown that the slope directly encodes a well-established measure of coarticulation resistance. It is also shown that intercepts are directly related to the degree to which the tongue body assists the formation of the constriction for the consonant. Moreover, it is shown that the linearity of locus equations and the linear relation between locus equation slopes and intercepts originates in linearity in articulation between the horizontal position of the tongue dorsum in the consonant and to that in the vowel. It is concluded that slopes and intercepts of acoustic locus equations are measures of articulator synergy.

The reality of phonological forms: a reply to Port.

I suggest four grounds on which an argument can be made that phonological language forms are not merely emergent properties of the public language use of members of a language community. They are: 1) the existence of spontaneous errors of speech production in which whole consonants or vowels misorder or are replaced; 2) the necessary existence of language "particles" used by individual language users in order for words to be able to be coined; 3) the remarkable effectiveness of alphabetic writing systems and the tight coupling among skilled readers of orthographic and phonological language forms; 4) the finding that, by late infancy, children have discovered phonological constancies despite phonetic variation.

A critical examination of the spectral contrast account of compensation for coarticulation.

Vocal tract gestures for adjacent phones overlap temporally, rendering the acoustic speech signal highly context dependent. For example, following a segment with an anterior place of articulation, a posterior segment's place of articulation is pulled frontward, and listeners' category boundaries shift appropriately. Some theories assume that listeners perceptually attune or compensate for coarticulatory context. An alternative is that shifts result from spectral contrast. Indeed, shifts occur when speech precursors are replaced by pure tones, frequency matched to the formant offset at the assumed locus of contrast (Lotto & Kluender, 1998). However, tone analogues differ from natural formants in several ways, raising the possibility that conditions for contrast may not exist in natural speech. When we matched tones to natural formant intensities and trajectories, boundary shifts diminished. When we presented only the critical spectral region of natural speech tokens, no compensation was observed. These results suggest that conditions for spectral contrast do not exist in typical speech.

Articulatory constraints on interpersonal postural coordination.

Cooperative conversation has been shown to foster interpersonal postural coordination. The authors investigated whether such coordination is mediated by the influence of articulation on postural sway. In Experiment 1, talkers produced words in synchrony or in alternation, as the authors varied speaking rate and word similarity. Greater shared postural activity was found for the faster speaking rate. In Experiment 2, the authors demonstrated that shared postural activity also increases when individuals speak the same words or speak words that have similar stress patterns. However, this increase in shared postural activity is present only when participants' data are compared with those of their partner, who was present during the task, but not when compared with the data of a member of a different pair speaking the same word sequences as those of the original partner. The authors' findings suggest that interpersonal postural coordination observed during conversation is mediated by convergent speaking patterns.

Phonetic drift in Spanish-English bilinguals: Experiment and a self-organizing model.

Studies of speech accommodation provide evidence for change in use of language structures beyond the critical/sensitive period. For example, Sancier and Fowler (1997) found changes in the voice-onset-times (VOTs) of both languages of a Portuguese-English bilingual as a function of her language context. Though accommodation has been studied widely within a monolingual context, it has received less attention in and between the languages of bilinguals. We tested whether these findings of phonetic accommodation, speech accommodation at the phonetic level, would generalize to a sample of Spanish-English bilinguals. We recorded participants reading Spanish and English sentences after 3-4 months in the US and after 2-4 weeks in a Spanish speaking country and measured the VOTs of their voiceless plosives. Our statistical analyses show that participants' English VOTs drifted towards those of the ambient language, but their Spanish VOTs did not. We found considerable variation in the extent of individual participants' drift in English. Further analysis of our results suggested that native-likeness of L2 VOTs and extent of active language use predict the extent of drift. We provide a model based on principles of self-organizing dynamical systems to account for our Spanish-English phonetic drift findings and the Portuguese-English findings.

The motor theory of speech perception reviewed.

More than 50 years after the appearance of the motor theory of speech perception, it is timely to evaluate its three main claims that (1) speech processing is special, (2) perceiving speech is perceiving gestures, and (3) the motor system is recruited for perceiving speech. We argue that to the extent that it can be evaluated, the first claim is likely false. As for the second claim, we review findings that support it and argue that although each of these findings may be explained by alternative accounts, the claim provides a single coherent account. As for the third claim, we review findings in the literature that support it at different levels of generality and argue that the claim anticipated a theme that has become widespread in cognitive science.

Visual influences on perception of speech and nonspeech vocal-tract events.

We report four experiments designed to determine whether visual information affects judgments of acoustically-specified nonspeech events as well as speech events (the "McGurk effect"). Previous findings have shown only weak McGurk effects for nonspeech stimuli, whereas strong effects are found for consonants. We used click sounds that serve as consonants in some African languages, but that are perceived as nonspeech by American English listeners. We found a significant McGurk effect for clicks presented in isolation that was much smaller than that found for stop-consonant-vowel syllables. In subsequent experiments, we found strong McGurk effects, comparable to those found for English syllables, for click-vowel syllables, and weak effects, comparable to those found for isolated clicks, for excised release bursts of stop consonants presented in isolation. We interpret these findings as evidence that the potential contributions of speech-specific processes on the McGurk effect are limited, and discuss the results in relation to current explanations for the McGurk effect.

"Perception of the speech code" revisited: Speech is alphabetic after all.

We revisit an article, "Perception of the Speech Code" (PSC), published in this journal 50 years ago (Liberman, Cooper, Shankweiler, & Studdert-Kennedy, 1967) and address one of its legacies concerning the status of phonetic segments, which persists in theories of speech today. In the perspective of PSC, segments both exist (in language as known) and do not exist (in articulation or the acoustic speech signal). Findings interpreted as showing that speech is not a sound alphabet, but, rather, phonemes are encoded in the signal, coupled with findings that listeners perceive articulation, led to the motor theory of speech perception, a highly controversial legacy of PSC. However, a second legacy, the paradoxical perspective on segments has been mostly unquestioned. We remove the paradox by offering an alternative supported by converging evidence that segments exist in language both as known and as used. We support the existence of segments in both language knowledge and in production by showing that phonetic segments are articulatory and dynamic and that coarticulation does not eliminate them. We show that segments leave an acoustic signature that listeners can track. This suggests that speech is well-adapted to public communication in facilitating, not creating a barrier to, exchange of language forms.

Mutual interpersonal postural constraints are involved in cooperative conversation.

The research was designed to evaluate interpersonal coordination during conversation with a new measurement tool. The experiment uses an analysis based on recurrence strategies, known as cross recurrence quantification, to evaluate the shared activity between 2 postural time series in reconstructed phase space. Pairs of participants were found to share more locations in phase space (greater recurrence) in conditions where they were conversing with one another to solve a puzzle task than in conditions in which they convened with others. The trajectories of pairs of participants also showed less divergence when they conversed with each other than when they conversed with others well. This is offered as objective evidence of interpersonal coordination of postural sway in the context of a cooperative verbal task.

Information for coarticulation: Static signal properties or formant dynamics?

Perception of a speech segment changes depending on properties of surrounding segments in a phenomenon called compensation for coarticulation (Mann, 1980). The nature of information that drives these perceptual changes is a matter of debate. One account attributes perceptual shifts to low-level auditory system contrast effects based on static portions of the signal (e.g., third formant [F3] center or average frequency; Lotto & Kluender, 1998). An alternative account is that listeners' perceptual shifts result from listeners attuning to the acoustic effects of gestural overlap and that this information for coarticulation is necessarily dynamic (Fowler, 2006). In a pair of experiments, we used sinewave speech precursors to investigate the nature of information for compensation for coarticulation. In Experiment 1, as expected by both accounts, we found that sinewave speech precursors produce shifts in following segments. In Experiment 2, we investigated whether effects in Experiment 1 were driven by static F3 offsets of sinewave speech precursors, or by dynamic relationships among their formants. We temporally reversed F1 and F2 in sinewave precursors, preserving static F3 offset and average F1, F2 and F3 frequencies, but disrupting dynamic formant relationships. Despite having identical F3s, selectively reversed precursors produced effects that were significantly smaller and restricted to only a small portion of the continuum. We conclude that dynamic formant relations rather than static properties of the precursor provide information for compensation for coarticulation.

Listening with a foreign-accent: The interlanguage speech intelligibility benefit in Mandarin speakers of English.

This study examined the intelligibility of native and Mandarin-accented English speech for native English and native Mandarin listeners. In the latter group, it also examined the role of the language environment and English proficiency. Three groups of listeners were tested: native English listeners (NE), Mandarin-speaking Chinese listeners in the US (M-US) and Mandarin listeners in Beijing, China (M-BJ). As a group, M-US and M-BJ listeners were matched on English proficiency and age of acquisition. A nonword transcription task was used. Identification accuracy for word-final stops in the nonwords established two independent interlanguage intelligibility effects. An interlanguage speech intelligibility benefit for listeners (ISIB-L) was manifest by both groups of Mandarin listeners outperforming native English listeners in identification of Mandarin-accented speech. In the benefit for talkers (ISIB-T), only M-BJ listeners were more accurate identifying Mandarin-accented speech than native English speech. Thus, both Mandarin groups demonstrated an ISIB-L while only the M-BJ group overall demonstrated an ISIB-T. The English proficiency of listeners was found to modulate the magnitude of the ISIB-T in both groups. Regression analyses also suggested that the listener groups differ in their use of acoustic information to identify voicing in stop consonants.

Similar response patterns do not imply identical origins: an energetic masking account of nonspeech effects in compensation for coarticulation.

Nonspeech materials are widely used to identify basic mechanisms underlying speech perception. For instance, they have been used to examine the origin of compensation for coarticulation, the observation that listeners' categorization of phonetic segments depends on neighboring segments (Mann, 1980). Specifically, nonspeech precursors matched to critical formant frequencies of speech precursors have been shown to produce similar categorization shifts as speech contexts. This observation has been interpreted to mean that spectrally contrastive frequency relations between neighboring segments underlie the categorization shifts observed after speech, as well as nonspeech precursors (Lotto & Kluender, 1998). From the gestural perspective, however, categorization shifts in speech contexts occur because of listeners' sensitivity to acoustic information for coarticulatory gestural overlap in production; in nonspeech contexts, this occurs because of energetic masking of acoustic information for gestures. In 2 experiments, we distinguish the energetic masking and spectral contrast accounts. In Experiment 1, we investigated the effects of varying precursor tone frequency on speech categorization. Consistent only with the masking account, tonal effects were greater for frequencies close enough to those in the target syllables for masking to occur. In Experiment 2, we filtered the target stimuli to simulate effects of masking and obtained behavioral outcomes that closely resemble those with nonspeech tones. We conclude that masking provides the more plausible account of nonspeech context effects. More generally, we suggest that similar results from the use of speech and nonspeech materials do not automatically imply identical origins and that the use of nonspeech in speech studies entails careful examination of the nature of information in the nonspeech materials.

Articulatory events are imitated under rapid shadowing.

We tested the hypothesis that rapid shadowers imitate the articulatory gestures that structure acoustic speech signals-not just acoustic patterns in the signals themselves-overcoming highly practiced motor routines and phonological conditioning in the process. In a first experiment, acoustic evidence indicated that participants reproduced allophonic differences between American English /l/ types (light and dark) in the absence of the positional variation cues more typically present with lateral allophony. However, imitative effects were small. In a second experiment, varieties of /l/ with exaggerated light/dark differences were presented by ear. Acoustic measures indicated that all participants reproduced differences between /l/ types; larger average imitative effects obtained. Finally, we examined evidence for imitation in articulation. Participants ranged in behavior from one who did not imitate to another who reproduced distinctions among light laterals, dark laterals and /w/, but displayed a slight but inconsistent tendency toward enhancing imitation of lingual gestures through a slight lip protrusion. Overall, results indicated that most rapid shadowers need not substitute familiar allophones as they imitate reorganized gestural constellations even in the absence of explicit instruction to imitate, but that the extent of the imitation is small. Implications for theories of speech perception are discussed.

Embodied, Embedded Language Use.

Language use has a public face that is as important to study as the private faces under intensive psycholinguistic study. In the domain of phonology, public use of speech must meet an interpersonal "parity" constraint if it is to serve to communicate. That is, spoken language forms must reliably be identified by listeners. To that end, language forms are embodied, at the lowest level of description, as phonetic gestures of the vocal tract that lawfully structure informational media such as air and light. Over time, under the parity constraint, sound inventories emerge over communicative exchanges that have the property of sufficient identifiability.Communicative activities involve more than vocal tract actions. Talkers gesture and use facial expressions and eye gaze to communicate. Listeners embody their language understandings, exhibiting dispositions to behave in ways related to language understanding. Moreover, linguistic interchanges are embedded in the larger context of language use. Talkers recruit the environment in their communicative activities, for example, in using deictic points. Moreover, in using language as a "coordination device," interlocutors mutually entrain.

Listeners' perception of compensatory shortening.

English exhibits compensatory shortening, whereby a stressed syllable followed by an unstressed syllable is measured to be shorter than the same stressed syllable alone. This anticipatory shortening is much greater than backward shortening, whereby an unstressed syllable is measured to shorten a following stressed syllable. We speculated that measured shortening reflects not true shortening, but coarticulatory hiding. Hence, we asked whether listeners are sensitive to parts of stressed syllables hidden by following or preceding unstressed syllables. In two experiments (Experiments 1A and 1B), we found the point of subjective equality-that is, the durational difference between a stressed syllable in isolation and one followed by an unstressed syllable-at which listeners cannot tell which is longer. In a third experiment (Experiment 2), we found the point of subjective equality for stressed monosyllables and disyllables with a weak-strong stress pattern. In all of the experiments, the points of subjective equality occurred when stressed syllables in disyllables were measured to be shorter than those in monosyllables, as if the listeners heard the coarticulatory onset or the continuation of a stressed syllable within unstressed syllables.