Refocusing on Relevance: Personalization in NLG.

Many NLG tasks such as summarization, dialogue response, or open domain question answering focus primarily on a source text in order to generate a target response. This standard approach falls short, however, when a user's intent or context of work is not easily recoverable based solely on that source text-a scenario that we argue is more of the rule than the exception. In this work, we argue that NLG systems in general should place a much higher level of emphasis on making use of additional context, and suggest that relevance (as used in Information Retrieval) be thought of as a crucial tool for designing user-oriented text-generating tasks. We further discuss possible harms and hazards around such personalization, and argue that value-sensitive design represents a crucial path forward through these challenges.

SSVEP BCI and Eye Tracking Use by Individuals With Late-Stage ALS and Visual Impairments.

Access to communication is critical for individuals with late-stage amyotrophic lateral sclerosis (ALS) and minimal volitional movement, but they sometimes present with concomitant visual or ocular motility impairments that affect their performance with eye tracking or visual brain-computer interface (BCI) systems. In this study, we explored the use of modified eye tracking and steady state visual evoked potential (SSVEP) BCI, in combination with the Shuffle Speller typing interface, for this population. Two participants with late-stage ALS, visual impairments, and minimal volitional movement completed a single-case experimental research design comparing copy-spelling performance with three different typing systems: (1) commercially available eye tracking communication software, (2) Shuffle Speller with modified eye tracking, and (3) Shuffle Speller with SSVEP BCI. Participant 1 was unable to type any correct characters with the commercial system, but achieved accuracies of up to 50% with Shuffle Speller eye tracking and 89% with Shuffle Speller BCI. Participant 2 also had higher maximum accuracies with Shuffle Speller, typing with up to 63% accuracy with eye tracking and 100% accuracy with BCI. However, participants' typing accuracy for both Shuffle Speller conditions was highly variable, particularly in the BCI condition. Both the Shuffle Speller interface and SSVEP BCI input show promise for improving typing performance for people with late-stage ALS. Further development of innovative BCI systems for this population is needed.

Are Some Words Worth More than Others?

Current evaluation metrics for language modeling and generation rely heavily on the accuracy of predicted (or generated) words as compared to a reference ground truth. While important, token-level accuracy only captures one aspect of a language model's behavior, and ignores linguistic properties of words that may allow some mis-predicted tokens to be useful in practice. Furthermore, statistics directly tied to prediction accuracy (including perplexity) may be confounded by the Zipfian nature of written language, as the majority of the prediction attempts will occur with frequently-occurring types. A model's performance may vary greatly between high- and low-frequency words, which in practice could lead to failure modes such as repetitive and dull generated text being produced by a downstream consumer of a language model. To address this, we propose two new intrinsic evaluation measures within the framework of a simple word prediction task that are designed to give a more holistic picture of a language model's performance. We evaluate several commonly-used large English language models using our proposed metrics, and demonstrate that our approach reveals functional differences in performance between the models that are obscured by more traditional metrics.

Automatic Analysis of Pronunciations for Children with Speech Sound Disorders.

Computer-Assisted Pronunciation Training (CAPT) systems aim to help a child learn the correct pronunciations of words. However, while there are many online commercial CAPT apps, there is no consensus among Speech Language Therapists (SLPs) or non-professionals about which CAPT systems, if any, work well. The prevailing assumption is that practicing with such programs is less reliable and thus does not provide the feedback necessary to allow children to improve their performance. The most common method for assessing pronunciation performance is the Goodness of Pronunciation (GOP) technique. Our paper proposes two new GOP techniques. We have found that pronunciation models that use explicit knowledge about error pronunciation patterns can lead to more accurate classification whether a phoneme was correctly pronounced or not. We evaluate the proposed pronunciation assessment methods against a baseline state of the art GOP approach, and show that the proposed techniques lead to classification performance that is more similar to that of a human expert.

Compositional Language Modeling for Icon-Based Augmentative and Alternative Communication.

Icon-based communication systems are widely used in the field of Augmentative and Alternative Communication. Typically, icon-based systems have lagged behind word- and character-based systems in terms of predictive typing functionality, due to the challenges inherent to training icon-based language models. We propose a method for synthesizing training data for use in icon-based language models, and explore two different modeling strategies.

A Multi-Context Character Prediction Model for a Brain-Computer Interface.

Brain-computer interfaces and other augmentative and alternative communication devices introduce language-modeing challenges distinct from other character-entry methods. In particular, the acquired signal of the EEG (electroencephalogram) signal is noisier, which, in turn, makes the user intent harder to decipher. In order to adapt to this condition, we propose to maintain ambiguous history for every time step, and to employ, apart from the character language model, word information to produce a more robust prediction system. We present preliminary results that compare this proposed Online-Context Language Model (OCLM) to current algorithms that are used in this type of setting. Evaluations on both perplexity and predictive accuracy demonstrate promising results when dealing with ambiguous histories in order to provide to the front end a distribution of the next character the user might type.

Effects of simulated visual acuity and ocular motility impairments on SSVEP brain-computer interface performance: An experiment with Shuffle Speller.

Individuals with severe speech and physical impairments may have concomitant visual acuity impairments (VAI) or ocular motility impairments (OMI) impacting visual BCI use. We report on the use of the Shuffle Speller typing interface for an SSVEP BCI copy-spelling task under three conditions: simulated VAI, simulated OMI, and unimpaired vision. To mitigate the effect of visual impairments, we introduce a method that adaptively selects a user-specific trial length to maximize expected information transfer rate (ITR); expected ITR is shown to closely approximate the rate of correct letter selections. All participants could type under the unimpaired and simulated VAI conditions, with no significant differences in typing accuracy or speed. Most participants (31 of 37) could not type under the simulated OMI condition; some achieved high accuracy but with slower typing speeds. Reported workload and discomfort were low, and satisfaction high, under the unimpaired and simulated VAI conditions. Implications and future directions to examine effect of visual impairment on BCI use is discussed.

Pronunciation analysis for children with speech sound disorders.

Phonological disorders affect 10% of preschool and school-age children, adversely affecting their communication, academic performance, and interaction level. Effective pronunciation training requires prolonged supervised practice and interaction. Unfortunately, many children do not have access or only limited access to a speech-language pathologist. Computer-assisted pronunciation training has the potential for being a highly effective teaching aid; however, to-date such systems remain incapable of identifying pronunciation errors with sufficient accuracy. In this paper, we propose to improve accuracy by (1) learning acoustic models from a large children's speech database, (2) using an explicit model of typical pronunciation errors of children in the target age range, and (3) explicit modeling of the acoustics of distorted phonemes.