MagTrack: A Wearable Tongue Motion Tracking System for Silent Speech Interfaces.

PURPOSE: Current electromagnetic tongue tracking devices are not amenable for daily use and thus not suitable for silent speech interface and other applications. We have recently developed MagTrack, a novel wearable electromagnetic articulograph tongue tracking device. This study aimed to validate MagTrack for potential silent speech interface applications. METHOD: We conducted two experiments: (a) classification of eight isolated vowels in consonant-vowel-consonant form and (b) continuous silent speech recognition. In these experiments, we used data from healthy adult speakers collected with MagTrack. The performance of vowel classification was measured by accuracies. The continuous silent speech recognition was measured by phoneme error rates. The performance was then compared with results using data collected with commercial electromagnetic articulograph in a prior study. RESULTS: The isolated vowel classification using MagTrack achieved an average accuracy of 89.74% when leveraging all MagTrack signals (x, y, z coordinates; orientation; and magnetic signals), which outperformed the accuracy using commercial electromagnetic articulograph data (only y, z coordinates) in our previous study. The continuous speech recognition from two subjects using MagTrack achieved phoneme error rates of 73.92% and 66.73%, respectively. The commercial electromagnetic articulograph achieved 64.53% from the same subject (66.73% using MagTrack data). CONCLUSIONS: MagTrack showed comparable results with the commercial electromagnetic articulograph when using the same localized information. Adding raw magnetic signals would improve the performance of MagTrack. Our preliminary testing demonstrated the potential for silent speech interface as a lightweight wearable device. This work also lays the foundation to support MagTrack's potential for other applications including visual feedback-based speech therapy and second language learning.

Evaluation of a Head-Tongue Controller for Power Wheelchair Driving by People With Quadriplegia.

The head-tongue controller (HTC) is a multimodal alternative controller designed for people with quadriplegia to access complex control capabilities by combining tongue and head tracking to offer both discrete and proportional controls in a single controller. In this human study, 17 patients with quadriplegia and current users of alternative controllers were asked to perform four trials of either simple driving tasks or advanced maneuvers in a custom-designed course. Completion time and accuracy were compared between their personal alternative controller (PAC) and various combinations of driving modalities with the HTC. Out of 8 subjects assigned to simple driving, the best HTC trial of 3 subjects was completed faster than their PAC for the tasks of rolling forward and turning around cones, and 5 subjects in rolling backward. Across all these subjects, the average completion time of their best HTC modality is 23 s for rolling forward, 15 s for rolling backward, and 70 s for turning around cones as compared to 19 s, 17 s, and 45 s with their PAC. For advanced driving, the course was completed faster with the HTC by 1 out of 9 subjects, while the best HTC trials of all subjects are less than 1.3 times of their best PAC completion time with an average of 170 s for the HTC and 140 s for their PAC. The qualitative feedback provided by all subjects to a post-study questionnaire scored to an average of 7.5 out of 10 which shows their interests in the HTC and acknowledgement of its usefulness for this population.

Inertial Measurements for Tongue Motion Tracking Based on Magnetic Localization with Orientation Compensation.

Permanent magnet localization (PML) is designed for applications requiring non-line-of-sight motion tracking with millimetric accuracy. Current PML-based tongue tracking is not only impractical for daily use due to many sensors being placed around the mouth, but also requires a large training set of tracer motion. Our method was designed to overcome these shortcomings by generating a local magnetic field and removing the need for the localization to be trained with tracer rotations. An inertial measurement unit (IMU) is used as a tracer that moves in a local magnetic field generated by a magnet strip. The magnetic strength can be optimized to enable the strip to be placed further away from the tracer, thus hidden from view. The tracer is small (6×6×0.8 mm3) to reduce hindrance to natural tongue movements, and the strip is designed to be worn as a neckband. The IMU's magnetometer measures the local magnetic field which is compensated for the tracer's orientation by using the IMU's accelerometer and gyroscope. The orientation-compensated magnetic measurements are then fed into a localization algorithm that estimates the tracer's 3D position. The objective of this study is to evaluate the tracking accuracy of our method. In a 8×8×5 cm3 volume, positional errors of 1.6 mm (median) and 2.4 mm (third quartile, Q3) were achieved on a tracer being rotated ±50° along both pitch and roll. These results indicate this technology is promising for tongue tracking applications.

Evaluation of a Wireless Tongue Tracking System on the Identification of Phoneme Landmarks.

OBJECTIVE: Evaluate the accuracy of a tongue tracking system based on the localization of a permanent magnet to generate a baseline of phoneme landmarks. The positional variability of the landmarks provides an indirect measure of the tracking errors to estimate the position of a small tracer attached on the tongue. The creation of a subject-independent (universal) baseline was also attempted for the first time. METHOD: 2,500 tongue trajectories were collected from 10 subjects tasked to utter 10 repetitions of 25 phonemes. A landmark was identified from each tongue trajectory, and tracking errors were calculated by comparing the distance of each repetition landmark to a final landmark set as their mean position. RESULTS: In the subject-dependent baseline, the tracking errors were found to be generally consistent across all phonemes, and subjects, with less than 25% of the errors reported to be greater than 5.8 mm (median: 3.9 mm). However, the inter-subject variability showed that current limitations of our system resulted in appreciable errors (median: 55 mm, Q3: 65 mm). CONCLUSION: The tracking errors reported in the subject-dependent case demonstrated the potential of our system to generate a baseline of phoneme landmarks. We have identified areas of improvement that will reduce the gap between the subject-dependent, and universal baseline, while lowering tracking errors to be comparable to the gold standard. SIGNIFICANCE: Creating a baseline of phoneme landmarks can help people affected by speech sound disorders to improve their intelligibility using visual feedback that guides their tongue placement to the proper position.