An Eye-Fixation Related Electroencephalography Technique for Predicting Situation Awareness: Implications for Driver State Monitoring Systems.

OBJECTIVE: This study developed a fixation-related electroencephalography band power (FRBP) approach for situation awareness (SA) assessment in automated driving. BACKGROUND: Maintaining good SA in Level 3 automated vehicles is crucial to drivers' takeover performance when the automated system fails. A multimodal fusion approach that enables the analysis of the visual behavioral and cognitive processes of SA can facilitate real-time assessment of SA in future driver state monitoring systems. METHOD: Thirty participants performed three simulated automated driving tasks. After each task, the Situation Awareness Global Assessment Technique (SAGAT) was deployed to capture their SA about key elements that could affect their takeover task performance. Participants eye movements and brain activities were recorded. Data on their brain activity after each eye fixation on the key elements were extracted and labeled according to the correctness of the SAGAT. Mixed-effects models were used to identify brain regions that were indicative of SA, and machine learning models for SA assessment were developed based on the identified brain regions. RESULTS: Participants' alpha and theta oscillation at frontal and temporal areas are indicative of SA. In addition, the FRBP technique can be used to predict drivers' SA with an accuracy of 88% using a neural network model. CONCLUSION: The FRBP technique, which incorporates eye movements and brain activities, can provide more comprehensive evaluation of SA. Findings highlight the potential of utilizing FRBP to monitor drivers' SA in real-time. APPLICATION: The proposed framework can be expanded and applied to driver state monitoring systems to measure human SA in real-world driving.

Physiological Measurements of Situation Awareness: A Systematic Review.

OBJECTIVE: The goal of this systematic literature review is to investigate the relationship between indirect physiological measurements and direct measures of situation awareness (SA). BACKGROUND: Across different environments and tasks, assessments of SA are often performed using techniques designed specifically to directly measure SA, such as SAGAT, SPAM, and/or SART. However, research suggests that indirect physiological sensing methods may also be capable of predicting SA. Currently, it is unclear which particular physiological approaches are sensitive to changes in SA. METHOD: Seven databases were searched using the PRISMA reporting guidelines. Eligibility criteria included human-subject experiments that used at least one direct SA assessment technique, as well as at least one physiological measurement. Information extracted from each article was the physiological metric(s), the direct SA measurement(s), the correlation between these two metrics, and the experimental task(s). All studies underwent a quality assessment. RESULTS: Twenty-five articles were included in this review. Eye tracking techniques were the most commonly used physiological measures, and correlations between conscious aspects of eye movement measures and direct SA scores were observed. Evidence for cardiovascular predictors of SA were mixed. EEG studies were too few to form strong conclusions, but were consistently positive. CONCLUSION: Further investigation is needed to methodically collect more relevant data and comprehensively model the relationships between a wider range of physiological measurements and direct assessments of SA. APPLICATION: This review will guide researchers and practitioners in methods to indirectly assess SA with sensors and highlight opportunities for future research on wearables and SA.

What You Don't Notice Can Harm You: Age-Related Differences in Detecting Concurrent Visual, Auditory, and Tactile Cues.

Objective This research sought to determine whether people can perceive and process three nonredundant (and unrelated) signals in vision, hearing, and touch at the same time and how aging and concurrent task demands affect this ability. Background Multimodal displays have been shown to improve multitasking and attention management; however, their potential limitations are not well understood. The majority of studies on multimodal information presentation have focused on the processing of only two concurrent and, most often, redundant cues by younger participants. Method Two experiments were conducted in which younger and older adults detected and responded to a series of singles, pairs, and triplets of visual, auditory, and tactile cues in the absence (Experiment 1) and presence (Experiment 2) of an ongoing simulated driving task. Detection rates, response times, and driving task performance were measured. Results Compared to younger participants, older adults showed longer response times and higher error rates in response to cues/cue combinations. Older participants often missed the tactile cue when three cues were combined. They sometimes falsely reported the presence of a visual cue when presented with a pair of auditory and tactile signals. Driving performance suffered most in the presence of cue triplets. Conclusion People are more likely to miss information if more than two concurrent nonredundant signals are presented to different sensory channels. Application The findings from this work help inform the design of multimodal displays and ensure their usefulness across different age groups and in various application domains.

Takeover requests for automated driving: The effects of signal direction, lead time, and modality on takeover performance.

Vehicle-to-driver takeover will still be needed in semi-autonomous vehicles. Due to the complexity of the takeover process, it is important to develop interfaces to support good takeover performance. Multimodal displays have been proposed as a candidate for the design of takeover requests (TORs), but many questions remain unanswered regarding the effectiveness of this approach. This study investigated the effects of takeover signal direction (ipsilateral vs. contralateral), lead time (4 vs. 7 s), and modality (uni-, bi-, and trimodal combinations of visual, auditory, and tactile signals) on automated vehicle takeover performance. Twenty-four participants rode in a simulated SAE Level 3 vehicle and performed a series of takeover tasks when presented with a TOR. Overall, single and multimodal signals with a tactile component were correlated with the faster takeover and information processing times, and were perceived as most useful. Ipsilateral signals showed a marginally significant benefit to takeover times compared to contralateral signals. Finally, a shorter lead time was associated with faster takeover times, but also poorer takeover quality. Findings from this study can inform the design of in-vehicle information and warning systems for next-generation transportation.

The effects of age and physical exercise on multimodal signal responses: Implications for semi-autonomous vehicle takeover requests.

The present study examined whether the non-chronological age factor, engagement in physical exercise, affected responses to multimodal (combinations of visual, auditory, and/or tactile) signals differently between younger and older adults in complex environments. Forty-eight younger and older adults were divided into exercise and non-exercise groups, and rode in a simulated Level 3 autonomous vehicle under four different task conditions (baseline, video watching, headway estimation, and video-headway combination), while being asked to respond to various multimodal warning signals. Overall, bi- and trimodal warnings had faster response times for both age groups across driving conditions, but was more pronounced for older adults. Engagement in physical exercise was associated with smaller maximum braking force for younger participants only, and also corresponded to longer average fixation durations, compared to the non-exercise group. Findings from this research can help to guide decisions about the design of warning and information systems for semi-autonomous vehicles.

Age is more than just a number: The relationship among age, non-chronological age factors, self-perceived driving abilities, and autonomous vehicle acceptance.

Globally, adults aged 65 and older are a rapidly-growing population. Aging is associated with declines in perceptual, cognitive, and physical abilities, which often creates challenges in completing daily activities, such as driving. Autonomous vehicles (AVs) promise to provide older adults one way to maintain their mobility and independence. However, recent surveys of AV acceptance suggest that older adults have a lower AV acceptance compared to younger generations. One challenge is that most of these assessments have not accounted for the various non-chronological age factors that contribute to how older adults perceive their own driving skills and the utility of AVs. To fill this research gap, this study investigated the effects of non-chronological age factors and rated self-perceived driving abilities on AV acceptance across three age groups. An online survey was conducted using Amazon Mechanical Turk (MTurk), for which 438 valid responses were received. Respondents were categorized into a younger (18-40 years), middle-aged (41-64 years), and older (65-79 years) adult age group. Results showed that drivers of a younger age, with higher educational attainment, who rated themselves to have higher social support, and who have lower rated self-perceived driving abilities, report being more willing to accept AVs. Findings from this work can help to inform models of AV technology acceptance and guide in the development of marketing strategies to promote knowledge of AVs.

Using eye-tracking to investigate the effects of pre-takeover visual engagement on situation awareness during automated driving.

Automated driving systems are becoming increasingly prevalent throughout society. In conditionally automated vehicles, drivers may engage in non-driving-related tasks (NDRTs), which can negatively affect their situation awareness (SA) and preparedness to resume control of the vehicle, when necessary. Previous work has investigated engagement in NDRTs, but questions remain unanswered regarding its effect on drivers' SA during a takeover event. The objective of the current study is to use eye-tracking to aid in understanding how visual engagement in NDRTs affects changes in SA of the driving environment after a takeover request (TOR) has been issued. Thirty participants rode in a simulated SAE Level 3 automated driving environment and engaged in three separate pre-TOR tasks (Surrogate Reference Task, Monitoring Task, and Peripheral Detection Task) until presented with a TOR. Situation Awareness Global Assessment Technique (SAGAT) scores and gaze behavior were recorded during the post-TOR segment. Overall, longer times spent viewing the driving scene, and more dispersed visual attention allocation, were observed to be associated with better overall SA. Also, location-based eye tracking metrics show most promise in differentiating between task conditions with significantly different SAGAT scores. Findings from this work can inform the development of real-time SA assessment techniques using eye movements and ultimately contribute to improved operator roadway awareness for next-generation automated transportation.