Near-Perfect Automation: Investigating Performance, Trust, and Visual Attention Allocation.

OBJECTIVE: Assess performance, trust, and visual attention during the monitoring of a near-perfect automated system. BACKGROUND: Research rarely attempts to assess performance, trust, and visual attention in near-perfect automated systems even though they will be relied on in high-stakes environments. METHODS: Seventy-three participants completed a 40-min supervisory control task where they monitored three search feeds. All search feeds were 100% reliable with the exception of two automation failures: one miss and one false alarm. Eye-tracking and subjective trust data were collected. RESULTS: Thirty-four percent of participants correctly identified the automation miss, and 67% correctly identified the automation false alarm. Subjective trust increased when participants did not detect the automation failures and decreased when they did. Participants who detected the false alarm had a more complex scan pattern in the 2 min centered around the automation failure compared with those who did not. Additionally, those who detected the failures had longer dwell times in and transitioned to the center sensor feed significantly more often. CONCLUSION: Not only does this work highlight the limitations of the human when monitoring near-perfect automated systems, it begins to quantify the subjective experience and attentional cost of the human. It further emphasizes the need to (1) reevaluate the role of the operator in future high-stakes environments and (2) understand the human on an individual level and actively design for the given individual when working with near-perfect automated systems. APPLICATION: Multiple operator-level measures should be collected in real-time in order to monitor an operator's state and leverage real-time, individualized assistance.

Limitations of current spatial ability testing for military aviators.

Spatial ability has long been considered an important attribute when identifying military aviators. This paper examines the Direction Orientation Task (DOT), which is currently used by the US military to assess spatial ability in aviation applicants. Several limitations of the test, such as a limited number of trials and the availability of practice trials online, make it subject to potential ceiling effects. The paper presents historical data of all Naval Aviator applicants over a six-year time period and revealed that 22% of applicants answered 90% or more of the questions correctly. Furthermore, test performance has significantly increased in the years since the test was first administered and there is evidence that DOT is no longer contributing incremental validity. A follow-up empirical study looked at DOT performance and strategy in a group of military student aviators and student air traffic controllers. The results of the empirical investigation reveal that the use of an analytic strategy was associated with higher performance on the DOT, whereas the use of a spatial strategy was not associated with performance. Taken together, the improved performance data over time and the data on strategy use suggest the test's ability to measure spatial ability may be diminishing, ultimately reducing its construct and incremental validity. This is problematic and should be addressed, since the DOT is the only measure of spatial ability used by the Navy to assess aviation applicants.

Detecting automation failures in a simulated supervisory control environment.

The goal of this research was to determine how individuals perform and allocate their visual attention when monitoring multiple automated displays that differ in automation reliability. Ninety-six participants completed a simulated supervisory control task where each automated display had a different level of reliability (namely 70%, 85% and 95%). In addition, participants completed a high and low workload condition. The performance data revealed that (1) participants' failed to detect automation misses approximately 2.5 times more than automation false alarms, (2) participants' had worse automation failure detection in the high workload condition and (3) participant automation failure detection remained mostly static across reliability. The eye tracking data revealed that participants spread their attention relatively equally across all three of the automated displays for the duration of the experiment. Together, these data support a system-wide trust approach as the default position of an individual monitoring multiple automated displays. Practitioner Summary: Given the rapid growth of automation throughout the workforce, there is an immediate need to better understand how humans monitor multiple automated displays concurrently. The data in this experiment support a system-wide trust approach as the default position of an individual monitoring multiple automated displays. Abbreviations: DoD: Department of Defense; UA: unmanned aircraft; SCOUT: Supervisory Control Operations User Testbed; UAV: unmanned aerial vehicle; AOI: areas of interest.

An examination of relations between baseline pupil measures and cognitive abilities.

Examining individual differences in pupil size and pupillary dynamics have revealed important insights into the nature of individual differences in cognitive abilities like working memory capacity, long-term memory, attention control, and fluid intelligence. These findings are often tied to the locus coeruleus-norepinephrine (LC-NE) system, as this system has a tight temporal correlation with pupil diameter. Some recent research has demonstrated positive correlations between resting pupil size and cognitive ability, specifically fluid intelligence. The present study attempted to replicate such relations. Across three studies, a large sample of participants (N = 845) completed batteries of cognitive ability measures and measures of resting pupil size and pupillary hippus (fluctuations in pupil diameter). The cognitive measures comprised tasks previously used to measure attention control, visual short-term memory capacity, fluid intelligence, working memory capacity, and visuospatial ability. At the factor level, cognitive ability and pupil size correlated near zero. We did observe some limited evidence for a negative correlation between resting pupillary hippus and cognitive ability. Given the null findings in the present data, we encourage further replication of relations between resting pupil measures and cognitive abilities before making any strong theoretical conclusions about such relations.

Scan-based eye tracking measures are predictive of workload transition performance.

Given there is no unifying theory or design guidance for workload transitions, this work investigated how visual attention allocation patterns could inform both topics, by understanding if scan-based eye tracking metrics could predict workload transition performance trends in a context-relevant domain. The eye movements of sixty Naval flight students were tracked as workload transitioned at a slow, medium, and fast pace in an unmanned aerial vehicle testbed. Four scan-based metrics were significant predictors across the different growth curve models of response time and accuracy. Stationary gaze entropy (a measure of how dispersed visual attention transitions are across tasks) was predictive across all three transition rates. The other three predictive scan-based metrics captured different aspects of visual attention, including its spread, directness, and duration. The findings specify several missing details in both theory and design guidance, which is unprecedented, and serves as a basis of future workload transition research.

Pupil size as a measure of within-task learning.

Pupillometry is commonly used in research to determine how much mental effort an individual is exerting while completing tasks. Traditionally, larger pupils are associated with increased mental effort when completing more difficult tasks. However, little research has investigated how pupils change as individuals learn a new task. In theory, as one repeatedly completes a task, the task demands should reduce, reliance on working memory should decrease, and the task should become more automatic. This should translate to faster completion times and smaller peak pupil dilations. We tested this hypothesis by having participants complete multiple trials of a cognitive task that requires individuals to orient themselves in space relative to a target. We found that trial completion times and maximum pupil size significantly reduced across trials. These data suggest that measuring changes in pupil dilation may help researchers determine whether individuals have shifted from a learned procedure to an automatic processing of information when learning a new task.