Operator selection for human-automation teaming: The role of manual task skill in predicting automation failure intervention.

Humans working in modern work systems are increasingly required to supervise task automation. We examined whether manual aircraft conflict detection skill predicted participants' ability to respond to conflict detection automation failures in simulated air traffic control. In a conflict discrimination task (to assess manual skill), participants determined whether pairs of aircraft were in conflict or not by judging their relative-arrival time at common intersection points. Then in a simulated air traffic control task, participants supervised automation which either partially or fully detected and resolved conflicts on their behalf. Automation supervision required participants to detect when automation may have failed and effectively intervene. When automation failed, participants who had better manual conflict detection skill were faster and more accurate to intervene. However, a substantial proportion of variance in failure intervention was not explained by manual conflict detection skill, potentially reflecting that future research should consider other cognitive skills underlying automation supervision.

How do humans learn about the reliability of automation?

In a range of settings, human operators make decisions with the assistance of automation, the reliability of which can vary depending upon context. Currently, the processes by which humans track the level of reliability of automation are unclear. In the current study, we test cognitive models of learning that could potentially explain how humans track automation reliability. We fitted several alternative cognitive models to a series of participants' judgements of automation reliability observed in a maritime classification task in which participants were provided with automated advice. We examined three experiments including eight between-subjects conditions and 240 participants in total. Our results favoured a two-kernel delta-rule model of learning, which specifies that humans learn by prediction error, and respond according to a learning rate that is sensitive to environmental volatility. However, we found substantial heterogeneity in learning processes across participants. These outcomes speak to the learning processes underlying how humans estimate automation reliability and thus have implications for practice.

Prospective memory decision control: A computational model of context effects on prospective memory.

Prospective memory (PM) tasks require remembering to perform a deferred action and can be associated with predictable contexts. We present a theory and computational model, prospective memory decision control (PMDC), of the cognitive processes by which context supports PM. Under control conditions, participants completed lexical decisions. Under PM conditions, participants had the additional PM task of responding to letter strings containing certain syllables. Stimuli were presented in one of two colors, with color potentially changing after each set of four trials. A pretrial colored fixation was presented before each set. Under control and PM standard conditions, fixation color was meaningless. Under PM context conditions, fixation color indicated whether a PM target could occur within the next set. We replicated prior findings of higher PM accuracy for context compared to standard conditions, and the expected variation in PM costs (slowed lexical decisions) as a function of context relevance. PMDC, which formalizes PM as a process of evidence accumulation among ongoing and PM task responses, accounted for the impact of context on PM costs and accuracy via proactive and reactive cognitive control. Increased ongoing task thresholds and decreased PM thresholds in relevant contexts indicated proactive control. With context provision, PM accumulation rates on PM trials increased, as did inhibition of accumulation to competing responses, indicating reactive control. Although an observed capacity-sharing effect explained some portion of PM costs, we found no evidence that participants redirected more capacity from the ongoing to the PM task when contextually cued to relevant contexts. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Automated decision aids: When are they advisors and when do they take control of human decision making?

We applied a computational model to examine the extent to which participants used an automated decision aid as an advisor, as compared to a more autonomous trigger of responding, at varying levels of decision aid reliability. In an air traffic control conflict detection task, we found higher accuracy when the decision aid was correct, and more errors when the decision aid was incorrect, as compared to a manual condition (no decision aid). Responses that were correct despite incorrect automated advice were slower than matched manual responses. Decision aids set at lower reliability (75%) had smaller effects on choices and response times, and were subjectively trusted less, than decision aids set at higher reliability (95%). We fitted an evidence accumulation model to choices and response times to measure how information processing was affected by decision aid inputs. Participants primarily treated low-reliability decision aids as an advisor rather than directly accumulating evidence based on its advice. Participants directly accumulated evidence based upon the advice of high-reliability decision aids, consistent with granting decision aids more autonomous influence over decisions. Individual differences in the level of direct accumulation correlated with subjective trust, suggesting a cognitive mechanism by which trust impacts human decisions. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

Detecting a Single Automation Failure: The Impact of Expected (But Not Experienced) Automation Reliability.

OBJECTIVE: Examine the impact of expected automation reliability on trust, workload, task disengagement, nonautomated task performance, and the detection of a single automation failure in simulated air traffic control. BACKGROUND: Prior research has focused on the impact of experienced automation reliability. However, many operational settings feature automation that is reliable to the extent that operators will seldom experience automation failures. Despite this, operators must remain aware of when automation is at greater risk of failing. METHOD: Participants performed the task with or without conflict detection/resolution automation. Automation failed to detect/resolve one conflict (i.e., an automation miss). Expected reliability was manipulated via instructions such that the expected level of reliability was (a) constant or variable, and (b) the single automation failure occurred when expected reliability was high or low. RESULTS: Trust in automation increased with time on task prior to the automation failure. Trust was higher when expecting high relative to low reliability. Automation failure detection was improved when the failure occurred under low compared with high expected reliability. Subjective workload decreased with automation, but there was no improvement to nonautomated task performance. Automation increased perceived task disengagement. CONCLUSIONS: Both automation reliability expectations and task experience played a role in determining trust. Automation failure detection was improved when the failure occurred at a time it was expected to be more likely. Participants did not effectively allocate any spared capacity to nonautomated tasks. APPLICATIONS: The outcomes are applicable because operators in field settings likely form contextual expectations regarding the reliability of automation.

The Perception of Automation Reliability and Acceptance of Automated Advice.

OBJECTIVE: Examine (1) the extent to which humans can accurately estimate automation reliability and calibrate to changes in reliability, and how this is impacted by the recent accuracy of automation; and (2) factors that impact the acceptance of automated advice, including true automation reliability, reliability perception, and the difference between an operator's perception of automation reliability and perception of their own reliability. BACKGROUND: Existing evidence suggests humans can adapt to changes in automation reliability but generally underestimate reliability. Cognitive science indicates that humans heavily weight evidence from more recent experiences. METHOD: Participants monitored the behavior of maritime vessels (contacts) in order to classify them, and then received advice from automation regarding classification. Participants were assigned to either an initially high (90%) or low (60%) automation reliability condition. After some time, reliability switched to 75% in both conditions. RESULTS: Participants initially underestimated automation reliability. After the change in true reliability, estimates in both conditions moved towards the common true reliability, but did not reach it. There were recency effects, with lower future reliability estimates immediately following incorrect automation advice. With lower initial reliability, automation acceptance rates tracked true reliability more closely than perceived reliability. A positive difference between participant assessments of the reliability of automation and their own reliability predicted greater automation acceptance. CONCLUSION: Humans underestimate the reliability of automation, and we have demonstrated several critical factors that impact the perception of automation reliability and automation use. APPLICATION: The findings have potential implications for training and adaptive human-automation teaming.

Evidence accumulation modelling in the wild: understanding safety-critical decisions.

Evidence accumulation models (EAMs) are a class of computational cognitive model used to understand the latent cognitive processes that underlie human decisions and response times (RTs). They have seen widespread application in cognitive psychology and neuroscience. However, historically, the application of these models was limited to simple decision tasks. Recently, researchers have applied these models to gain insight into the cognitive processes that underlie observed behaviour in applied domains, such as air-traffic control (ATC), driving, forensic and medical image discrimination, and maritime surveillance. Here, we discuss how this modelling approach helps researchers understand how the cognitive system adapts to task demands and interventions, such as task automation. We also discuss future directions and argue for wider adoption of cognitive modelling in Human Factors research.

Human behavioral response to fluctuating automation reliability.

Human perception of automation reliability and automation acceptance behaviours are key to effective human-automation teaming. This study examined factors that impact perceptions of automation reliability over time and the acceptance of automated advice. Participants completed a maritime vessel classification task in which they classified vessels (contacts) with the assistance of automation. In Experiment 1 automation reliability successively switched from high to low (or vice versa). In Experiment 2 automation reliability decreased by varying magnitudes before returning to high. Participants did not initially calibrate to true reliability and experiencing low automation reliability reduced future reliability estimates when experiencing subsequent high reliability. Automation acceptance was predicted by positive differences between participant perception of automation reliability and confidence in their own manual classification reliability. Experiencing low automation reliability caused perceptions of reliability and automation acceptance rates to diverge. These findings have important implications for training and adaptive human-automation teaming in complex work environments.

Target learning in event-based prospective memory.

Event-based prospective memory (PM) tasks require individuals to remember to perform a previously planned action when they encounter a specific event. Often, the natural environments in which PM tasks occur are embedded are constantly changing, requiring humans to adapt by learning. We examine one such adaptation by integrating PM target learning with the prospective memory decision control (PMDC) cognitive model. We apply this augmented model to an experiment that manipulated exposure to PM targets, comparing a single-target PM condition where the target was well learned from the outset, to a multiple-target PM condition with less initial PM target exposure, allowing us to examine the effect of continued target learning opportunities. Single-target PM accuracy was near ceiling whereas multiple-target PM accuracy was initially poorer but improved throughout the course of the experiment. PM response times were longer for the multiple- compared with single-target PM task but this difference also decreased over time. The model indicated that PM trial evidence accumulation rates, and the inhibition of competing responses, were initially higher for single compared to multiple PM targets, but that this difference decreased over time due to the learning of multiple-targets over the target repetitions. These outcomes provide insight into how the processes underlying event-based PM can dynamically evolve over time, and a modeling framework to further investigate the effect of learning on event-based PM decision processes. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

OSARI, an Open-Source Anticipated Response Inhibition Task.

The stop-signal paradigm has become ubiquitous in investigations of inhibitory control. Tasks inspired by the paradigm, referred to as stop-signal tasks, require participants to make responses on go trials and to inhibit those responses when presented with a stop-signal on stop trials. Currently, the most popular version of the stop-signal task is the 'choice-reaction' variant, where participants make choice responses, but must inhibit those responses when presented with a stop-signal. An alternative to the choice-reaction variant of the stop-signal task is the 'anticipated response inhibition' task. In anticipated response inhibition tasks, participants are required to make a planned response that coincides with a predictably timed event (such as lifting a finger from a computer key to stop a filling bar at a predefined target). Anticipated response inhibition tasks have some advantages over the more traditional choice-reaction stop-signal tasks and are becoming increasingly popular. However, currently, there are no openly available versions of the anticipated response inhibition task, limiting potential uptake. Here, we present an open-source, free, and ready-to-use version of the anticipated response inhibition task, which we refer to as the OSARI (the Open-Source Anticipated Response Inhibition) task.

Integrated responding improves prospective memory accuracy.

Prospective memory (PM) supports the planning and execution of future activities, and is particularly important in applied settings. We investigate a new response method that aims to improve PM accuracy by integrating the responses to an occasional PM task and a routine ongoing lexical-decision task. Instead of the most common three-choice method where the PM response replaces the ongoing response, participants were obligated to make explicit PM (present vs. absent) and ongoing (word vs. non-word) classifications on every trial through a four-choice response. Although replacement and obligatory responses were initially similar in PM accuracy, an advantage emerged with practice for the new obligatory method that was not simply due to slower responding associated with making four versus three choices. The nature of the errors differed between methods, with obligatory responding being characterised by fast PM errors and replacement by slower errors, suggesting avenues for further potential improvements in PM accuracy.

Inhibitory Cognitive Control Allows Automated Advice to Improve Accuracy While Minimizing Misuse.

Humans increasingly use automated decision aids. However, environmental uncertainty means that automated advice can be incorrect, creating the potential for humans to act on incorrect advice or to disregard correct advice. We present a quantitative model of the cognitive process by which humans use automation when deciding whether aircraft would violate requirements for minimum separation. The model closely fitted the performance of 24 participants, who each made 2,400 conflict-detection decisions (conflict vs. nonconflict), either manually (with no assistance) or with the assistance of 90% reliable automation. When the decision aid was correct, conflict-detection accuracy improved, but when the decision aid was incorrect, accuracy and response time were impaired. The model indicated that participants integrated advice into their decision process by inhibiting evidence accumulation toward the task response that was incongruent with that advice, thereby ensuring that decisions could not be made solely on automated advice without first sampling information from the task environment.

Understanding fatigue in a naval submarine: Applying biomathematical models and workload measurement in an intensive longitudinal design.

Fatigue is a critically important aspect of crew endurance in submarine operations, with continuously high fatigue being associated with increased risk of human error and long-term negative health ramifications. Submarines pose several unique challenges to fatigue mitigation, including requirements for continuous manning for long durations, a lack of access to critical environmental zeitgebers (stimuli pertinent to circadian physiology; e.g., natural sunlight), and work, rest and sleep occurring within an encapsulated environment. In this paper, we examine the factors that underlie fatigue in such a context with the aim of evaluating the predictive utility of a biomathematical model (BMM) of fatigue. Three experience sampling studies were conducted with submarine crews using a participant-led measurement protocol that included assessments of subjective sleepiness, workload (NASA-Task Load Index [TLX] and a bespoke underload-overload scale), and sleep. As expected, results indicated that predicting KSS with a BMM approach outperformed more conventional linear modelling approaches (e.g., time-of-day, sleep duration, time awake). Both the homeostatic and circadian components of the BMM were significantly associated with KSS and used as controls in the workload models. We found increased NASA-TLX workload was significantly associated with increased average KSS ratings at the between-person level. However, counter to expectations, the two workload measures were not found to have significant linear or quadratic relationship with fatigue at the within-person level. An important outcome of the research is that applied fatigue researchers should be extremely cautious applying conventional linear predictors when predicting fatigue. Practical implications for the submarine and related extreme work context are discussed. Important avenues for continued research are outlined, including directly estimating BMM parameters.

Investigating the effects of ongoing-task bias on prospective memory.

Event-based prospective memory (PM) refers to the cognitive processes required to perform a planned action upon encountering a future event. Event-based PM studies engage participants in an ongoing task (e.g., lexical decision-making) with an instruction to make an alternative PM response to certain items (e.g., items containing "tor"). The Prospective Memory Decision Control (PMDC) model, which provides a quantitative process account of ongoing-task and PM decisions, proposes that PM and ongoing-task processes compete in a race to threshold. We use PMDC to test whether, as proposed by the Delay Theory of PM costs, PM can be improved by biasing decision-making against a specific ongoing-task choice, so that the PM process is more likely to win the race. We manipulated bias in a lexical decision task with an accompanying PM intention. In one condition, a bias was induced against deciding items were words, and in another, a bias was induced against deciding items were non-words. The bias manipulation had little effect on PM accuracy but did affect the types of ongoing-task responses made on missed PM trials. PMDC fit the observed data well and verified that the bias manipulation had the intended effect on ongoing-task processes. Furthermore, although simulations from PMDC could produce an improvement in PM accuracy due to ongoing-task bias, this required implausible parameter values. These results illustrate the importance of understanding event-based PM in terms of a comprehensive model of the processes that interact to determine all aspects of task performance.

Prospective Memory Performance in Simulated Air Traffic Control : Robust to Interruptions but Impaired by Retention Interval.

OBJECTIVE: To examine the effects of interruptions and retention interval on prospective memory for deferred tasks in simulated air traffic control. BACKGROUND: In many safety-critical environments, operators need to remember to perform a deferred task, which requires prospective memory. Laboratory experiments suggest that extended prospective memory retention intervals, and interruptions in those retention intervals, could impair prospective memory performance. METHOD: Participants managed a simulated air traffic control sector. Participants were sometimes instructed to perform a deferred handoff task, requiring them to deviate from a routine procedure. We manipulated whether an interruption occurred during the prospective memory retention interval or not, the length of the retention interval (37-117 s), and the temporal proximity of the interruption to deferred task encoding and execution. We also measured performance on ongoing tasks. RESULTS: Increasing retention intervals (37-117 s) decreased the probability of remembering to perform the deferred task. Costs to ongoing conflict detection accuracy and routine handoff speed were observed when a prospective memory intention had to be maintained. Interruptions did not affect individuals' speed or accuracy on the deferred task. CONCLUSION: Longer retention intervals increase risk of prospective memory error and of ongoing task performance being impaired by cognitive load; however, prospective memory can be robust to effects of interruptions when the task environment provides cuing and offloading. APPLICATION: To support operators in performing complex and dynamic tasks, prospective memory demands should be reduced, and the retention interval of deferred tasks should be kept as short as possible.

Strategic attention and decision control support prospective memory in a complex dual-task environment.

Human performance in complex multiple-task environments depends critically on the interplay between cognitive control and cognitive capacity. In this paper we propose a tractable computational model of how cognitive control and capacity influence the speed and accuracy of decisions made in the event-based prospective memory (PM) paradigm, and in doing so test a new quantitative formulation that measures two distinct components of cognitive capacity (gain and focus) that apply generally to choices among two or more options. Consistent with prior work, individuals used proactive control (increased ongoing task thresholds under PM load) and reactive control (inhibited ongoing task accumulation rates to PM items) to support PM performance. Individuals used cognitive gain to increase the amount of resources allocated to the ongoing task under time pressure and PM load. However, when demands exceeded the capacity limit, resources were reallocated (shared) between ongoing task and PM processes. Extending previous work, individuals used cognitive focus to control the quality of processing for the ongoing and PM tasks based on the particular demand and payoff structure of the environment (e.g., higher focus for higher priority tasks; lower focus under high time pressure and with PM load). Our model provides the first detailed quantitative understanding of cognitive gain and focus as they apply to evidence accumulation models, which - along with cognitive control mechanisms - support decision-making in complex multiple-task environments.

Prospective memory in the red zone: Cognitive control and capacity sharing in a complex, multi-stimulus task.

Remembering to perform a planned action upon encountering a future event requires event-based Prospective Memory (PM). PM is required in many human factors settings in which operators must process a great deal of complex, uncertain information from an interface. We study event-based PM in such an environment. Our task, which previous research has found is very demanding (Palada, Neal, Tay, & Heathcote, 2018), requires monitoring ships as they cross the ocean on a display. We applied the Prospective Memory Decision Control Model (Strickland, Loft, Remington, & Heathcote, 2018) to understand the cognitive mechanisms that underlie PM performance in such a demanding environment. We found evidence of capacity sharing between monitoring for PM items and performing the ongoing surveillance task, whereas studies of PM in simpler paradigms have not (e.g., Strickland et al., 2018). We also found that participants applied proactive and reactive control (Braver, 2012) to adapt to the demanding task environment. Our findings illustrate the value of human factors simulations to study capacity sharing between competing task processes. They also illustrate the value of cognitive models to illuminate the processes underlying adaptive behavior in complex environments. (PsycINFO Database Record (c) 2019 APA, all rights reserved).

Cognitive control and capacity for prospective memory in complex dynamic environments.

Performing deferred actions in the future relies upon Prospective Memory (PM). Often, PM demands arise in complex dynamic tasks. Not only can PM be challenging in such environments, the processes required for PM may affect the performance of other tasks. To adapt to PM demands in such environments, humans may use a range of strategies, including flexible allocation of cognitive resources and cognitive control mechanisms. We sought to understand such mechanisms by using the Prospective Memory Decision Control (Strickland, Loft, Remington, & Heathcote, 2018) model to provide a comprehensive, quantitative account of dual task performance in a complex dynamic environment (a simulated air traffic control conflict detection task). We found that PM demands encouraged proactive control over ongoing task decisions, but that this control was reduced at high time pressure to facilitate fast responding. We found reactive inhibitory control over ongoing task processes when PM targets were encountered, and that time pressure and PM demand both affect the attentional system, increasing the amount of cognitive resources available. However, as demands exceeded the capacity limit of the cognitive system, resources were reallocated (shared) between the ongoing and PM tasks. As the ongoing task used more resources to compensate for additional time pressure demands, it drained resources that would have otherwise been available for PM task processing. This study provides the first detailed quantitative understanding of how attentional resources and cognitive control mechanisms support PM and ongoing task performance in complex dynamic environments. (PsycINFO Database Record (c) 2019 APA, all rights reserved).

Dynamic models of choice.

Parameter estimation in evidence-accumulation models of choice response times is demanding of both the data and the user. We outline how to fit evidence-accumulation models using the flexible, open-source, R-based Dynamic Models of Choice (DMC) software. DMC provides a hands-on introduction to the Bayesian implementation of two popular evidence-accumulation models: the diffusion decision model (DDM) and the linear ballistic accumulator (LBA). It enables individual and hierarchical estimation, as well as assessment of the quality of a model's parameter estimates and descriptive accuracy. First, we introduce the basic concepts of Bayesian parameter estimation, guiding the reader through a simple DDM analysis. We then illustrate the challenges of fitting evidence-accumulation models using a set of LBA analyses. We emphasize best practices in modeling and discuss the importance of parameter- and model-recovery simulations, exploring the strengths and weaknesses of models in different experimental designs and parameter regions. We also demonstrate how DMC can be used to model complex cognitive processes, using as an example a race model of the stop-signal paradigm, which is used to measure inhibitory ability. We illustrate the flexibility of DMC by extending this model to account for mixtures of cognitive processes resulting from attention failures. We then guide the reader through the practical details of a Bayesian hierarchical analysis, from specifying priors to obtaining posterior distributions that encapsulate what has been learned from the data. Finally, we illustrate how the Bayesian approach leads to a quantitatively cumulative science, showing how to use posterior distributions to specify priors that can be used to inform the analysis of future experiments.

Racing to remember: A theory of decision control in event-based prospective memory.

Event-based prospective memory (PM) requires remembering to perform intended deferred actions when particular stimuli or events are encountered in the future. We propose a detailed process theory within Braver's (2012) proactive and reactive framework of the way control is maintained over the competing demands of prospective memory decisions and decisions associated with ongoing task activities. The theory is instantiated in a quantitative "Prospective Memory Decision Control" (PMDC) architecture, which uses linear ballistic evidence accumulation (Brown & Heathcote, 2008) to model both PM and ongoing decision processes. Prospective control is exerted via decision thresholds, as in Heathcote, Loft, and Remington's (2015) "Delay Theory" of the impact of PM demands on ongoing-task decisions. However, PMDC goes beyond Delay Theory by simultaneously accounting for both PM task decisions and ongoing task decisions. We use Bayesian estimation to apply PMDC to experiments manipulating PM target focality (i.e., the extent to which the ongoing task directs attention to the features of PM targets processed at encoding) and the relative importance of the PM task. As well as confirming Delay Theory's proactive control of ongoing task thresholds, the comprehensive account provided by PMDC allowed us to detect both proactive control of the PM accumulator threshold and reactive control of the relative rates of the PM and ongoing-task evidence accumulation processes. We discuss potential extensions of PMDC to account for other factors that may be prevalent in real-world PM, such as failures of memory retrieval. (PsycINFO Database Record (c) 2018 APA, all rights reserved).