Augmenting Quality Assurance Measures in Treatment Review with Machine Learning in Radiation Oncology.

Purpose: Pretreatment quality assurance (QA) of treatment plans often requires a high cognitive workload and considerable time expenditure. This study explores the use of machine learning to classify pretreatment chart check QA for a given radiation plan as difficult or less difficult, thereby alerting the physicists to increase scrutiny on difficult plans. Methods and Materials: Pretreatment QA data were collected for 973 cases between July 2018 and October 2020. The outcome variable, a degree of difficulty, was collected as a subjective rating by physicists who performed the pretreatment chart checks. Potential features were identified based on clinical relevance, contribution to plan complexity, and QA metrics. Five machine learning models were developed: support vector machine, random forest classifier, adaboost classifier, decision tree classifier, and neural network. These were incorporated into a voting classifier, where at least 2 algorithms needed to predict a case as difficult for it to be classified as such. Sensitivity analyses were conducted to evaluate feature importance. Results: The voting classifier achieved an overall accuracy of 77.4% on the test set, with 76.5% accuracy on difficult cases and 78.4% accuracy on less difficult cases. Sensitivity analysis showed features associated with plan complexity (number of fractions, dose per monitor unit, number of planning structures, and number of image sets) and clinical relevance (patient age) were sensitive across at least 3 algorithms. Conclusions: This approach can be used to equitably allocate plans to physicists rather than randomly allocate them, potentially improving pretreatment chart check effectiveness by reducing errors propagating downstream.

Multi-Institutional Stereotactic Body Radiation Therapy Incident Learning: Evaluation of Safety Barriers Using a Human Factors Analysis and Classification System.

OBJECTIVES: Stereotactic body radiation therapy (SBRT) can improve therapeutic ratios and patient convenience, but delivering higher doses per fraction increases the potential for patient harm. Incident learning systems (ILSs) are being increasingly adopted in radiation oncology to analyze reported events. This study used an ILS coupled with a Human Factor Analysis and Classification System (HFACS) and barriers management to investigate the origin and detection of SBRT events and to elucidate how safeguards can fail allowing errors to propagate through the treatment process. METHODS: Reported SBRT events were reviewed using an in-house ILS at 4 institutions over 2014-2019. Each institution used a customized care path describing their SBRT processes, including designated safeguards to prevent error propagation. Incidents were assigned a severity score based on the American Association of Physicists in Medicine Task Group Report 275. An HFACS system analyzed failing safeguards. RESULTS: One hundred sixty events were analyzed with 106 near misses (66.2%) and 54 incidents (33.8%). Fifty incidents were designated as low severity, with 4 considered medium severity. Incidents most often originated in the treatment planning stage (38.1%) and were caught during the pretreatment review and verification stage (37.5%) and treatment delivery stage (31.2%). An HFACS revealed that safeguard failures were attributed to human error (95.2%), routine violation (4.2%), and exceptional violation (0.5%) and driven by personnel factors 32.1% of the time, and operator condition also 32.1% of the time. CONCLUSIONS: Improving communication and documentation, reducing time pressures, distractions, and high workload should guide proposed improvements to safeguards in radiation oncology.

The Effect of a System-Level Tiered Huddle System on Reporting Patient Safety Events: An Interrupted Time Series Analysis.

BACKGROUND: The objective of this research was to evaluate the effect of implementing a system-level tiered huddle system (THS) on the reporting of patient safety events into the official event reporting system. METHODS: A quasi-experimental study using interrupted time series was conducted to assess the impact and changes to trends in the reporting of patient safety events pre- (February-July 2020; six months) and post- (September 2020-February 2021; six months) THS implementation within one health care system (238 clinics and 4 hospitals). The severity of harm was analyzed in July 2021 using a modified Agency for Healthcare Research and Quality (AHRQ) harm score classification. The primary outcome measure was the number of patient safety events reported per month. Secondary outcomes included the number of patient safety events reported per month by each AHRQ harm score classification. RESULTS: The system-level THS implementation led to a significant and immediate increase in the total number of patient safety events reported per month (777.73, 95% confidence interval [CI] 310.78-1,244.68, p = 0.004). Similar significant increases were seen for reported numbers of unsafe conditions, near misses, no-harm events that reached patients, and temporary harm (p < 0.05 for each). Reporting of events with permanent harm and deaths also increased but was not statistically significant, likely due to the small number of reported events involving actual harm. CONCLUSION: These findings suggest that system-level THS implementation may increase reporting of patient safety events in the official event reporting system.

A Novel Theory-Based Virtual Reality Training to Improve Patient Safety Culture in the Department of Surgery of a Large Academic Medical Center: Protocol for a Mixed Methods Study.

BACKGROUND: Preventable surgical errors of varying degrees of physical, emotional, and financial harm account for a significant number of adverse events. These errors are frequently tied to systemic problems within a health care system, including the absence of necessary policies/procedures, obstructive cultural hierarchy, and communication breakdown between staff. We developed an innovative, theory-based virtual reality (VR) training to promote understanding and sensemaking toward the holistic view of the culture of patient safety and high reliability. OBJECTIVE: We aim to assess the effect of VR training on health care workers' (HCWs') understanding of contributing factors to patient safety events, sensemaking of patient safety culture, and high reliability organization principles in the laboratory environment. Further, we aim to assess the effect of VR training on patient safety culture, TeamSTEPPS behavior scores, and reporting of patient safety events in the surgery department of an academic medical center in the clinical environment. METHODS: This mixed methods study uses a pre-VR versus post-VR training study design involving attending faculty, residents, nurses, technicians of the department of surgery, and frontline HCWs in the operation rooms at an academic medical center. HCWs' understanding of contributing factors to patient safety events will be assessed using a scale based on the Human Factors Analysis and Classification System. We will use the data frame theory framework, supported by a semistructured interview guide to capture the sensemaking process of patient safety culture and principles of high reliability organizations. Changes in the culture of patient safety will be quantified using the Agency for Healthcare Research and Quality surveys on patient safety culture. TeamSTEPPS behavior scores based on observation will be measured using the Teamwork Evaluation of Non-Technical Skills tool. Patient safety events reported in the voluntary institutional reporting system will be compared before the training versus those after the training. We will compare the Agency for Healthcare Research and Quality patient safety culture scores and patient safety events reporting before the training versus those after the training by using descriptive statistics and a within-subject 2-tailed, 2-sample t test with the significance level set at .05. RESULTS: Ethics approval was obtained in May 2021 from the institutional review board of the University of North Carolina at Chapel Hill (22-1150). The enrollment of participants for this study will start in fall 2022 and is expected to be completed by early spring 2023. The data analysis is expected to be completed by July 2023. CONCLUSIONS: Our findings will help assess the effectiveness of VR training in improving HCWs' understanding of contributing factors of patient safety events, sensemaking of patient safety culture, and principles and behaviors of high reliability organizations. These findings will contribute to developing VR training to improve patient safety culture in other specialties.

Assessment of Radiation Therapy Technologists' Workload and Situation Awareness: Monitoring 2 Versus 3 Collocated Display Monitors.

PURPOSE: This study aimed to assess the effect of monitoring 2 versus 3 collocated displays on radiation therapist technologists' (RTTs) workload (WL) and situation awareness (SA) during routine treatment delivery tasks. METHODS AND MATERIALS: Seven RTTs completed 4 simulated treatment delivery scenarios (2 scenarios per experimental condition; 2 vs 3 collocated displays) in a within-subject experiment. WL was subjectively measured using the National Aeronautics and Space Administration (NASA) Task Load Index, and objectively measured using eye activity measures. SA was subjectively measured using the SA rating technique, and objectively measured using the SA global assessment technique. Two-tailed paired t tests were conducted to test for differences in means when parametric assumptions were satisfied, otherwise Wilcoxon signed-rank tests were conducted. A .05 level of significance was applied to all statistical tests. RESULTS: No statistically and clinically significant differences were observed between monitoring 2 versus 3 monitors on eye tracking measures (blink rate: 9.4 [4.8] vs 9.6 [4.0]; task evoked pupillary response: 0.16 [0.14] vs 0.21 [0.15]; NASA Task Load Index: 34.7 [19.8] vs 35.3 [20.4]; SA rating technique: 19.3 [6.2] vs 19.5 [7.0]; and SA global assessment technique scores: 100 [0] vs 100 [0]). CONCLUSIONS: Our preliminary findings suggest that monitoring 3 collocated displays by 1 RTT does not impact WL and SA compared with monitoring 2 collocated displays. Only 2 of many possible configurations were investigated. If institutions removed the 3rd display based on the results of this study, there could be unforeseen error(s) if that display helped in situations not assessed in this study.

Effect of Simulation-Based Training and Neurofeedback Interventions on Radiation Technologists' Workload, Situation Awareness, and Performance.

PURPOSE: Our purpose was to assess the effect of a combined intervention - simulation-based training supported by neurofeedback sessions - on radiation technologists' (RTs') workload, situation awareness, and performance during routine quality assurance and treatment delivery tasks. METHODS AND MATERIALS: As part of a prospective institutional review board approved study, 32 RTs previously randomized to receive versus not receive simulation-based training focused on patient safety were again randomized to receive versus not receive a 3-week neurofeedback intervention (8 sessions of alpha-theta protocol) focused on stress reduction as well as conscious precision, strong focus, and ability to solve arising problems. Perceived workload was quantified using the NASA Task Load Index. Situation awareness was quantified using the situation awareness rating technique. Performance score was calculated using procedural compliance with time-out components and error detection. RESULTS: RTs randomized to simulation-based training followed by neurofeedback sessions demonstrated no significant changes in perceived workload or situation awareness scores, but did have better performance compared with other study groups (P < .01). CONCLUSIONS: This finding is encouraging and provides basis for using neurofeedback as means to possibly augment performance improvements gained during simulation-based training.

Pilot Study Using Neurofeedback as a Tool to Reduce Surgical Resident Burnout.

BACKGROUND: Burnout is prevalent among surgical residents. Neurofeedback is a technique to train the brain in self-regulation skills. We aimed to assess the impact of neurofeedback on the cognitive workload and personal growth areas of surgery residents with burnout and depression. STUDY DESIGN: Fifteen surgical residents with burnout (Maslach Burnout Inventory [MBI] score > 27) and depression (Patient Health Questionnaire-9 Depression Screen [PHQ-9] score >10), from 1 academic institution, were enrolled and participated in this institutional review board-approved prospective study. Ten residents with more severe burnout and depression scores were assigned to receive 8 weeks of neurofeedback treatments, and 5 others with less severe symptoms were treated as controls. Each participant's cognitive workload (or mental effort) was assessed initially, and again after treatment via electroencephalogram (EEG) while the subjects performed n-back working memory tasks. Analysis of variance (ANOVA) tested for significance between the degree of change in the treatment and control groups. Each subject was also asked to rate changes in growth areas, such as sleep and stress. RESULTS: Both groups showed high cognitive workload in the pre-assessment. After the neurofeedback intervention, the treatment group showed a significant (p < 0.01) improvement in cognitive workload via EEG during the working memory task. These differences were not noted in the control group. There was significant correlation between time (NFB sessions) and average improvement in all growth areas (r = 0.98) CONCLUSIONS: Residents demonstrated high levels of burnout, correlating with EEG patterns indicative of post-traumatic stress disorder. There was a notable change in cognitive workload after the neurofeedback treatment, suggesting a return to a more efficient neural network.

Impact of Workspace Design on Radiation Therapist Technicians' Physical Stressors, Mental Workload, Situation Awareness, and Performance.

PURPOSE: Our purpose was to assess the effect of workspace configuration on radiation therapists' (RTs) physical stressors, mental workload (WL), situational awareness (SA), and performance during routine treatment delivery tasks in a simulated environment. METHODS AND MATERIALS: Fourteen RTs were randomized to 2 workspace configurations while performing 4 simulated scenarios: current (not ergonomically optimized; n = 7) and enhanced (ergonomically optimized, n = 7). Physical stressors were objectively assessed using a rapid upper limb assessment tool. Mental WL was measured at the end of each simulated scenario subjectively using the NASA Task-Load Index and objectively throughout the scenario using eye-tracking metrics (pupil diameter and blink rate). SA was measured at the end of each simulated scenario subjectively using the situation awareness and review technique. Performance was measured objectively via assessment of time-out compliance, error detection, and procedural compliance. Analysis of variance was used to test the effect of workspace configuration on physical stressors, mental WL, SA, and performance. RESULTS: The enhanced configuration significantly reduced physical stressors (rapid upper limb assessment; P < .01) and resulted in a higher rate of time-out compliance (P = .01) compared with current workspace configuration. No significant effect on other metrics was measured. CONCLUSIONS: Our results suggest that an ergonomically designed workspace may minimize physical stressors and improve the performance of RTs.

Improving Cognitive Workload in Radiation Therapists: A Pilot EEG Neurofeedback Study.

Radiation therapy therapists (RTTs) face challenging daily tasks that leave them prone to high attrition and burnout and subsequent deficits in performance. Here, we employed an accelerated alpha-theta neurofeedback (NF) protocol that is implementable in a busy medical workplace to test if 12 RTTs could learn the protocol and exhibit behavior and brain performance-related benefits. Following the 3-week protocol, participants showed a decrease in subjective cognitive workload and a decrease in response time during a performance task, as well as a decrease in desynchrony of the alpha electroencephalogram (EEG) band. Additionally, novel microstate analysis for neurofeedback showed a significant decrease in global field power (GFP) following neurofeedback. These results suggest that the RTTs successfully learned the protocol and improved in perceived cognitive workload following 3 weeks of neurofeedback. In sum, this study presents promising behavioral improvements as well as brain performance-related evidence of neurophysiological changes following neurofeedback, supporting the feasibility of implementing neurofeedback in a busy workplace and encouraging the further study of neurofeedback as a tool to mitigate burnout.

Impact of Simulation-Based Training on Radiation Therapists' Workload, Situation Awareness, and Performance.

PURPOSE: This study aimed to assess the impact of simulation-based training intervention on radiation therapy therapist (RTT) mental workload, situation awareness, and performance during routine quality assurance (QA) and treatment delivery tasks. METHODS AND MATERIALS: As part of a prospective institutional review board-approved study, 32 RTTs completed routine QA and treatment delivery tasks on clinical scenarios in a simulation laboratory. Participants, randomized to receive (n = 16) versus not receive (n = 16) simulation-based training had pre- and postintervention assessments of mental workload, situation awareness, and performance. We used linear regression models to compare the postassessment scores between the study groups while controlling for baseline scores. Mental workload was quantified subjectively using the NASA Task Load Index. Situation awareness was quantified subjectively using the situation awareness rating technique and objectively using the situation awareness global assessment technique. Performance was quantified based on procedural compliance (adherence to preset/standard QA timeout tasks) and error detection (detection and correction of embedded treatment planning errors). RESULTS: Simulation-based training intervention was associated with significant improvements in overall performance (P < .01), but had no significant impact on mental workload or subjective/objective quantifications of situation awareness. CONCLUSIONS: Simulation-based training might be an effective tool to improve RTT performance of QA-related tasks.

Incorporating Human Factors Analysis and Classification System (HFACS) Into Analysis of Reported Near Misses and Incidents in Radiation Oncology.

PURPOSE: Human factors analysis and classification system (HFACS) is a framework for investigation into causation of human errors. We herein assess whether radiation oncology professionals, with brief training, can conduct HFACS on reported near misses or safety incidents (NMSIs) in a reliable (eg, with a high level of agreement) and practical (eg, timely and with user satisfaction) manner. METHODS AND MATERIALS: We adapted a classical HFACS framework by selecting and modifying main headings, subheadings, and nano-codes that were most likely to apply to radiation oncology settings. The final modified HFACS included 3 main headings, 8 subheadings, and 20 nano-codes. The modified HFACS was first tested in a simulated trial on 8 NMSI and was analyzed by 5 to 10 radiation oncology professionals, with 2 endpoints: (1) agreement among participants at the main-heading, subheading, and nano-code level, and (2) time to complete the analysis. We then performed a prospective trial integrating this approach into a weekly NMSI review meeting, with 10 NMSIs analyzed by 8 to 13 radiation oncology professionals with the same endpoints, while also collecting survey data on participants' satisfaction. RESULTS: In the simulated trial, agreement among participants was 85% on the main headings, 73% on the subheadings, and 70% on the nano-codes. Participants needed, on average, 16.4 minutes (standard deviation, 5.7 minutes) to complete an analysis. In the prospective trial, agreement between participants was 81% on the main headings, 75% on the subheadings, and 74% on the nano-codes. Participants needed, on average, 8.3 minutes (standard deviation, 4.7 minutes) to complete an analysis. The average satisfaction with the proposed HFACS approach was 3.9 (standard deviation 1.0) on a scale from 1 to 5. CONCLUSIONS: This study demonstrates that, after relatively brief training, radiation oncology professionals were able to perform HFACS analysis in a reliable and timely manner and with a relatively high level of satisfaction.

Predicting Radiation Therapy Process Reliability Using Voluntary Incident Learning System Data.

PURPOSE: This study aimed to present an innovative approach to quantify, visualize, and predict radiation therapy (RT) process reliability using data captured from a voluntary incident learning system, with an overall aim to improve patient safety outcomes. METHODS AND MATERIALS: We analyzed 111 reported deviations that were tripped and caught within 159 mapped RT process steps included within 7 major stages of RT delivery, 94 of which were any type of quality assurance (QA) controls. This allowed for us to compute the trip rate and fail-to-catch-rate (FCR) per each QA control with the available data. Next, we used a logistic regression model to identify significant variables predictive of FCRs, predicted FCRs for each QA control without available data, and thus, attempted to quantify RT process reliability expressed as percentage of patients with uncaught deviations after treatment planning, before their first treatment, and during treatment delivery. RESULTS: Using the predicted FCRs, we computed the upper 95% likelihood that a deviation remains uncaught in a patient's course of treatment at the following RT process stages: immediately after treatment planning at 10.26%; before the first treatment at 0.0052%; and throughout treatment delivery at 0.0276%. CONCLUSIONS: The results suggest that RT process reliability can be predicted and visualized using data from incident learning systems. If implemented and used as a safety metric, this could help RT clinics to proactively maintain their preoccupation with patient safety. RT process reliability may also help guide future work on standardization and continuous improvement of the design of RT QA programs.

Promoting safety mindfulness: Recommendations for the design and use of simulation-based training in radiation therapy.

There is a need to better prepare radiation therapy (RT) providers to safely operate within the health information technology (IT) sociotechnical system. Simulation-based training has been preemptively used to yield meaningful improvements during providers' interactions with health IT, including RT settings. Therefore, on the basis of the available literature and our experience, we propose principles for the effective design and use of simulated scenarios and describe a conceptual framework for a debriefing approach to foster successful training that is focused on safety mindfulness during RT professionals' interactions with health IT.

Developing and assessing electronic checklists for safety mindfulness, workload, and performance.

PURPOSE: The aim of this study is to propose a set of innovative principles for the effective design of electronic checklists to enhance safety mindfulness (a specific safety mindful mindset that offers the opportunity to operate more preemptively during routine quality assurance tasks) and discuss some of our preliminary results from testing our proposed electronic checklist with dosimetrists and physicists. METHODS AND MATERIALS: A multidisciplinary team designed, developed, and evaluated the utility of the electronic checklist (vs paper-based checklist) to promote safety mindfulness. Subjective workload was measured at the end of each assessment/scenario. Performance was quantified on the basis of discovery of purposefully embedded errors, time to complete the scenario, and additional concerns that were documented by the participants. RESULTS: Use of the electronic checklist was associated with decreases in time to scenario completion (P < .01) and increases in documentation of additional patient safety and plan quality concerns (P = .04) but had no significant impact on the recognition of purposefully embedded errors or perceptions of workload. CONCLUSIONS: Our proposed principles for the design of electronic checklists may improve the efficiency of quality assurance procedures while enhancing users' safety mindfulness. Future research is needed to better understand the utility of our proposed design principles on patient safety from a long-term use perspective.

Identifying Factors and Root Causes Associated With Near-Miss or Safety Incidents in Patients Treated With Radiotherapy: A Case-Control Analysis.

PURPOSE: To identify factors associated with a near-miss or safety incident (NMSI) in patients undergoing radiotherapy and identify common root causes of NMSIs and their relationship with incident severity. METHODS: We retrospectively studied NMSIs filed between October 2014 and April 2016. We extracted patient-, treatment-, and disease-specific data from patients with an NMSI (n = 200; incident group) and a similar group of control patients (n = 200) matched in time, without an NMSI. A root cause and incident severity were determined for each NMSI. Univariable and multivariable analyses were performed to determine which specific factors were contributing to NMSIs. Multivariable logistic regression was used to determine root causes of NMSIs and their relationship with incident severity. RESULTS: NMSIs were associated with the following factors: head and neck sites (odds ratio [OR], 5.2; P = .01), image-guided intensity-modulated radiotherapy (OR, 3; P = .009), daily imaging (OR, 7; P < .001), and tumors staged as T2 (OR, 3.3; P = .004). Documentation and scheduling errors were the most common root causes (29%). Communication errors were more likely to affect patients ( P < .001), and technical treatment delivery errors were most associated with a higher severity score ( P = .005). CONCLUSION: Several treatment- and disease-specific factors were found to be associated with an NMSI. Overall, our results suggest that complexity (eg, head and neck, image-guided intensity-modulated radiotherapy, and daily imaging) might be a contributing factor for an NMSI. This promotes an idea of developing a more dedicated and robust quality assurance system for complex cases and highlights the importance of a strong reporting system to support a safety culture.

Improving radiation oncology providers' workload and performance: Can simulation-based training help?

PURPOSE: To help with ongoing safety challenges in radiation therapy (RT), the objective of this research was to develop and assess the impact of a simulation-based training intervention on radiation oncology providers' workload and performance during treatment planning and quality assurance (QA) tasks. METHODS AND MATERIALS: Eighteen radiation oncology professionals completed routine treatment planning and QA tasks on 2 clinical scenarios in a simulation laboratory as part of a prospective institutional review board-approved study. Workload was measured at the end of each assessment/scenario using the NASA Task-Load Index. Performance was quantified based on procedural compliance (adherence to preset/standard QA tasks), time-to-scenario completion, and clinically relevant performance. Participants were then randomized to receive (vs not receive) simulation-based training intervention (eg, standardized feedback on workload and performance) and underwent repeat measurements of workload and performance. Pre- and postintervention changes in workload and performance from participants who received (vs did not receive) were compared using 2-way analysis of variance. RESULTS: Simulation-based training was associated with significant improvements in procedural compliance (P = .01) and increases in time-to-scenario completion (P < .01) but had no significant impact on subjective workload or clinically relevant performance. CONCLUSION: Simulation-based training may be a tool to improve procedural compliance of RT professionals and to acquire new skills and knowledge to proactively maintain RT professionals' preoccupation with patient safety.

Relating physician's workload with errors during radiation therapy planning.

PURPOSE: To relate subjective workload (WL) levels to errors for routine clinical tasks. METHODS AND MATERIALS: Nine physicians (4 faculty and 5 residents) each performed 3 radiation therapy planning cases. The WL levels were subjectively assessed using National Aeronautics and Space Administration Task Load Index (NASA-TLX). Individual performance was assessed objectively based on the severity grade of errors. The relationship between the WL and performance was assessed via ordinal logistic regression. RESULTS: There was an increased rate of severity grade of errors with increasing WL (P value = .02). As the majority of the higher NASA-TLX scores, and the majority of the performance errors were in the residents, our findings are likely most pertinent to radiation oncology centers with training programs. CONCLUSIONS: WL levels may be an important factor contributing to errors during radiation therapy planning tasks.

Subjective and objective quantification of physician's workload and performance during radiation therapy planning tasks.

PURPOSE: To quantify, and compare, workload for several common physician-based treatment planning tasks using objective and subjective measures of workload. To assess the relationship between workload and performance to define workload levels where performance could be expected to decline. METHODS AND MATERIALS: Nine physicians performed the same 3 tasks on each of 2 cases ("easy" vs "hard"). Workload was assessed objectively throughout the tasks (via monitoring of pupil size and blink rate), and subjectively at the end of each case (via National Aeronautics and Space Administration Task Load Index; NASA-TLX). NASA-TLX assesses the 6 dimensions (mental, physical, and temporal demands, frustration, effort, and performance); scores > or ≈ 50 are associated with reduced performance in other industries. Performance was measured using participants' stated willingness to approve the treatment plan. Differences in subjective and objective workload between cases, tasks, and experience were assessed using analysis of variance (ANOVA). The correlation between subjective and objective workload measures were assessed via the Pearson correlation test. The relationships between workload and performance measures were assessed using the t test. RESULTS: Eighteen case-wise and 54 task-wise assessments were obtained. Subjective NASA-TLX scores (P < .001), but not time-weighted averages of objective scores (P > .1), were significantly lower for the easy vs hard case. Most correlations between the subjective and objective measures were not significant, except between average blink rate and NASA-TLX scores (r = -0.34, P = .02), for task-wise assessments. Performance appeared to decline at NASA-TLX scores of ≥55. CONCLUSIONS: The NASA-TLX may provide a reasonable method to quantify subjective workload for broad activities, and objective physiologic eye-based measures may be useful to monitor workload for more granular tasks within activities. The subjective and objective measures, as herein quantified, do not necessarily track each other, and more work is needed to assess their utilities. From a series of controlled experiments, we found that performance appears to decline at subjective workload levels ≥55 (as measured via NASA-TLX), which is consistent with findings from other industries.