A multi-institutional assessment of COVID-19-related risk in radiation oncology.

PURPOSE: The COVID-19 pandemic has presented challenges to delivering safe and timely care for cancer patients. The oncology community has undertaken substantial workflow adaptations to reduce transmission risk for patients and providers. While various control measureshave been proposed and implemented, little is known about their impact on safety of the radiation oncology workflow and potential for transmission. The objective of this study was to assess potential safety impacts of control measures employed during the COVID-19 pandemic. METHODS: A multi-institutional study was undertaken to assess the risks of pandemic-associated workflow adaptations using failure mode and effects analysis (FMEA). Failure modes were identified and scored using FMEA formalism. FMEA scores were used to identify highest-risk aspects of the radiation therapy process. The impact of control measures on overall risk was quantified. Agreement among institutions was evaluated. RESULTS: Thirty three failure modes and 22 control measures were identified. Control measures resulted in risk score reductions for 22 of the failure modes, with the largest reductions from screening of patients and staff, requiring use of masks, and regular cleaning of patient areas. The median risk score for all failure modes was reduced from 280 to 168. There was high institutional agreement for 90.3% of failure modes but only 47% of control measures. CONCLUSIONS: COVID-related risks are similar across oncology practices in this study. While control measures can reducerisk, their use varied. The effectiveness of control measures on risk may guide selection of the highest-impact workflow adaptions to ensure safe care in oncology.

Durable Improvement in Patient Safety Culture Over 5 Years With Use of High-volume Incident Learning System.

PURPOSE: Incident learning systems (ILSs) require substantial time and effort to maintain, risking staff burnout and ILS disuse. Herein, we assess the durability of ILS-associated safety culture improvements and ILS engagement at 5 years. METHODS AND MATERIALS: A validated survey assessing safety culture was administered to all staff of an academic radiation oncology department before starting ILS and annually thereafter for 5 years. The survey consists of 70 questions assessing key cultural domains, overall patient safety grade, and barriers to incident reporting. A χ2 test was used to compare baseline scores before starting the ILS (pre-ILS) with the aggregate 5 years during which ILS was in use (with ILS). ILS engagement was measured by the self-reported number of ILS entries submitted in the previous 12 months. RESULTS: The survey response rate was ≥68% each year (range, 68%-80%). High-volume event reporting was sustained (4673 reports; average of 0.9 ILS entries per treatment course). ILS engagement increased, with 43% of respondents submitting reports during the 12 months pre-ILS compared with 64% with ILS in use (P < .001). Significant improvements (pre- vs. with-ILS) were observed in the cultural domains of patient safety perceptions (25% vs 39%; P < .03), and responsibility and self-efficacy (43% vs 60%; P < .01). The overall patient safety grade of very good or excellent significantly increased (69% vs 85%; P < .01). Significant reductions were seen in the following barriers to error reporting: embarrassment in front of colleagues, getting colleagues into trouble, and effect on department reputation. CONCLUSIONS: Comprehensive incident learning was sustained over 5 years and is associated with significant durable improvements in metrics of patient safety culture.

Evaluation of near-miss and adverse events in radiation oncology using a comprehensive causal factor taxonomy.

PURPOSE: Incident learning systems (ILSs) are a popular strategy for improving safety in radiation oncology (RO) clinics, but few reports focus on the causes of errors in RO. The goal of this study was to test a causal factor taxonomy developed in 2012 by the American Association of Physicists in Medicine and adopted for use in the RO: Incident Learning System (RO-ILS). METHODS AND MATERIALS: Three hundred event reports were randomly selected from an institutional ILS database and Safety in Radiation Oncology (SAFRON), an international ILS. The reports were split into 3 groups of 100 events each: low-risk institutional, high-risk institutional, and SAFRON. Three raters retrospectively analyzed each event for contributing factors using the American Association of Physicists in Medicine taxonomy. RESULTS: No events were described by a single causal factor (median, 7). The causal factor taxonomy was found to be applicable for all events, but 4 causal factors were not described in the taxonomy: linear accelerator failure (n = 3), hardware/equipment failure (n = 2), failure to follow through with a quality improvement intervention (n = 1), and workflow documentation was misleading (n = 1). The most common causal factor categories contributing to events were similar in all event types. The most common specific causal factor to contribute to events was a "slip causing physical error." Poor human factors engineering was the only causal factor found to contribute more frequently to high-risk institutional versus low-risk institutional events. CONCLUSIONS: The taxonomy in the study was found to be applicable for all events and may be useful in root cause analyses and future studies. Communication and human behaviors were the most common errors affecting all types of events. Poor human factors engineering was found to specifically contribute to high-risk more than low-risk institutional events, and may represent a strategy for reducing errors in all types of events.

Are we making an impact with incident learning systems? Analysis of quality improvement interventions using total body irradiation as a model system.

PURPOSE: Despite increasing interest in incident learning systems (ILS) to improve safety and quality in radiation oncology, little is known about interventions developed in response to safety data. We used total body irradiation (TBI) as a model system to study the effectiveness of interventions from our institutional ILS. METHODS AND MATERIALS: Near-miss event reports specific to TBI were identified from a departmental ILS from March 2012 to December 2015. The near-miss risk index was rated at multidisciplinary review from 0 (no potential harm) to 4 (critical potential harm). Interventions were analyzed for effectiveness with a schema adapted from The Joint Commission and other agencies: "most reliable" (eg, forcing functions, automation), "somewhat reliable" (eg, checklists, standardization), and "least reliable" (eg, training, rules, procedures). Causal factors of each event were drawn from the casual factor schema used in radiation oncology ILS. RESULTS: Of 4007 safety-related reports, 266 reports pertained to TBI. TBI reports had a somewhat higher proportion of high-risk events (near-miss risk index 3-4) compared with non-TBI reports (25% vs 17%, P = .0045). A total of 117 interventions were implemented. The reliability indicators for the interventions were: most reliable (11% of interventions), somewhat reliable (17%), and least reliable (72%). Interventions were more likely to be applied to high-risk events (54% vs 41%, P = .03). There was a pattern of high-reliability interventions with increased risk score of events. Events involving human error (eg, slips) and equipment/information technology lent themselves more often to high-reliability interventions. Events related to communication, standardization, and training were associated with low-reliability interventions. CONCLUSIONS: Over a 3.5-year period, 117 quality improvement strategies were developed for TBI based on ILS. Interventions were more likely to be applied to high-risk events and high-risk events were more likely to be associated with high-quality interventions. These results may be useful to institutions seeking to develop interventions based on ILS data.

Targeting safety improvements through identification of incident origination and detection in a near-miss incident learning system.

PURPOSE: Radiation treatment planning involves a complex workflow that has multiple potential points of vulnerability. This study utilizes an incident reporting system to identify the origination and detection points of near-miss errors, in order to guide their departmental safety improvement efforts. Previous studies have examined where errors arise, but not where they are detected or applied a near-miss risk index (NMRI) to gauge severity. METHODS: From 3/2012 to 3/2014, 1897 incidents were analyzed from a departmental incident learning system. All incidents were prospectively reviewed weekly by a multidisciplinary team and assigned a NMRI score ranging from 0 to 4 reflecting potential harm to the patient (no potential harm to potential critical harm). Incidents were classified by point of incident origination and detection based on a 103-step workflow. The individual steps were divided among nine broad workflow categories (patient assessment, imaging for radiation therapy (RT) planning, treatment planning, pretreatment plan review, treatment delivery, on-treatment quality management, post-treatment completion, equipment/software quality management, and other). The average NMRI scores of incidents originating or detected within each broad workflow area were calculated. Additionally, out of 103 individual process steps, 35 were classified as safety barriers, the process steps whose primary function is to catch errors. The safety barriers which most frequently detected incidents were identified and analyzed. Finally, the distance between event origination and detection was explored by grouping events by the number of broad workflow area events passed through before detection, and average NMRI scores were compared. RESULTS: Near-miss incidents most commonly originated within treatment planning (33%). However, the incidents with the highest average NMRI scores originated during imaging for RT planning (NMRI = 2.0, average NMRI of all events = 1.5), specifically during the documentation of patient positioning and localization of the patient. Incidents were most frequently detected during treatment delivery (30%), and incidents identified at this point also had higher severity scores than other workflow areas (NMRI = 1.6). Incidents identified during on-treatment quality management were also more severe (NMRI = 1.7), and the specific process steps of reviewing portal and CBCT images tended to catch highest-severity incidents. On average, safety barriers caught 46% of all incidents, most frequently at physics chart review, therapist's chart check, and the review of portal images; however, most of the incidents that pass through a particular safety barrier are not designed to be capable of being captured at that barrier. CONCLUSIONS: Incident learning systems can be used to assess the most common points of error origination and detection in radiation oncology. This can help tailor safety improvement efforts and target the highest impact portions of the workflow. The most severe near-miss events tend to originate during simulation, with the most severe near-miss events detected at the time of patient treatment. Safety barriers can be improved to allow earlier detection of near-miss events.

Interrater reliability of a near-miss risk index for incident learning systems in radiation oncology.

PURPOSE: Tools for assessing the severity and risk of near-miss events in radiation oncology are few and needed. Recent work has described guidelines for the use of a 5-tier near-miss risk index (NMRI) for the classification of near-miss events. The purpose of this study was to assess the reliability of the NMRI among users in a radiation oncology department. METHODS AND MATERIALS: Reliability of the NMRI was assessed using an online survey distributed to members of a radiation oncology department. The survey contained 70 events extracted from the department's incident learning system (ILS). Survey participants rated each event using the NMRI guidelines, reported their attendance to weekly ILS meetings (used as a surrogate for familiarity with the ILS), and indicated their familiarity with the radiation oncology workflow. Interrater reliability was determined using Krippendorff's alpha. Use of the NMRI to rate actual events during 5 weekly ILS meetings was also assessed and interrater reliability determined. RESULTS: Twenty-eight survey respondents represented a wide variety of care providers. Krippendorff's alpha was calculated for the whole respondent cohort to be 0.376, indicating fair agreement among raters. Respondents who had the most participation at ILS meetings (n = 4) had moderate agreement with an alpha of 0.501. Interestingly, there were significant differences in reliability and median NMRI scores between professions. NMRI use during weekly NMRI meetings (80 events rated), participants showed moderate reliability (alpha = 0.607). CONCLUSIONS: Using the NMRI guidelines, raters from a wide variety of professions were able to assess the severity of near-miss incidents with fair agreement. Those experienced with the ILS showed better agreement, and higher agreement was seen during multidisciplinary ILS meetings. These data support the use the indices such as the NMRI for near-miss risk assessment in patient safety and prioritization of process improvements in radiation oncology.

Influence of planning time and treatment complexity on radiation therapy errors.

PURPOSE: Radiation treatment planning is a complex process with potential for error. We hypothesized that shorter time from simulation to treatment would result in rushed work and higher incidence of errors. We examined treatment planning factors predictive for near-miss events. METHODS AND MATERIALS: Treatments delivered from March 2012 through October 2014 were analyzed. Near-miss events were prospectively recorded and coded for severity on a 0 to 4 scale; only grade 3-4 (potentially severe/critical) events were studied in this report. For 4 treatment types (3-dimensional conformal, intensity modulated radiation therapy, stereotactic body radiation therapy [SBRT], neutron), logistic regression was performed to test influence of treatment planning time and clinical variables on near-miss events. RESULTS: There were 2257 treatment courses during the study period, with 322 grade 3-4 near-miss events. SBRT treatments had more frequent events than the other 3 treatment types (18% vs 11%, P = .04). For the 3-dimensional conformal group (1354 treatments), univariate analysis showed several factors predictive of near-miss events: longer time from simulation to first treatment (P = .01), treatment of primary site versus metastasis (P < .001), longer treatment course (P < .001), and pediatric versus adult patient (P = .002). However, on multivariate regression only pediatric versus adult patient remained predictive of events (P = 0.02). For the intensity modulated radiation therapy, SBRT, and neutron groups, time between simulation and first treatment was not found to be predictive of near-miss events on univariate or multivariate regression. CONCLUSIONS: When controlling for treatment technique and other clinical factors, there was no relationship between time spent in radiation treatment planning and near-miss events. SBRT and pediatric treatments were more error-prone, indicating that clinical and technical complexity of treatments should be taken into account when targeting safety interventions.

Can emergent treatments result in more severe errors?: An analysis of a large institutional near-miss incident reporting database.

PURPOSE: Emergent radiation treatments may be subject to more errors because of the compressed time frame. Few data exist on the magnitude of this problem or how to guide safety improvement interventions. The purpose of this study is to examine patterns of near-miss events in emergent treatments using a large institutional incident reporting system. METHODS AND MATERIALS: Events in the incident reporting database from February 2012 to October 2013 were reviewed prospectively by a multidisciplinary team to identify emergent treatments. Reports were scored for potential near-miss risk index (NMRI) on a 0 to 4 scale. Workflow steps of where events originated and were detected were analyzed. Events were categorized by use of the causal factor system from the Radiation Oncology Incident Learning System. Mann-Whitney U tests were used to compare mean NMRI score, and Fisher exact tests were performed to compare the proportion of high-risk events between emergent and nonemergent treatments and between emergent treatments on weekdays and weekends or holidays. RESULTS: Over the study period, approximately 1600 patients were treated, 190 of them emergently. Seventy-one incident reports were submitted for 55 unique patients. Fewer events were reported for emergent treatments than for nonemergent treatments (0.37 events per new treatment vs 0.86; P < .01). Mean risk index for emergent reports was 1.90 versus 1.48 for nonemergent reports (P < .01). Rate of NMRI 4 was 10% for emergent treatments versus 4% for nonemergent treatments (P < .01). Emergent treatments started on a weekend or holiday had a higher proportion of critical near-miss events than emergent treatments started during the week (37% vs 7.9%, P = .034). CONCLUSIONS: In this study, fewer near-miss incidents were reported per treatment course for emergent treatments. This may be attributable to reporting bias. More importantly, when emergent near misses occur, they are of greater severity.

Metrics of success: Measuring impact of a departmental near-miss incident learning system.

PURPOSE: There is a growing interest in the application of incident learning systems (ILS) to radiation oncology. The purpose of the present study is to define statistical metrics that may serve as benchmarks for successful operation of an incident learning system. METHODS AND MATERIALS: A departmental safety and quality ILS was developed to monitor errors, near-miss events, and process improvement suggestions. Event reports were reviewed by a multiprofessional quality improvement committee. Events were scored by a near-miss risk index (NMRI) and categorized by event point of origination and discovery. Reporting trends were analyzed over a 2-year period, including total number and rates of events reported, users reporting, NMRI, and event origination and discovery. RESULTS: A total of 1897 reports were evaluated (1.0 reports/patient, 0.9 reports/unique treatment course). Participation in the ILS increased as demonstrated by total events (2.1 additional reports/month) and unique users (0.5 new users/month). Sixteen percent of reports had an NMRI of 0 (none), 42% had an NMRI of 1 (mild), 25% had an NMRI of 2 (moderate), 12% had an NMRI of 3 (severe), and 5% had an NMRI of 4 (critical). Event NMRI showed a significant decrease in the first 6 months (1.68-1.42, P < .001). Trends in origination and discovery of reports were broadly distributed between radiation therapy process steps and staff groups. The highest risk events originated in imaging for treatment planning (NMRI = 2.0 ± 1.1; P < .0001) and were detected in on-treatment quality management (NMRI = 1.7 ± 1.1; P = .003). CONCLUSIONS: Over the initial 2-year period of ILS operation, rates of reporting increased, staff participation increased, and NMRI of reported events declined. These data mirror previously reported findings of improvement in safety culture endpoints. These metrics may be useful for other institutions seeking to create or evaluate their own ILS.

Measurable improvement in patient safety culture: A departmental experience with incident learning.

PURPOSE: Rigorous use of departmental incident learning is integral to improving patient safety and quality of care. The goal of this study was to quantify the impact of a high-volume, departmental incident learning system on patient safety culture. METHODS AND MATERIALS: A prospective, voluntary, electronic incident learning system was implemented in February 2012 with the intent of tracking near-miss/no-harm incidents. All incident reports were reviewed weekly by a multiprofessional team with regular department-wide feedback. Patient safety culture was measured at baseline with validated patient safety culture survey questions. A repeat survey was conducted after 1 and 2 years of departmental incident learning. Proportional changes were compared by χ(2) or Fisher exact test, where appropriate. RESULTS: Between 2012 and 2014, a total of 1897 error/near-miss incidents were reported, representing an average of 1 near-miss report per patient treated. Reports were filed by a cross section of staff, with the majority of incidents reported by therapists, dosimetrists, and physicists. Survey response rates at baseline and 1 and 2 years were 78%, 80%, and 80%, respectively. Statistically significant and sustained improvements were noted in several safety metrics, including belief that the department was openly discussing ways to improve safety, the sense that reports were being used for safety improvement, and the sense that changes were being evaluated for effectiveness. None of the surveyed dimensions of patient safety culture worsened. Fewer punitive concerns were noted, with statistically significant decreases in the worry of embarrassment in front of colleagues and fear of getting colleagues in trouble. CONCLUSIONS: A comprehensive incident learning system can identify many areas for improvement and is associated with significant and sustained improvements in patient safety culture. These data provide valuable guidance as incident learning systems become more widely used in radiation oncology.