Implementation of machine learning in the clinic: challenges and lessons in prospective deployment from the System for High Intensity EvaLuation During Radiation Therapy (SHIELD-RT) randomized controlled study.

BACKGROUND: Artificial intelligence (AI) and machine learning (ML) have resulted in significant enthusiasm for their promise in healthcare. Despite this, prospective randomized controlled trials and successful clinical implementation remain limited. One clinical application of ML is mitigation of the increased risk for acute care during outpatient cancer therapy. We previously reported the results of the System for High Intensity EvaLuation During Radiation Therapy (SHIELD-RT) study (NCT04277650), which was a prospective, randomized quality improvement study demonstrating that ML based on electronic health record (EHR) data can direct supplemental clinical evaluations and reduce the rate of acute care during cancer radiotherapy with and without chemotherapy. The objective of this study is to report the workflow and operational challenges encountered during ML implementation on the SHIELD-RT study. RESULTS: Data extraction and manual review steps in the workflow represented significant time commitments for implementation of clinical ML on a prospective, randomized study. Barriers include limited data availability through the standard clinical workflow and commercial products, the need to aggregate data from multiple sources, and logistical challenges from altering the standard clinical workflow to deliver adaptive care. CONCLUSIONS: The SHIELD-RT study was an early randomized controlled study which enabled assessment of barriers to clinical ML implementation, specifically those which leverage the EHR. These challenges build on a growing body of literature and may provide lessons for future healthcare ML adoption. TRIAL REGISTRATION: NCT04277650. Registered 20 February 2020. Retrospectively registered quality improvement study.

Health Care Cost Reductions with Machine Learning-Directed Evaluations during Radiation Therapy - An Economic Analysis of a Randomized Controlled Study.

BACKGROUND: Machine learning (ML) may cost-effectively direct health care by identifying patients most likely to benefit from preventative interventions to avoid negative and expensive outcomes. System for High-Intensity Evaluation During Radiation Therapy (SHIELD-RT; NCT04277650) was a single-institution, randomized controlled study in which electronic health record-based ML accurately identified patients at high risk for acute care (emergency visit or hospitalization) during radiotherapy (RT) and targeted them for supplemental clinical evaluations. This ML-directed intervention resulted in decreased acute care utilization. Given the limited prospective data showing the ability of ML to direct interventions cost-efficiently, an economic analysis was performed. METHODS: A post hoc economic analysis was conducted of SHIELD-RT that included RT courses from January 7, 2019, to June 30, 2019. ML-identified high-risk courses (≥10% risk of acute care during RT) were randomized to receive standard of care weekly clinical evaluations with ad hoc supplemental evaluations per clinician discretion versus mandatory twice-weekly evaluations. The primary outcome was difference in mean total medical costs during and 15 days after RT. Acute care costs were obtained via institutional cost accounting. Physician and intervention costs were estimated via Medicare and Medicaid data. Negative binomial regression was used to estimate cost outcomes after adjustment for patient and disease factors. RESULTS: A total of 311 high-risk RT courses among 305 patients were randomized to the standard (n=157) or the intervention (n=154) group. Unadjusted mean intervention group supplemental visit costs were $155 per course (95% confidence interval, $142 to $168). The intervention group had fewer acute care visits per course (standard, 0.47; intervention, 0.31; P=0.04). Total mean adjusted costs were $3110 per course for the standard group and $1494 for the intervention group (difference in means, $1616 [95% confidence interval, $1450 to $1783]; P=0.03). CONCLUSIONS: In this economic analysis of a randomized controlled, health care ML study, mandatory supplemental evaluations for ML-identified high-risk patients were associated with both reduced total medical costs and improved clinical outcomes. Further study is needed to determine whether economic results are generalizable. (Funded in part by The Duke Endowment, The Conquer Cancer Foundation, the Duke Department of Radiation Oncology, and the National Cancer Institute of the National Institutes of Health [R01CA277782]; ClinicalTrials.gov number, NCT04277650.).

Migration of implanted markers for image-guided lung tumor stereotactic ablative radiotherapy.

The purpose of this study was to quantify postimplantation migration of percutaneously implanted cylindrical gold seeds ("seeds") and platinum endovascular embolization coils ("coils") for tumor tracking in pulmonary stereotactic ablative radiotherapy (SABR). We retrospectively analyzed the migration of markers in 32 consecutive patients with computed tomography scans postimplantation and at simulation. We implanted 147 markers (59 seeds, 88 coils) in or around 34 pulmonary tumors over 32 procedures, with one lesion implanted twice. Marker coordinates were rigidly aligned by minimizing fiducial registration error (FRE), the root mean square of the differences in marker locations for each tumor between scans. To also evaluate whether single markers were responsible for most migration, we aligned with and without the outlier causing the largest FRE increase per tumor. We applied the resultant transformation to all markers. We evaluated migration of individual markers and FRE of each group. Median scan interval was 8 days. Median individual marker migration was 1.28 mm (interquartile range [IQR] 0.78-2.63 mm). Median lesion FRE was 1.56 mm (IQR 0.92-2.95 mm). Outlier identification yielded 1.03 mm median migration (IQR 0.52-2.21 mm) and 1.97 mm median FRE (IQR 1.44-4.32 mm). Outliers caused a mean and median shift in the centroid of 1.22 and 0.80 mm (95th percentile 2.52 mm). Seeds and coils had no statistically significant difference. Univariate analysis suggested no correlation of migration with the number of markers, contact with the chest wall, or time elapsed. Marker migration between implantation and simulation is limited and unlikely to cause geometric miss during tracking.

Healthcare provider evaluation of machine learning-directed care: reactions to deployment on a randomised controlled study.

OBJECTIVES: Clinical artificial intelligence and machine learning (ML) face barriers related to implementation and trust. There have been few prospective opportunities to evaluate these concerns. System for High Intensity EvaLuation During Radiotherapy (NCT03775265) was a randomised controlled study demonstrating that ML accurately directed clinical evaluations to reduce acute care during cancer radiotherapy. We characterised subsequent perceptions and barriers to implementation. METHODS: An anonymous 7-question Likert-type scale survey with optional free text was administered to multidisciplinary staff focused on workflow, agreement with ML and patient experience. RESULTS: 59/71 (83%) responded. 81% disagreed/strongly disagreed their workflow was disrupted. 67% agreed/strongly agreed patients undergoing intervention were high risk. 75% agreed/strongly agreed they would implement the ML approach routinely if the study was positive. Free-text feedback focused on patient education and ML predictions. CONCLUSIONS: Randomised data and firsthand experience support positive reception of clinical ML. Providers highlighted future priorities, including patient counselling and workflow optimisation.

System for High-Intensity Evaluation During Radiation Therapy (SHIELD-RT): A Prospective Randomized Study of Machine Learning-Directed Clinical Evaluations During Radiation and Chemoradiation.

PURPOSE: Patients undergoing outpatient radiotherapy (RT) or chemoradiation (CRT) frequently require acute care (emergency department evaluation or hospitalization). Machine learning (ML) may guide interventions to reduce this risk. There are limited prospective studies investigating the clinical impact of ML in health care. The objective of this study was to determine whether ML can identify high-risk patients and direct mandatory twice-weekly clinical evaluation to reduce acute care visits during treatment. PATIENTS AND METHODS: During this single-institution randomized quality improvement study (ClinicalTrials.gov identifier: NCT04277650), 963 outpatient adult courses of RT and CRT started from January 7 to June 30, 2019, were evaluated by an ML algorithm. Among these, 311 courses identified by ML as high risk (> 10% risk of acute care during treatment) were randomized to standard once-weekly clinical evaluation (n = 157) or mandatory twice-weekly evaluation (n = 154). Both arms allowed additional evaluations on the basis of clinician discretion. The primary end point was the rate of acute care visits during RT. Model performance was evaluated using receiver operating characteristic area under the curve (AUC) and decile calibration plots. RESULTS: Twice-weekly evaluation reduced rates of acute care during treatment from 22.3% to 12.3% (difference, -10.0%; 95% CI, -18.3 to -1.6; relative risk, 0.556; 95% CI, 0.332 to 0.924; P = .02). Low-risk patients had a 2.7% acute care rate. Model discrimination was good in high- and low-risk patients undergoing standard once-weekly evaluation (AUC, 0.851). CONCLUSION: In this prospective randomized study, ML accurately triaged patients undergoing RT and CRT, directing clinical management with reduced acute care rates versus standard of care. This prospective study demonstrates the potential benefit of ML in health care and offers opportunities to enhance care quality and reduce health care costs.

Mid-radiotherapy PET/CT for prognostication and detection of early progression in patients with stage III non-small cell lung cancer.

BACKGROUND AND PURPOSE: Pre- and mid-radiotherapy FDG-PET metrics have been proposed as biomarkers of recurrence and survival in patients treated for stage III non-small cell lung cancer. We evaluated these metrics in patients treated with definitive radiation therapy (RT). We also evaluated outcomes after progression on mid-radiotherapy PET/CT. MATERIAL AND METHODS: Seventy-seven patients treated with RT with or without chemotherapy were included in this retrospective study. Primary tumor and involved nodes were delineated. PET metrics included metabolic tumor volume (MTV), total lesion glycolysis (TLG), and SUVmax. For mid-radiotherapy PET, both absolute value of these metrics and percentage decrease were analyzed. The influence of PET metrics on time to death, local recurrence, and regional/distant recurrence was assessed using Cox regression. RESULTS: 91% of patients had concurrent chemotherapy. Median follow-up was 14months. None of the PET metrics were associated with overall survival. Several were positively associated with local recurrence: pre-radiotherapy MTV, and mid-radiotherapy MTV and TLG (p=0.03-0.05). Ratio of mid- to pre-treatment SUVmax was associated with regional/distant recurrence (p=0.02). 5/77 mid-radiotherapy scans showed early out-of-field progression. All of these patients died. CONCLUSIONS: Several PET metrics were associated with risk of recurrence. Progression on mid-radiotherapy PET/CT was a poor prognostic factor.

Feasibility and potential utility of multicomponent exhaled breath analysis for predicting development of radiation pneumonitis after stereotactic ablative radiotherapy.

INTRODUCTION: In this prospective pilot study, we evaluated the feasibility and potential utility of measuring multiple exhaled gases as biomarkers of radiation pneumonitis (RP) in patients receiving stereotactic ablative radiotherapy (SABR) for lung tumors. METHODS: Breath analysis was performed for 26 patients receiving SABR for lung tumors. Concentrations of exhaled nitric oxide (eNO), carbon monoxide (eCO), nitrous oxide (eN2O), and carbon dioxide (eCO2) were measured before and immediately after each fraction using real-time, infrared laser spectroscopy. RP development (CTCAE grade ≥2) was correlated with baseline gas concentrations, acute changes in gas concentrations after each SABR fraction, and dosimetric parameters. RESULTS: Exhaled breath analysis was successfully completed in 77% of patients. Five of 20 evaluable patients developed RP at a mean of 5.4 months after SABR. Acute changes in eNO and eCO concentrations, defined as percent changes between each pre-fraction and post-fraction measurement, were significantly smaller in RP versus non-RP cases (p = 0.022 and 0.015, respectively). In an exploratory analysis, a combined predictor of baseline eNO greater than 24 parts per billion and acute decrease in eCO less than 5.5% strongly correlated with RP incidence (p =0.0099). Neither eN2O nor eCO2 concentrations were significantly associated with RP development. Although generally higher in patients destined to develop RP, dosimetric parameters were not significantly associated with RP development. CONCLUSIONS: The majority of SABR patients in this pilot study were able to complete exhaled breath analysis. Baseline concentrations and acute changes in concentrations of exhaled breath components were associated with RP development after SABR. If our findings are validated, exhaled breath analysis may become a useful approach for noninvasive identification of patients at highest risk for developing RP after SABR.

An ultrasensitive method for quantitating circulating tumor DNA with broad patient coverage.

Circulating tumor DNA (ctDNA) is a promising biomarker for noninvasive assessment of cancer burden, but existing ctDNA detection methods have insufficient sensitivity or patient coverage for broad clinical applicability. Here we introduce cancer personalized profiling by deep sequencing (CAPP-Seq), an economical and ultrasensitive method for quantifying ctDNA. We implemented CAPP-Seq for non-small-cell lung cancer (NSCLC) with a design covering multiple classes of somatic alterations that identified mutations in >95% of tumors. We detected ctDNA in 100% of patients with stage II-IV NSCLC and in 50% of patients with stage I, with 96% specificity for mutant allele fractions down to ∼0.02%. Levels of ctDNA were highly correlated with tumor volume and distinguished between residual disease and treatment-related imaging changes, and measurement of ctDNA levels allowed for earlier response assessment than radiographic approaches. Finally, we evaluated biopsy-free tumor screening and genotyping with CAPP-Seq. We envision that CAPP-Seq could be routinely applied clinically to detect and monitor diverse malignancies, thus facilitating personalized cancer therapy.

Clinical impact of dose overestimation by effective path length calculation in stereotactic ablative radiation therapy of lung tumors.

PURPOSE: To determine the clinical impact of calculated dose differences between effective path length (EPL) and Monte Carlo (MC) algorithms in stereotactic ablative radiation therapy (SABR) of lung tumors. METHODS AND MATERIALS: We retrospectively analyzed the treatment plans and clinical outcomes of 77 consecutive patients treated with SABR for 82 lung tumors between 2003 and 2009 at our institution. Sixty treatments were originally planned using EPL, and 22 using MC. All plans were recalculated for the same beam specifications using MC and EPL, respectively. The doses covering 95%, 50%, and 5% (D95, D50, D5, respectively) of the target volumes were compared between EPL and MC (assumed to be the actual delivered dose), both as physical dose and biologically effective dose. Time to local recurrence was correlated with dose by Cox regression analysis. The relationship between tumor control probability (TCP) and biologically effective dose was determined via logistic regression and used to estimate the TCP decrements due to prescribing by EPL calculations. RESULTS: EPL overestimated dose compared with MC in all tumor dose-volume histogram parameters in all plans. The difference was >10% of the MC D95 to the planning target volume and gross tumor volume in 60 of 82 (73%) and 52 of 82 plans (63%), respectively. Local recurrence occurred in 13 of 82 tumors. Controlling for gross tumor volume, higher physical and biologically effective planning target volume D95 correlated significantly with local control (P = .007 and P = .045, respectively). Compared with MC, prescribing based on EPL translated to a median TCP decrement of 4.3% (range, 1.2%-37%) and a >5% decrement in 46% of tumors. CONCLUSIONS: Clinical follow-up for local lung tumor control in a sizable cohort of patients treated with SABR demonstrates that EPL overestimates dose by amounts that substantially decrease TCP in a large proportion. EPL algorithms should be avoided for lung tumor SABR.