Survival in Patients With Spinal Metastatic Disease Treated Nonoperatively With Radiotherapy: Are the SORG-ML Algorithms Relevant?

SUMMARY OF BACKGROUND DATA: The SORG-ML algorithms for survival in spinal metastatic disease were developed in patients who underwent surgery and were externally validated for patients managed operatively. OBJECTIVE: To externally validate the SORG-ML algorithms for survival in spinal metastatic disease in patients managed nonoperatively with radiation. STUDY DESIGN: Retrospective cohort. METHODS: The performance of the SORG-ML algorithms was assessed by discrimination [receiver operating curves and area under the receiver operating curve (AUC)], calibration (calibration plots), decision curve analysis, and overall performance (Brier score). The primary outcomes were 90-day and 1-year mortality. RESULTS: Overall, 2074 adult patients underwent radiation for spinal metastatic disease and 29% (n=521) and 59% (n=917) had 90-day and 1-year mortality, respectively. On complete case analysis (n=415), the AUC was 0.76 (95% CI: 0.71-0.80) and 0.78 (95% CI: 0.73-0.83) for 90-day and 1-year mortality with fair calibration and positive net benefit confirmed by the decision curve analysis. With multiple imputation (n=2074), the AUC was 0.85 (95% CI: 0.83-0.87) and 0.87 (95% CI: 0.85-0.89) for 90-day and 1-year mortality with fair calibration and positive net benefit confirmed by the decision curve analysis. CONCLUSION: The SORG-ML algorithms for survival in spinal metastatic disease generalize well to patients managed nonoperatively with radiation.

Does the SORG Machine-learning Algorithm for Extremity Metastases Generalize to a Contemporary Cohort of Patients? Temporal Validation From 2016 to 2020.

BACKGROUND: The ability to predict survival accurately in patients with osseous metastatic disease of the extremities is vital for patient counseling and guiding surgical intervention. We, the Skeletal Oncology Research Group (SORG), previously developed a machine-learning algorithm (MLA) based on data from 1999 to 2016 to predict 90-day and 1-year survival of surgically treated patients with extremity bone metastasis. As treatment regimens for oncology patients continue to evolve, this SORG MLA-driven probability calculator requires temporal reassessment of its accuracy. QUESTION/PURPOSE: Does the SORG-MLA accurately predict 90-day and 1-year survival in patients who receive surgical treatment for a metastatic long-bone lesion in a more recent cohort of patients treated between 2016 and 2020? METHODS: Between 2017 and 2021, we identified 674 patients 18 years and older through the ICD codes for secondary malignant neoplasm of bone and bone marrow and CPT codes for completed pathologic fractures or prophylactic treatment of an impending fracture. We excluded 40% (268 of 674) of patients, including 18% (118) who did not receive surgery; 11% (72) who had metastases in places other than the long bones of the extremities; 3% (23) who received treatment other than intramedullary nailing, endoprosthetic reconstruction, or dynamic hip screw; 3% (23) who underwent revision surgery, 3% (17) in whom there was no tumor, and 2% (15) who were lost to follow-up within 1 year. Temporal validation was performed using data on 406 patients treated surgically for bony metastatic disease of the extremities from 2016 to 2020 at the same two institutions where the MLA was developed. Variables used to predict survival in the SORG algorithm included perioperative laboratory values, tumor characteristics, and general demographics. To assess the models' discrimination, we computed the c-statistic, commonly referred to as the area under the receiver operating characteristic (AUC) curve for binary classification. This value ranged from 0.5 (representing chance-level performance) to 1.0 (indicating excellent discrimination) Generally, an AUC of 0.75 is considered high enough for use in clinical practice. To evaluate the agreement between predicted and observed outcomes, a calibration plot was used, and the calibration slope and intercept were calculated. Perfect calibration would result in a slope of 1 and intercept of 0. For overall performance, the Brier score and null-model Brier score were determined. The Brier score can range from 0 (representing perfect prediction) to 1 (indicating the poorest prediction). Proper interpretation of the Brier score necessitates a comparison with the null-model Brier score, which represents the score for an algorithm that predicts a probability equal to the population prevalence of the outcome for each patient. Finally, a decision curve analysis was conducted to compare the potential net benefit of the algorithm with other decision-support methods, such as treating all or none of the patients. Overall, 90-day and 1-year mortality were lower in the temporal validation cohort than in the development cohort (90 day: 23% versus 28%; p < 0.001, and 1 year: 51% versus 59%; p<0.001). RESULTS: Overall survival of the patients in the validation cohort improved from 28% mortality at the 90-day timepoint in the cohort on which the model was trained to 23%, and 59% mortality at the 1-year timepoint to 51%. The AUC was 0.78 (95% CI 0.72 to 0.82) for 90-day survival and 0.75 (95% CI 0.70 to 0.79) for 1-year survival, indicating the model could distinguish the two outcomes reasonably. For the 90-day model, the calibration slope was 0.71 (95% CI 0.53 to 0.89), and the intercept was -0.66 (95% CI -0.94 to -0.39), suggesting the predicted risks were overly extreme, and that in general, the risk of the observed outcome was overestimated. For the 1-year model, the calibration slope was 0.73 (95% CI 0.56 to 0.91) and the intercept was -0.67 (95% CI -0.90 to -0.43). With respect to overall performance, the model's Brier scores for the 90-day and 1-year models were 0.16 and 0.22. These scores were higher than the Brier scores of internal validation of the development study (0.13 and 0.14) models, indicating the models' performance has declined over time. CONCLUSION: The SORG MLA to predict survival after surgical treatment of extremity metastatic disease showed decreased performance on temporal validation. Moreover, in patients undergoing innovative immunotherapy, the possibility of mortality risk was overestimated in varying severity. Clinicians should be aware of this overestimation and discount the prediction of the SORG MLA according to their own experience with this patient population. Generally, these results show that temporal reassessment of these MLA-driven probability calculators is of paramount importance because the predictive performance may decline over time as treatment regimens evolve. The SORG-MLA is available as a freely accessible internet application at https://sorg-apps.shinyapps.io/extremitymetssurvival/ .Level of Evidence Level III, prognostic study.

Worldwide epidemiology of foot and ankle injuries during military training: a systematic review.

INTRODUCTION: Musculoskeletal foot and ankle injuries are commonly experienced by soldiers during military training. We performed a systematic review to assess epidemiological patterns of foot and ankle injuries occurring during military training. METHODS: A review of the literature was performed according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines. The search, done on 14 February 2019, resulted in 1603 reports on PubMed, 565 on Embase and 3 on the Cochrane Library. After reading the remaining full-text articles, we included 91 studies. RESULTS: Among a population of 8 092 281 soldiers from 15 countries, 788 469 (9.74%) foot and ankle injuries were recorded. Among the 49 studies that reported on length of training, there were 36 770/295 040 (18.17%) injuries recorded among women and 248 660/1 501 672 (16.56%) injuries recorded among men over a pooled mean (±SD) training period of 4.51±2.34 months. Ankle injuries were roughly 7 times more common than foot injuries, and acute injuries were roughly 24 times more common than non-acute injuries. Our findings indicated that, during a 3-month training period, soldiers have a 3.14% chance of sustaining a foot and ankle injury. The incidence of foot or ankle injury during military parachutist training was 3.1 injuries per thousand jumps. CONCLUSIONS: Our findings provide an overview of epidemiological patterns of foot and ankle injuries during military training. These data can be used to compare incidence rates of foot and ankle injuries due to acute or non-acute mechanisms during training. Cost-effective methods of preventing acute ankle injuries and non-acute foot injuries are needed to address this problem.

Natural language processing for automated quantification of bone metastases reported in free-text bone scintigraphy reports.

BACKGROUND: The widespread use of electronic patient-generated health data has led to unprecedented opportunities for automated extraction of clinical features from free-text medical notes. However, processing this rich resource of data for clinical and research purposes, depends on labor-intensive and potentially error-prone manual review. The aim of this study was to develop a natural language processing (NLP) algorithm for binary classification (single metastasis versus two or more metastases) in bone scintigraphy reports of patients undergoing surgery for bone metastases. MATERIAL AND METHODS: Bone scintigraphy reports of patients undergoing surgery for bone metastases were labeled each by three independent reviewers using a binary classification (single metastasis versus two or more metastases) to establish a ground truth. A stratified 80:20 split was used to develop and test an extreme-gradient boosting supervised machine learning NLP algorithm. RESULTS: A total of 704 free-text bone scintigraphy reports from 704 patients were included in this study and 617 (88%) had multiple bone metastases. In the independent test set (n = 141) not used for model development, the NLP algorithm achieved an 0.97 AUC-ROC (95% confidence interval [CI], 0.92-0.99) for classification of multiple bone metastases and an 0.99 AUC-PRC (95% CI, 0.99-0.99). At a threshold of 0.90, NLP algorithm correctly identified multiple bone metastases in 117 of the 124 who had multiple bone metastases in the testing cohort (sensitivity 0.94) and yielded 3 false positives (specificity 0.82). At the same threshold, the NLP algorithm had a positive predictive value of 0.97 and F1-score of 0.96. CONCLUSIONS: NLP has the potential to automate clinical data extraction from free text radiology notes in orthopedics, thereby optimizing the speed, accuracy, and consistency of clinical chart review. Pending external validation, the NLP algorithm developed in this study may be implemented as a means to aid researchers in tackling large amounts of data.