Does the Presence of Missing Data Affect the Performance of the SORG Machine-learning Algorithm for Patients With Spinal Metastasis? Development of an Internet Application Algorithm.

BACKGROUND: The Skeletal Oncology Research Group machine-learning algorithm (SORG-MLA) was developed to predict the survival of patients with spinal metastasis. The algorithm was successfully tested in five international institutions using 1101 patients from different continents. The incorporation of 18 prognostic factors strengthens its predictive ability but limits its clinical utility because some prognostic factors might not be clinically available when a clinician wishes to make a prediction. QUESTIONS/PURPOSES: We performed this study to (1) evaluate the SORG-MLA's performance with data and (2) develop an internet-based application to impute the missing data. METHODS: A total of 2768 patients were included in this study. The data of 617 patients who were treated surgically were intentionally erased, and the data of the other 2151 patients who were treated with radiotherapy and medical treatment were used to impute the artificially missing data. Compared with those who were treated nonsurgically, patients undergoing surgery were younger (median 59 years [IQR 51 to 67 years] versus median 62 years [IQR 53 to 71 years]) and had a higher proportion of patients with at least three spinal metastatic levels (77% [474 of 617] versus 72% [1547 of 2151]), more neurologic deficit (normal American Spinal Injury Association [E] 68% [301 of 443] versus 79% [1227 of 1561]), higher BMI (23 kg/m2 [IQR 20 to 25 kg/m2] versus 22 kg/m2 [IQR 20 to 25 kg/m2]), higher platelet count (240 × 103/µL [IQR 173 to 327 × 103/µL] versus 227 × 103/µL [IQR 165 to 302 × 103/µL], higher lymphocyte count (15 × 103/µL [IQR 9 to 21× 103/µL] versus 14 × 103/µL [IQR 8 to 21 × 103/µL]), lower serum creatinine level (0.7 mg/dL [IQR 0.6 to 0.9 mg/dL] versus 0.8 mg/dL [IQR 0.6 to 1.0 mg/dL]), less previous systemic therapy (19% [115 of 617] versus 24% [526 of 2151]), fewer Charlson comorbidities other than cancer (28% [170 of 617] versus 36% [770 of 2151]), and longer median survival. The two patient groups did not differ in other regards. These findings aligned with our institutional philosophy of selecting patients for surgical intervention based on their level of favorable prognostic factors such as BMI or lymphocyte counts and lower levels of unfavorable prognostic factors such as white blood cell counts or serum creatinine level, as well as the degree of spinal instability and severity of neurologic deficits. This approach aims to identify patients with better survival outcomes and prioritize their surgical intervention accordingly. Seven factors (serum albumin and alkaline phosphatase levels, international normalized ratio, lymphocyte and neutrophil counts, and the presence of visceral or brain metastases) were considered possible missing items based on five previous validation studies and clinical experience. Artificially missing data were imputed using the missForest imputation technique, which was previously applied and successfully tested to fit the SORG-MLA in validation studies. Discrimination, calibration, overall performance, and decision curve analysis were applied to evaluate the SORG-MLA's performance. The discrimination ability was measured with an area under the receiver operating characteristic curve. It ranges from 0.5 to 1.0, with 0.5 indicating the worst discrimination and 1.0 indicating perfect discrimination. An area under the curve of 0.7 is considered clinically acceptable discrimination. Calibration refers to the agreement between the predicted outcomes and actual outcomes. An ideal calibration model will yield predicted survival rates that are congruent with the observed survival rates. The Brier score measures the squared difference between the actual outcome and predicted probability, which captures calibration and discrimination ability simultaneously. A Brier score of 0 indicates perfect prediction, whereas a Brier score of 1 indicates the poorest prediction. A decision curve analysis was performed for the 6-week, 90-day, and 1-year prediction models to evaluate their net benefit across different threshold probabilities. Using the results from our analysis, we developed an internet-based application that facilitates real-time data imputation for clinical decision-making at the point of care. This tool allows healthcare professionals to efficiently and effectively address missing data, ensuring that patient care remains optimal at all times. RESULTS: Generally, the SORG-MLA demonstrated good discriminatory ability, with areas under the curve greater than 0.7 in most cases, and good overall performance, with up to 25% improvement in Brier scores in the presence of one to three missing items. The only exceptions were albumin level and lymphocyte count, because the SORG-MLA's performance was reduced when these two items were missing, indicating that the SORG-MLA might be unreliable without these values. The model tended to underestimate the patient survival rate. As the number of missing items increased, the model's discriminatory ability was progressively impaired, and a marked underestimation of patient survival rates was observed. Specifically, when three items were missing, the number of actual survivors was up to 1.3 times greater than the number of expected survivors, while only 10% discrepancy was observed when only one item was missing. When either two or three items were omitted, the decision curves exhibited substantial overlap, indicating a lack of consistent disparities in performance. This finding suggests that the SORG-MLA consistently generates accurate predictions, regardless of the two or three items that are omitted. We developed an internet application (https://sorg-spine-mets-missing-data-imputation.azurewebsites.net/) that allows the use of SORG-MLA with up to three missing items. CONCLUSION: The SORG-MLA generally performed well in the presence of one to three missing items, except for serum albumin level and lymphocyte count (which are essential for adequate predictions, even using our modified version of the SORG-MLA). We recommend that future studies should develop prediction models that allow for their use when there are missing data, or provide a means to impute those missing data, because some data are not available at the time a clinical decision must be made. CLINICAL RELEVANCE: The results suggested the algorithm could be helpful when a radiologic evaluation owing to a lengthy waiting period cannot be performed in time, especially in situations when an early operation could be beneficial. It could help orthopaedic surgeons to decide whether to intervene palliatively or extensively, even when the surgical indication is clear.

Ten existing osteoporosis prediction tools for the successful application of National Health Insurance-reimbursed anti-osteoporosis medications in long-term care residents in Taiwan.

BACKGROUND/PURPOSE: Osteoporotic fracture introduce enormous societal and economic burden, especially for long-term care residents (LTCRs). Although osteoporosis prevention for LTCRs is urgently needed, obstacles such as frail status and inconvenient hospital visits hurdled them from necessary examinations and diagnoses. We aimed to test 10 existing osteoporosis screening tools (OSTs), which can be easily used in institutions and serve as a prediction, for accurately determining the outcome of a Taiwan's National Health Insurance (NHI)-reimbursed anti-osteoporosis medications (AOMs) application for LTCRs. METHODS: This prospective analysis recruited 444 patients from LTC institutions between October 2018 and November 2019. Predictions of whether the NHI-reimbursed AOMs criteria was met were tested for 10 OSTs. The results of OSTs categorized into self-reported or validated based on previous fracture history were self-reported by LTCRs or validated by imaging data and medical records, respectively. The receiver operating characteristic curve and the optimal cut-off points for LTCRs based on Youden's index were explored. RESULTS: Overall, the validated OSTs had a higher positive predictive value (PPV) and negative predictive value (NPV) summation than the corresponding reported OSTs. The validated FRAX-Major was the best OST (PPV = 63.6%, NPV = 82.4% for the male group and, PPV = 78.8%, NPV = 90.0% for the female group). After applying the optimum cut-off derived from Youden's index, the validated FRAX-Major (PPV = 75.4%, NPV = 92.0%)) remained performed best for men. In female population, validated FRAX-Major (PPV = 87.2%, NPV = 84.1%) and validated osteoporosis prescreening risk assessment (OPERA; PPV = 96.1%, NPV = 79.7%)) both provided good prediction results. CONCLUSION: FRAX-Major and OPERA have better prediction ability for LTCRs to acquire NHI-reimbursed AOMs. The validated fracture history and adjusted cut-off points could prominently increase the PPV during prediction.

Prognostic significance of lab data and performance comparison by validating survival prediction models for patients with spinal metastases after radiotherapy.

BACKGROUND AND PURPOSE: Well-performing survival prediction models (SPMs) help patients and healthcare professionals to choose treatment aligning with prognosis. This retrospective study aims to investigate the prognostic impacts of laboratory data and to compare the performances of Metastases location, Elderly, Tumor primary, Sex, Sickness/comorbidity, and Site of radiotherapy (METSSS) model, New England Spinal Metastasis Score (NESMS), and Skeletal Oncology Research Group machine learning algorithm (SORG-MLA) for spinal metastases (SM). MATERIALS AND METHODS: From 2010 to 2018, patients who received radiotherapy (RT) for SM at a tertiary center were enrolled and the data were retrospectively collected. Multivariate logistic and Cox-proportional-hazard regression analyses were used to assess the association between laboratory values and survival. The area under receiver-operating characteristics curve (AUROC), calibration analysis, Brier score, and decision curve analysis were used to evaluate the performance of SPMs. RESULTS: A total of 2786 patients were included for analysis. The 90-day and 1-year survival rates after RT were 70.4% and 35.7%, respectively. Higher albumin, hemoglobin, or lymphocyte count were associated with better survival, while higher alkaline phosphatase, white blood cell count, neutrophil count, neutrophil-to-lymphocyte ratio, platelet-to-lymphocyte ratio, or international normalized ratio were associated with poor prognosis. SORG-MLA has the best discrimination (AUROC 90-day, 0.78; 1-year 0.76), best calibrations, and the lowest Brier score (90-day 0.16; 1-year 0.18). The decision curve of SORG-MLA is above the other two competing models with threshold probabilities from 0.1 to 0.8. CONCLUSION: Laboratory data are of prognostic significance in survival prediction after RT for SM. Machine learning-based model SORG-MLA outperforms statistical regression-based model METSSS model and NESMS in survival predictions.

Can a Bayesian belief network for survival prediction in patients with extremity metastases (PATHFx) be externally validated in an Asian cohort of 356 surgically treated patients?

BACKGROUND AND PURPOSE: Predicted survival may influence the treatment decision for patients with skeletal extremity metastasis, and PATHFx was designed to predict the likelihood of a patient dying in the next 24 months. However, the performance of prediction models could have ethnogeographical variations. We asked if PATHFx generalized well to our Taiwanese cohort consisting of 356 surgically treated patients with extremity metastasis. PATIENTS AND METHODS: We included 356 patients who underwent surgery for skeletal extremity metastasis in a tertiary center in Taiwan between 2014 and 2019 to validate PATHFx's survival predictions at 6 different time points. Model performance was assessed by concordance index (c-index), calibration analysis, decision curve analysis (DCA), Brier score, and model consistency (MC). RESULTS: The c-indexes for the 1-, 3-, 6-, 12-, 18-, and 24-month survival estimations were 0.71, 0.66, 0.65, 0.69, 0.68, and 0.67, respectively. The calibration analysis demonstrated positive calibration intercepts for survival predictions at all 6 timepoints, indicating PATHFx tended to underestimate the actual survival. The Brier scores for the 6 models were all less than their respective null model's. DCA demonstrated that only the 6-, 12-, 18-, and 24-month predictions appeared useful for clinical decision-making across a wide range of threshold probabilities. The MC was < 0.9 when the 6- and 12-month models were compared with the 12-month and 18-month models, respectively. INTERPRETATION: In this Asian cohort, PATHFx's performance was not as encouraging as those of prior validation studies. Clinicians should be cognizant of the potential decline in validity of any tools designed using data outside their particular patient population. Developers of survival prediction tools such as PATHFx might refine their algorithms using data from diverse, contemporary patients that is more reflective of the world's population.

A machine learning algorithm for predicting prolonged postoperative opioid prescription after lumbar disc herniation surgery. An external validation study using 1,316 patients from a Taiwanese cohort.

BACKGROUND CONTEXT: Preoperative prediction of prolonged postoperative opioid prescription helps identify patients for increased surveillance after surgery. The SORG machine learning model has been developed and successfully tested using 5,413 patients from the United States (US) to predict the risk of prolonged opioid prescription after surgery for lumbar disc herniation. However, external validation is an often-overlooked element in the process of incorporating prediction models in current clinical practice. This cannot be stressed enough in prediction models where medicolegal and cultural differences may play a major role. PURPOSE: The authors aimed to investigate the generalizability of the US citizens prediction model SORG to a Taiwanese patient cohort. STUDY DESIGN: Retrospective study at a large academic medical center in Taiwan. PATIENT SAMPLE: Of 1,316 patients who were 20 years or older undergoing initial operative management for lumbar disc herniation between 2010 and 2018. OUTCOME MEASURES: The primary outcome of interest was prolonged opioid prescription defined as continuing opioid prescription to at least 90 to 180 days after the first surgery for lumbar disc herniation at our institution. METHODS: Baseline characteristics were compared between the external validation cohort and the original developmental cohorts. Discrimination (area under the receiver operating characteristic curve and the area under the precision-recall curve), calibration, overall performance (Brier score), and decision curve analysis were used to assess the performance of the SORG ML algorithm in the validation cohort. This study had no funding source or conflict of interests. RESULTS: Overall, 1,316 patients were identified with sustained postoperative opioid prescription in 41 (3.1%) patients. The validation cohort differed from the development cohort on several variables including 93% of Taiwanese patients receiving NSAIDS preoperatively compared with 22% of US citizens patients, while 30% of Taiwanese patients received opioids versus 25% in the US. Despite these differences, the SORG prediction model retained good discrimination (area under the receiver operating characteristic curve of 0.76 and the area under the precision-recall curve of 0.33) and good overall performance (Brier score of 0.028 compared with null model Brier score of 0.030) while somewhat overestimating the chance of prolonged opioid use (calibration slope of 1.07 and calibration intercept of -0.87). Decision-curve analysis showed the SORG model was suitable for clinical use. CONCLUSIONS: Despite differences at baseline and a very strict opioid policy, the SORG algorithm for prolonged opioid use after surgery for lumbar disc herniation has good discriminative abilities and good overall performance in a Han Chinese patient group in Taiwan. This freely available digital application can be used to identify high-risk patients and tailor prevention policies for these patients that may mitigate the long-term adverse consequence of opioid dependence: https://sorg-apps.shinyapps.io/lumbardiscopioid/.

Decreased psoas muscle area is a prognosticator for 90-day and 1-year survival in patients undergoing surgical treatment for spinal metastasis.

BACKGROUND AND AIMS: Survival estimation for patients with spinal metastasis is crucial to treatment decisions. Psoas muscle area (PMA), a surrogate for total muscle mass, has been proposed as a useful survival prognosticator. However, few studies have validated the predictive value of decreased PMA in an Asian cohort or its predictive value after controlling for existing preoperative scoring systems (PSSs). In this study, we aim to answer: (1) Is PMA associated with survival in Han Chinese patients with spinal metastasis? (2) Is PMA a good prognosticator according to concordance index (c-index) and decision curve analysis (DCA) after controlling for six existing and commonly used PSSs? METHODS: This study included 180 adult (≥18 years old) Taiwanese patients with a mean age of 58.3 years (range: 22-85) undergoing surgical treatment for spinal metastasis. A patient's PMA was classified into decreased, medium, and large if it fell into the lower (0-33%), middle (33-67%), and upper (67-100%) 1/3 in the study cohort, respectively. We used logistic and cox proportional-hazard regressions to assess whether PMA was associated with 90-day, 1-year, and overall survival. The model performance before and after addition of PMA to six commonly used PSSs, including Tomita score, original Tokuhashi score, revised Tokuhashi score, modified Bauer score, New England Spinal Metastasis Score, and Skeletal Oncology Research Group machine learning algorithms (SORG-MLAs), was compared by c-index and DCA to determine if PMA was a useful survival prognosticator. RESULTS: Patients with a larger PMA is associated with better 90-day, but not 1-year, survival. The model performance of 90-day survival prediction improved after PMA was incorporated into all PSSs except SORG-MLAs. PMA barely improved the discriminatory ability (c-index, 0.74; 95% confidence interval [CI], 0.67-0.82 vs. c-index, 0.74; 95% CI, 0.66-0.81) and provided little gain of clinical net benefit on DCA for SORG-MLAs' 90-day survival prediction. CONCLUSIONS: PMA is a prognosticator for 90-day survival and improves the discriminatory ability of earlier-proposed PSSs in our Asian cohort. However, incorporating PMA into more modern PSSs such as SORG-MLAs did not significantly improve its prediction performance.

International external validation of the SORG machine learning algorithms for predicting 90-day and one-year survival of patients with spine metastases using a Taiwanese cohort.

BACKGROUND CONTEXT: Accurately predicting the survival of patients with spinal metastases is important for guiding surgical intervention. The SORG machine-learning (ML) algorithm for the 90-day and one-year mortality of patients with metastatic cancer to the spine has been multiply validated, with a high degree of accuracy in both internal and external validation studies. However, prior external validations were conducted using patient groups located on the east coast of the United States, representing a generally homogeneous population. The aim of this study was to externally validate the SORG algorithms with a Taiwanese population. STUDY DESIGN/SETTING: Retrospective study at a single tertiary care center in Taiwan PATIENT SAMPLE: Four hundred and twenty-seven patients who underwent surgery for metastatic spine disease from November 1, 2010 to December 31, 2018 OUTCOME MEASURES: 90-day and one-year mortality METHODS: The baseline characteristics of our validation cohort were compared with those of the previously published developmental and external validation cohorts. Discrimination (c-statistic and receiver operating curve), calibration (calibration plot, intercept, and slope), overall performance (Brier score), and decision curve analysis were used to assess the performance of the SORG ML algorithms in this cohort. RESULTS: Ninety-day and one-year mortality rates were 110 of 427 (26%) and 256 of 427 (60%), respectively. The external validation cohort and the developmental cohort differed in body mass index (BMI), preoperative performance status, American Spinal Injury Association impairment scale, primary tumor histology and in several laboratory measurements. The SORG ML algorithm for 90-day and 1-year mortality demonstrated a high level of discriminative ability (c-statistics of 0.73 [95% confidence interval [CI], 0.67-0.78] and 0.74 [95% CI, 0.69-0.79]), overall performance, and had a positive net benefit throughout the range of threshold probabilities in decision curve analysis. The algorithm for 1-year mortality had a calibration intercept of 0.08, representing a good calibration. However, the 90-day mortality algorithm underestimated mortality for the lowest predicted probabilities, with an overall intercept of 0.81. CONCLUSIONS: The SORG algorithms for predicting 90-day and 1-year mortality in patients with spinal metastatic disease generally performed well on international external validation in a predominately Taiwanese population. However, 90-day mortality was underestimated in this group. Whether this inconsistency was due to different primary tumor characteristics, body mass index, selection bias or other factors remains unclear, and may be better understood with further validative works that utilize international and/or diverse populations.