Survival in Patients With Spinal Metastatic Disease Treated Nonoperatively With Radiotherapy: Are the SORG-ML Algorithms Relevant?

SUMMARY OF BACKGROUND DATA: The SORG-ML algorithms for survival in spinal metastatic disease were developed in patients who underwent surgery and were externally validated for patients managed operatively. OBJECTIVE: To externally validate the SORG-ML algorithms for survival in spinal metastatic disease in patients managed nonoperatively with radiation. STUDY DESIGN: Retrospective cohort. METHODS: The performance of the SORG-ML algorithms was assessed by discrimination [receiver operating curves and area under the receiver operating curve (AUC)], calibration (calibration plots), decision curve analysis, and overall performance (Brier score). The primary outcomes were 90-day and 1-year mortality. RESULTS: Overall, 2074 adult patients underwent radiation for spinal metastatic disease and 29% (n=521) and 59% (n=917) had 90-day and 1-year mortality, respectively. On complete case analysis (n=415), the AUC was 0.76 (95% CI: 0.71-0.80) and 0.78 (95% CI: 0.73-0.83) for 90-day and 1-year mortality with fair calibration and positive net benefit confirmed by the decision curve analysis. With multiple imputation (n=2074), the AUC was 0.85 (95% CI: 0.83-0.87) and 0.87 (95% CI: 0.85-0.89) for 90-day and 1-year mortality with fair calibration and positive net benefit confirmed by the decision curve analysis. CONCLUSION: The SORG-ML algorithms for survival in spinal metastatic disease generalize well to patients managed nonoperatively with radiation.

Does the Presence of Missing Data Affect the Performance of the SORG Machine-learning Algorithm for Patients With Spinal Metastasis? Development of an Internet Application Algorithm.

BACKGROUND: The Skeletal Oncology Research Group machine-learning algorithm (SORG-MLA) was developed to predict the survival of patients with spinal metastasis. The algorithm was successfully tested in five international institutions using 1101 patients from different continents. The incorporation of 18 prognostic factors strengthens its predictive ability but limits its clinical utility because some prognostic factors might not be clinically available when a clinician wishes to make a prediction. QUESTIONS/PURPOSES: We performed this study to (1) evaluate the SORG-MLA's performance with data and (2) develop an internet-based application to impute the missing data. METHODS: A total of 2768 patients were included in this study. The data of 617 patients who were treated surgically were intentionally erased, and the data of the other 2151 patients who were treated with radiotherapy and medical treatment were used to impute the artificially missing data. Compared with those who were treated nonsurgically, patients undergoing surgery were younger (median 59 years [IQR 51 to 67 years] versus median 62 years [IQR 53 to 71 years]) and had a higher proportion of patients with at least three spinal metastatic levels (77% [474 of 617] versus 72% [1547 of 2151]), more neurologic deficit (normal American Spinal Injury Association [E] 68% [301 of 443] versus 79% [1227 of 1561]), higher BMI (23 kg/m2 [IQR 20 to 25 kg/m2] versus 22 kg/m2 [IQR 20 to 25 kg/m2]), higher platelet count (240 × 103/µL [IQR 173 to 327 × 103/µL] versus 227 × 103/µL [IQR 165 to 302 × 103/µL], higher lymphocyte count (15 × 103/µL [IQR 9 to 21× 103/µL] versus 14 × 103/µL [IQR 8 to 21 × 103/µL]), lower serum creatinine level (0.7 mg/dL [IQR 0.6 to 0.9 mg/dL] versus 0.8 mg/dL [IQR 0.6 to 1.0 mg/dL]), less previous systemic therapy (19% [115 of 617] versus 24% [526 of 2151]), fewer Charlson comorbidities other than cancer (28% [170 of 617] versus 36% [770 of 2151]), and longer median survival. The two patient groups did not differ in other regards. These findings aligned with our institutional philosophy of selecting patients for surgical intervention based on their level of favorable prognostic factors such as BMI or lymphocyte counts and lower levels of unfavorable prognostic factors such as white blood cell counts or serum creatinine level, as well as the degree of spinal instability and severity of neurologic deficits. This approach aims to identify patients with better survival outcomes and prioritize their surgical intervention accordingly. Seven factors (serum albumin and alkaline phosphatase levels, international normalized ratio, lymphocyte and neutrophil counts, and the presence of visceral or brain metastases) were considered possible missing items based on five previous validation studies and clinical experience. Artificially missing data were imputed using the missForest imputation technique, which was previously applied and successfully tested to fit the SORG-MLA in validation studies. Discrimination, calibration, overall performance, and decision curve analysis were applied to evaluate the SORG-MLA's performance. The discrimination ability was measured with an area under the receiver operating characteristic curve. It ranges from 0.5 to 1.0, with 0.5 indicating the worst discrimination and 1.0 indicating perfect discrimination. An area under the curve of 0.7 is considered clinically acceptable discrimination. Calibration refers to the agreement between the predicted outcomes and actual outcomes. An ideal calibration model will yield predicted survival rates that are congruent with the observed survival rates. The Brier score measures the squared difference between the actual outcome and predicted probability, which captures calibration and discrimination ability simultaneously. A Brier score of 0 indicates perfect prediction, whereas a Brier score of 1 indicates the poorest prediction. A decision curve analysis was performed for the 6-week, 90-day, and 1-year prediction models to evaluate their net benefit across different threshold probabilities. Using the results from our analysis, we developed an internet-based application that facilitates real-time data imputation for clinical decision-making at the point of care. This tool allows healthcare professionals to efficiently and effectively address missing data, ensuring that patient care remains optimal at all times. RESULTS: Generally, the SORG-MLA demonstrated good discriminatory ability, with areas under the curve greater than 0.7 in most cases, and good overall performance, with up to 25% improvement in Brier scores in the presence of one to three missing items. The only exceptions were albumin level and lymphocyte count, because the SORG-MLA's performance was reduced when these two items were missing, indicating that the SORG-MLA might be unreliable without these values. The model tended to underestimate the patient survival rate. As the number of missing items increased, the model's discriminatory ability was progressively impaired, and a marked underestimation of patient survival rates was observed. Specifically, when three items were missing, the number of actual survivors was up to 1.3 times greater than the number of expected survivors, while only 10% discrepancy was observed when only one item was missing. When either two or three items were omitted, the decision curves exhibited substantial overlap, indicating a lack of consistent disparities in performance. This finding suggests that the SORG-MLA consistently generates accurate predictions, regardless of the two or three items that are omitted. We developed an internet application (https://sorg-spine-mets-missing-data-imputation.azurewebsites.net/) that allows the use of SORG-MLA with up to three missing items. CONCLUSION: The SORG-MLA generally performed well in the presence of one to three missing items, except for serum albumin level and lymphocyte count (which are essential for adequate predictions, even using our modified version of the SORG-MLA). We recommend that future studies should develop prediction models that allow for their use when there are missing data, or provide a means to impute those missing data, because some data are not available at the time a clinical decision must be made. CLINICAL RELEVANCE: The results suggested the algorithm could be helpful when a radiologic evaluation owing to a lengthy waiting period cannot be performed in time, especially in situations when an early operation could be beneficial. It could help orthopaedic surgeons to decide whether to intervene palliatively or extensively, even when the surgical indication is clear.

Comparison of eight modern preoperative scoring systems for survival prediction in patients with extremity metastasis.

BACKGROUND: Survival is an important factor to consider when clinicians make treatment decisions for patients with skeletal metastasis. Several preoperative scoring systems (PSSs) have been developed to aid in survival prediction. Although we previously validated the Skeletal Oncology Research Group Machine-learning Algorithm (SORG-MLA) in Taiwanese patients of Han Chinese descent, the performance of other existing PSSs remains largely unknown outside their respective development cohorts. We aim to determine which PSS performs best in this unique population and provide a direct comparison between these models. METHODS: We retrospectively included 356 patients undergoing surgical treatment for extremity metastasis at a tertiary center in Taiwan to validate and compare eight PSSs. Discrimination (c-index), decision curve (DCA), calibration (ratio of observed:expected survivors), and overall performance (Brier score) analyses were conducted to evaluate these models' performance in our cohort. RESULTS: The discriminatory ability of all PSSs declined in our Taiwanese cohort compared with their Western validations. SORG-MLA is the only PSS that still demonstrated excellent discrimination (c-indexes>0.8) in our patients. SORG-MLA also brought the most net benefit across a wide range of risk probabilities on DCA with its 3-month and 12-month survival predictions. CONCLUSIONS: Clinicians should consider potential ethnogeographic variations of a PSS's performance when applying it onto their specific patient populations. Further international validation studies are needed to ensure that existing PSSs are generalizable and can be integrated into the shared treatment decision-making process. As cancer treatment keeps advancing, researchers developing a new prediction model or refining an existing one could potentially improve their algorithm's performance by using data gathered from more recent patients that are reflective of the current state of cancer care.

Clockwise torque results in higher reoperation rates in left-sided femur fractures.

PURPOSE: Effects of clockwise torque rotation onto proximal femoral fracture fixation have been subject of ongoing debate: fixated right-sided trochanteric fractures seem more rotationally stable than left-sided fractures in the biomechanical setting, but this theoretical advantage has not been demonstrated in the clinical setting to date. The purpose of this study was to identify a difference in early reoperation rate between patients undergoing surgery for left- versus right-sided proximal femur fractures using cephalomedullary nailing (CMN). MATERIALS AND METHODS: The American College of Surgeons National Surgical Quality Improvement Program was queried from 2016-2019 to identify patients aged 50 years and older undergoing CMN for a proximal femoral fracture. The primary outcome was any unplanned reoperation within 30 days following surgery. The difference was calculated using a Chi-square test, and observed power calculated using post-hoc power analysis. RESULTS: In total, of 20,122 patients undergoing CMN for proximal femoral fracture management, 1.8% (n=371) had to undergo an unplanned reoperation within 30 days after surgery. Overall, 208 (2.0%) were left-sided and 163 (1.7%) right-sided fractures (p=0.052, risk ratio [RR] 1.22, 95% confidence interval [CI] 1.00-1.50), odds ratio [OR] 1.23 (95%CI 1.00-1.51), power 49.2% (α=0.05). CONCLUSION: This study shows a higher risk of reoperation for left-sided compared to right-sided proximal femur fractures after CMN in a large sample size. Although results may be underpowered and statistically insignificant, this finding might substantiate the hypothesis that clockwise rotation during implant insertion and (postoperative) weightbearing may lead to higher reoperation rates. LEVEL OF EVIDENCE: Therapeutic level II.

Does the SORG Machine-learning Algorithm for Extremity Metastases Generalize to a Contemporary Cohort of Patients? Temporal Validation From 2016 to 2020.

BACKGROUND: The ability to predict survival accurately in patients with osseous metastatic disease of the extremities is vital for patient counseling and guiding surgical intervention. We, the Skeletal Oncology Research Group (SORG), previously developed a machine-learning algorithm (MLA) based on data from 1999 to 2016 to predict 90-day and 1-year survival of surgically treated patients with extremity bone metastasis. As treatment regimens for oncology patients continue to evolve, this SORG MLA-driven probability calculator requires temporal reassessment of its accuracy. QUESTION/PURPOSE: Does the SORG-MLA accurately predict 90-day and 1-year survival in patients who receive surgical treatment for a metastatic long-bone lesion in a more recent cohort of patients treated between 2016 and 2020? METHODS: Between 2017 and 2021, we identified 674 patients 18 years and older through the ICD codes for secondary malignant neoplasm of bone and bone marrow and CPT codes for completed pathologic fractures or prophylactic treatment of an impending fracture. We excluded 40% (268 of 674) of patients, including 18% (118) who did not receive surgery; 11% (72) who had metastases in places other than the long bones of the extremities; 3% (23) who received treatment other than intramedullary nailing, endoprosthetic reconstruction, or dynamic hip screw; 3% (23) who underwent revision surgery, 3% (17) in whom there was no tumor, and 2% (15) who were lost to follow-up within 1 year. Temporal validation was performed using data on 406 patients treated surgically for bony metastatic disease of the extremities from 2016 to 2020 at the same two institutions where the MLA was developed. Variables used to predict survival in the SORG algorithm included perioperative laboratory values, tumor characteristics, and general demographics. To assess the models' discrimination, we computed the c-statistic, commonly referred to as the area under the receiver operating characteristic (AUC) curve for binary classification. This value ranged from 0.5 (representing chance-level performance) to 1.0 (indicating excellent discrimination) Generally, an AUC of 0.75 is considered high enough for use in clinical practice. To evaluate the agreement between predicted and observed outcomes, a calibration plot was used, and the calibration slope and intercept were calculated. Perfect calibration would result in a slope of 1 and intercept of 0. For overall performance, the Brier score and null-model Brier score were determined. The Brier score can range from 0 (representing perfect prediction) to 1 (indicating the poorest prediction). Proper interpretation of the Brier score necessitates a comparison with the null-model Brier score, which represents the score for an algorithm that predicts a probability equal to the population prevalence of the outcome for each patient. Finally, a decision curve analysis was conducted to compare the potential net benefit of the algorithm with other decision-support methods, such as treating all or none of the patients. Overall, 90-day and 1-year mortality were lower in the temporal validation cohort than in the development cohort (90 day: 23% versus 28%; p < 0.001, and 1 year: 51% versus 59%; p<0.001). RESULTS: Overall survival of the patients in the validation cohort improved from 28% mortality at the 90-day timepoint in the cohort on which the model was trained to 23%, and 59% mortality at the 1-year timepoint to 51%. The AUC was 0.78 (95% CI 0.72 to 0.82) for 90-day survival and 0.75 (95% CI 0.70 to 0.79) for 1-year survival, indicating the model could distinguish the two outcomes reasonably. For the 90-day model, the calibration slope was 0.71 (95% CI 0.53 to 0.89), and the intercept was -0.66 (95% CI -0.94 to -0.39), suggesting the predicted risks were overly extreme, and that in general, the risk of the observed outcome was overestimated. For the 1-year model, the calibration slope was 0.73 (95% CI 0.56 to 0.91) and the intercept was -0.67 (95% CI -0.90 to -0.43). With respect to overall performance, the model's Brier scores for the 90-day and 1-year models were 0.16 and 0.22. These scores were higher than the Brier scores of internal validation of the development study (0.13 and 0.14) models, indicating the models' performance has declined over time. CONCLUSION: The SORG MLA to predict survival after surgical treatment of extremity metastatic disease showed decreased performance on temporal validation. Moreover, in patients undergoing innovative immunotherapy, the possibility of mortality risk was overestimated in varying severity. Clinicians should be aware of this overestimation and discount the prediction of the SORG MLA according to their own experience with this patient population. Generally, these results show that temporal reassessment of these MLA-driven probability calculators is of paramount importance because the predictive performance may decline over time as treatment regimens evolve. The SORG-MLA is available as a freely accessible internet application at https://sorg-apps.shinyapps.io/extremitymetssurvival/ .Level of Evidence Level III, prognostic study.

Development of a machine learning algorithm to identify surgical candidates for hip and knee arthroplasty without in-person evaluation.

INTRODUCTION: Arthroplasty care delivery is facing a growing supply-demand mismatch. To meet future demand for joint arthroplasty, systems will need to identify potential surgical candidates prior to evaluation by orthopaedic surgeons. MATERIALS AND METHODS: Retrospective review was conducted at two academic medical centers and three community hospitals from March 1 to July 31, 2020 to identify new patient telemedicine encounters (without prior in-person evaluation) for consideration of hip or knee arthroplasty. The primary outcome was surgical indication for joint replacement. Five machine learning algorithms were developed to predict likelihood of surgical indication and assessed by discrimination, calibration, overall performance, and decision curve analysis. RESULTS: Overall, 158 patients underwent new patient telemedicine evaluation for consideration of THA, TKA, or UKA and 65.2% (n = 103) were indicated for operative intervention prior to in-person evaluation. The median age was 65 (interquartile range 59-70) and 60.8% were women. Variables found to be associated with operative intervention were radiographic degree of arthritis, prior trial of intra-articular injection, trial of physical therapy, opioid use, and tobacco use. In the independent testing set (n = 46) not used for algorithm development, the stochastic gradient boosting algorithm achieved the best performance with AUC 0.83, calibration intercept 0.13, calibration slope 1.03, Brier score 0.15 relative to a null model Brier score of 0.23, and higher net benefit than the default alternatives on decision curve analysis. CONCLUSION: We developed a machine learning algorithm to identify potential surgical candidates for joint arthroplasty in the setting of osteoarthritis without an in-person evaluation or physical examination. If externally validated, this algorithm could be deployed by various stakeholders, including patients, providers, and health systems, to direct appropriate next steps in patients with osteoarthritis and improve efficiency in identifying surgical candidates. LEVEL OF EVIDENCE: III.

Intercontinental validation of a clinical prediction model for predicting 90-day and 2-year mortality in an Israeli cohort of 2033 patients with a femoral neck fracture aged 65 or above.

PURPOSE: Mortality prediction in elderly femoral neck fracture patients is valuable in treatment decision-making. A previously developed and internally validated clinical prediction model shows promise in identifying patients at risk of 90-day and 2-year mortality. Validation in an independent cohort is required to assess the generalizability; especially in geographically distinct regions. Therefore we questioned, is the SORG Orthopaedic Research Group (SORG) femoral neck fracture mortality algorithm externally valid in an Israeli cohort to predict 90-day and 2-year mortality? METHODS: We previously developed a prediction model in 2022 for estimating the risk of mortality in femoral neck fracture patients using a multicenter institutional cohort of 2,478 patients from the USA. The model included the following input variables that are available on clinical admission: age, male gender, creatinine level, absolute neutrophil, hemoglobin level, international normalized ratio (INR), congestive heart failure (CHF), displaced fracture, hemiplegia, chronic obstructive pulmonary disease (COPD), history of cerebrovascular accident (CVA) and beta-blocker use. To assess the generalizability, we used an intercontinental institutional cohort from the Sheba Medical Center in Israel (level I trauma center), queried between June 2008 and February 2022. Generalizability of the model was assessed using discrimination, calibration, Brier score, and decision curve analysis. RESULTS: The validation cohort included 2,033 patients, aged 65 years or above, that underwent femoral neck fracture surgery. Most patients were female 64.8% (n = 1317), the median age was 81 years (interquartile range = 75-86), and 80.4% (n = 1635) patients sustained a displaced fracture (Garden III/IV). The 90-day mortality was 9.4% (n = 190) and 2-year mortality was 30.0% (n = 610). Despite numerous baseline differences, the model performed acceptably to the validation cohort on discrimination (c-statistic 0.67 for 90-day, 0.67 for 2-year), calibration, Brier score, and decision curve analysis. CONCLUSIONS: The previously developed SORG femoral neck fracture mortality algorithm demonstrated good performance in an independent intercontinental population. Current iteration should not be relied on for patient care, though suggesting potential utility in assessing patients at low risk for 90-day or 2-year mortality. Further studies should evaluate this tool in a prospective setting and evaluate its feasibility and efficacy in clinical practice. The algorithm can be freely accessed: https://sorg-apps.shinyapps.io/hipfracturemortality/ . LEVEL OF EVIDENCE: Level III, Prognostic study.

External validation of a predictive algorithm for in-hospital and 90-day mortality after spinal epidural abscess.

BACKGROUND CONTEXT: Mortality in patients with spinal epidural abscess (SEA) remains high. Accurate prediction of patient-specific prognosis in SEA can improve patient counseling as well as guide management decisions. There are no externally validated studies predicting short-term mortality in patients with SEA. PURPOSE: The purpose of this study was to externally validate the Skeletal Oncology Research Group (SORG) stochastic gradient boosting algorithm for prediction of in-hospital and 90-day postdischarge mortality in SEA. STUDY DESIGN/SETTING: Retrospective, case-control study at a tertiary care academic medical center from 2003 to 2021. PATIENT SAMPLE: Adult patients admitted for radiologically confirmed diagnosis of SEA who did not initiate treatment at an outside institution. OUTCOME MEASURES: In-hospital and 90-day postdischarge mortality. METHODS: We tested the SORG stochastic gradient boosting algorithm on an independent validation cohort. We assessed its performance with discrimination, calibration, decision curve analysis, and overall performance. RESULTS: A total of 212 patients met inclusion criteria, with a short-term mortality rate of 10.4%. The area under the receiver operating characteristic curve (AUROC) of the SORG algorithm when tested on the full validation cohort was 0.82, the calibration intercept was -0.08, the calibration slope was 0.96, and the Brier score was 0.09. CONCLUSIONS: With a contemporaneous and geographically distinct independent cohort, we report successful external validation of a machine learning algorithm for prediction of in-hospital and 90-day postdischarge mortality in SEA.

The Variability and Contributions to Cost of Anterior Cervical Discectomy and Fusion Constructs.

STUDY DESIGN: This was a retrospective cohort study. OBJECTIVE: To characterize the variability in cost for anterior cervical discectomy and fusion (ACDF) constructs and to identify key predictors of procedural cost. SUMMARY OF BACKGROUND DATA: ACDF is commonly performed for surgical treatment of cervical radiculopathy and myelopathy. Numerous biomechanical constructs and graft/biological options are available, with most demonstrating relatively equivalent clinical results. Despite the substantial focus on value in spine care, the differences and contributions to procedural cost in ACDF have not been well defined. MATERIALS AND METHODS: We evaluated the records of patients who underwent a single level ACDF from 2016 to 2020 at 4 hospitals in a major metropolitan area. We abstracted demographics, insurance status, operative time, diagnosis, surgeon, institution, and components of procedural costs. Costs based on construct were compared using multivariable adjusted analyses using negative binomial regression. The primary outcome measures were cost differences between ACDF techniques. RESULTS: Two hundred sixty-four patients were included, with procedures by 13 surgeons across 4 institutions. The total procedural cost for ACDF had a mean of US$2317 with wide variation (range, US$967-US$7370). Multivariable analysis revealed body mass index and use of polyether ether ketone to be correlated with increased cost while carbon fiber and autograft correlated with decreased cost. When comparing standalone device constructs to cases with anterior instrumentation (plate/screws), the total cost was significantly higher in the plate/screw group (US$2686±US$921 vs. US$1466±US$878, P <0.001). CONCLUSIONS: We encountered wide variation in procedural costs associated with ACDF, including as much as an 8-fold difference in the cost of constructs. The most important drivers included instrumentation type and implant materials. Here, we identify potential targets of opportunity for health care organizations that are looking to reduce variance in procedural expenditures to improve health care savings associated with the performance of ACDF.

Development and internal validation of machine learning algorithms for predicting complications after primary total hip arthroplasty.

INTRODUCTION: Complications after total hip arthroplasty (THA) may result in readmission or reoperation and impose a significant cost on the healthcare system. Understanding which patients are at-risk for complications can potentially allow for targeted interventions to decrease complication rates through pursuing preoperative health optimization. The purpose of the current was to develop and internally validate machine learning (ML) algorithms capable of performing patient-specific predictions of all-cause complications within two years of primary THA. METHODS: This was a retrospective case-control study of clinical registry data from 616 primary THA patients from one large academic and two community hospitals. The primary outcome was all-cause complications at a minimum of 2-years after primary THA. Recursive feature elimination was applied to identify preoperative variables with the greatest predictive value. Five ML algorithms were developed on the training set using tenfold cross-validation and internally validated on the independent testing set of patients. Algorithms were assessed by discrimination, calibration, Brier score, and decision curve analysis to quantify performance. RESULTS: The observed complication rate was 16.6%. The stochastic gradient boosting model achieved the best performance with an AUC = 0.88, calibration intercept = 0.1, calibration slope = 1.22, and Brier score = 0.09. The most important factors for predicting complications were age, drug allergies, prior hip surgery, smoking, and opioid use. Individual patient-level explanations were provided for the algorithm predictions and incorporated into an open access digital application: https://sorg-apps.shinyapps.io/tha_complication/ CONCLUSIONS: The stochastic boosting gradient algorithm demonstrated good discriminatory capacity for identifying patients at high-risk of experiencing a postoperative complication and proof-of-concept for creating office-based applications from ML that can perform real-time prediction. However, this clinical utility of the current algorithm is unknown and definitions of complications broad. Further investigation on larger data sets and rigorous external validation is necessary prior to the assessment of clinical utility with respect to risk-stratification of patients undergoing primary THA. LEVEL OF EVIDENCE: III, therapeutic study.

Are Detailed, Patient-level Social Determinant of Health Factors Associated With Physical Function and Mental Health at Presentation Among New Patients With Orthopaedic Conditions?

BACKGROUND: It is well documented that routinely collected patient sociodemographic characteristics (such as race and insurance type) and geography-based social determinants of health (SDoH) measures (for example, the Area Deprivation Index) are associated with health disparities, including symptom severity at presentation. However, the association of patient-level SDoH factors (such as housing status) on musculoskeletal health disparities is not as well documented. Such insight might help with the development of more-targeted interventions to help address health disparities in orthopaedic surgery. QUESTIONS/PURPOSES: (1) What percentage of patients presenting for new patient visits in an orthopaedic surgery clinic who were unemployed but seeking work reported transportation issues that could limit their ability to attend a medical appointment or acquire medications, reported trouble paying for medications, and/or had no current housing? (2) Accounting for traditional sociodemographic factors and patient-level SDoH measures, what factors are associated with poorer patient-reported outcome physical health scores at presentation? (3) Accounting for traditional sociodemographic factor patient-level SDoH measures, what factors are associated with poorer patient-reported outcome mental health scores at presentation? METHODS: New patient encounters at one Level 1 trauma center clinic visit from March 2018 to December 2020 were identified. Included patients had to meet two criteria: they had completed the Patient-Reported Outcome Measure Information System (PROMIS) Global-10 at their new orthopaedic surgery clinic encounter as part of routine clinical care, and they had visited their primary care physician and completed a series of specific SDoH questions. The SDoH questionnaire was developed in our institution to improve data that drive interventions to address health disparities as part of our accountable care organization work. Over the study period, the SDoH questionnaire was only distributed at primary care provider visits. The SDoH questions focused on transportation, housing, employment, and ability to pay for medications. Because we do not have a way to determine how many patients had both primary care provider office visits and new orthopaedic surgery clinic visits over the study period, we were unable to determine how many patients could have been included; however, 9057 patients were evaluated in this cross-sectional study. The mean age was 61 ± 15 years, and most patients self-reported being of White race (83% [7561 of 9057]). Approximately half the patient sample had commercial insurance (46% [4167 of 9057]). To get a better sense of how this study cohort compared with the overall patient population seen at the participating center during the time in question, we reviewed all new patient clinic encounters (n = 135,223). The demographic information between the full patient sample and our study subgroup appeared similar. Using our study cohort, two multivariable linear regression models were created to determine which traditional metrics (for example, self-reported race or insurance type) and patient-specific SDoH factors (for example, lack of reliable transportation) were associated with worse physical and mental health symptoms (that is, lower PROMIS scores) at new patient encounters. The variance inflation factor was used to assess for multicollinearity. For all analyses, p values < 0.05 designated statistical significance. The concept of minimum clinically important difference (MCID) was used to assess clinical importance. Regression coefficients represent the projected change in PROMIS physical or mental health symptom scores (that is, the dependent variable in our regression analyses) accounting for the other included variables. Thus, a regression coefficient for a given variable at or above a known MCID value suggests a clinical difference between those patients with and without the presence of that given characteristic. In this manuscript, regression coefficients at or above 4.2 (or at and below -4.2) for PROMIS Global Physical Health and at or above 5.1 (or at and below -5.1) for PROMIS Global Mental Health were considered clinically relevant. RESULTS: Among the included patients, 8% (685 of 9057) were unemployed but seeking work, 4% (399 of 9057) reported transportation issues that could limit their ability to attend a medical appointment or acquire medications, 4% (328 of 9057) reported trouble paying for medications, and 2% (181 of 9057) had no current housing. Lack of reliable transportation to attend doctor visits or pick up medications (ß = -4.52 [95% CI -5.45 to -3.59]; p < 0.001), trouble paying for medications (ß = -4.55 [95% CI -5.55 to -3.54]; p < 0.001), Medicaid insurance (ß = -5.81 [95% CI -6.41 to -5.20]; p < 0.001), and workers compensation insurance (ß = -5.99 [95% CI -7.65 to -4.34]; p < 0.001) were associated with clinically worse function at presentation. Trouble paying for medications (ß = -6.01 [95% CI -7.10 to -4.92]; p < 0.001), Medicaid insurance (ß = -5.35 [95% CI -6.00 to -4.69]; p < 0.001), and workers compensation (ß = -6.07 [95% CI -7.86 to -4.28]; p < 0.001) were associated with clinically worse mental health at presentation. CONCLUSION: Although transportation issues and financial hardship were found to be associated with worse presenting physical function and mental health, Medicaid and workers compensation insurance remained associated with worse presenting physical function and mental health as well even after controlling for these more detailed, patient-level SDoH factors. Because of that, interventions to decrease health disparities should focus on not only sociodemographic variables (for example, insurance type) but also tangible patient-specific SDoH characteristics. For example, this may include giving patients taxi vouchers or ride-sharing credits to attend clinic visits for patients demonstrating such a need, initiating financial assistance programs for necessary medications, and/or identifying and connecting certain patient groups with social support services early on in the care cycle. LEVEL OF EVIDENCE: Level III, prognostic study.

Utilization of Machine Learning to Model Important Features of 30-day Readmissions following Surgery for Metastatic Spinal Column Tumors: The Influence of Frailty.

STUDY DESIGN: Retrospective cohort study. OBJECTIVE: The aim of this study was to determine the relative importance and predicative power of the Hospital Frailty Risk Score (HFRS) on unplanned 30-day readmission after surgical intervention for metastatic spinal column tumors. METHODS: All adult patients undergoing surgery for metastatic spinal column tumor were identified in the Nationwide Readmission Database from the years 2016 to 2018. Patients were categorized into 3 cohorts based on the criteria of the HFRS: Low(<5), Intermediate(5-14.9), and High(≥ 15). Random Forest (RF) classification was used to construct predictive models for 30-day patient readmission. Model performance was examined using the area under the receiver operating curve (AUC), and the Mean Decrease Gini (MDG) metric was used to quantify and rank features by relative importance. RESULTS: There were 4346 patients included. The proportion of patients who required any readmission were higher among the Intermediate and High frailty cohorts when compared to the Low frailty cohort (Low:33.9% vs. Intermediate:39.3% vs. High:39.2%, P < .001). An RF classifier was trained to predict 30-day readmission on all features (AUC = .60) and architecturally equivalent model trained using only ten features with highest MDG (AUC = .59). Both models found frailty to have the highest importance in predicting risk of readmission. On multivariate regression analysis, Intermediate frailty [OR:1.32, CI(1.06,1.64), P = .012] was found to be an independent predictor of unplanned 30-day readmission. CONCLUSION: Our study utilizes machine learning approaches and predictive modeling to identify frailty as a significant risk-factor that contributes to unplanned 30-day readmission after spine surgery for metastatic spinal column metastases.

Prognostic significance of lab data and performance comparison by validating survival prediction models for patients with spinal metastases after radiotherapy.

BACKGROUND AND PURPOSE: Well-performing survival prediction models (SPMs) help patients and healthcare professionals to choose treatment aligning with prognosis. This retrospective study aims to investigate the prognostic impacts of laboratory data and to compare the performances of Metastases location, Elderly, Tumor primary, Sex, Sickness/comorbidity, and Site of radiotherapy (METSSS) model, New England Spinal Metastasis Score (NESMS), and Skeletal Oncology Research Group machine learning algorithm (SORG-MLA) for spinal metastases (SM). MATERIALS AND METHODS: From 2010 to 2018, patients who received radiotherapy (RT) for SM at a tertiary center were enrolled and the data were retrospectively collected. Multivariate logistic and Cox-proportional-hazard regression analyses were used to assess the association between laboratory values and survival. The area under receiver-operating characteristics curve (AUROC), calibration analysis, Brier score, and decision curve analysis were used to evaluate the performance of SPMs. RESULTS: A total of 2786 patients were included for analysis. The 90-day and 1-year survival rates after RT were 70.4% and 35.7%, respectively. Higher albumin, hemoglobin, or lymphocyte count were associated with better survival, while higher alkaline phosphatase, white blood cell count, neutrophil count, neutrophil-to-lymphocyte ratio, platelet-to-lymphocyte ratio, or international normalized ratio were associated with poor prognosis. SORG-MLA has the best discrimination (AUROC 90-day, 0.78; 1-year 0.76), best calibrations, and the lowest Brier score (90-day 0.16; 1-year 0.18). The decision curve of SORG-MLA is above the other two competing models with threshold probabilities from 0.1 to 0.8. CONCLUSION: Laboratory data are of prognostic significance in survival prediction after RT for SM. Machine learning-based model SORG-MLA outperforms statistical regression-based model METSSS model and NESMS in survival predictions.

Sociodemographic Factors Are Associated with Patient-Reported Outcome Measure Completion in Orthopaedic Surgery: An Analysis of Completion Rates and Determinants Among New Patients.

Patient-reported outcome measures (PROMs) and, specifically, the Patient-Reported Outcomes Measurement Information System (PROMIS), are increasingly utilized for clinical research, clinical care, and health-care policy. However, completion of these outcome measures can be inconsistent and challenging. We hypothesized that sociodemographic variables are associated with the completion of PROM questionnaires. The purposes of the present study were to calculate the completion rate of assigned PROM forms and to identify sociodemographic and other variables associated with completion to help guide improved collection efforts. Methods: All new orthopaedic patients at a single academic medical center were identified from 2016 to 2020. On the basis of subspecialty and presenting condition, patients were assigned certain PROMIS forms and legacy PROMs. Demographic and clinical information was abstracted from the electronic medical record. Bivariate analyses were performed to compare characteristics among those who completed assigned PROMs and those who did not. A multivariable logistic regression model was created to determine which variables were associated with successful completion of assigned PROMs. Results: Of the 219,891 new patients, 88,052 (40%) completed all assigned PROMs. Patients who did not activate their internet-based patient portal had a 62% increased likelihood of not completing assigned PROMs (odds ratio [OR], 1.62; 95% confidence interval [CI], 1.58 to 1.66; p < 0.001). Non-English-speaking patients had a 90% (OR, 1.90; 95% CI, 1.82 to 2.00; p < 0.001) increased likelihood of not completing assigned PROMs at presentation. Older patients (≥65 years of age) and patients of Black race had a 23% (OR, 1.23; 95% CI, 1.19 to 1.27; p < 0.001) and 24% (OR, 1.24; 95% CI, 1.19 to 1.30; p < 0.001) increased likelihood of not completing assigned PROMs, respectively. Conclusions: The rate of completion of PROMs varies according to sociodemographic variables. This variability could bias clinical outcomes research in orthopaedic surgery. The present study highlights the need to uniformly increase completion rates so that outcomes research incorporates truly representative cohorts of patients treated. Furthermore, the use of these PROMs to guide health-care policy decisions necessitates a representative patient distribution to avoid bias in the health-care system. Level of Evidence: Prognostic Level III. See Instructions for Authors for a complete description of levels of evidence.

Development and external validation of predictive algorithms for six-week mortality in spinal metastasis using 4,304 patients from five institutions.

BACKGROUND CONTEXT: Historically, spine surgeons used expected postoperative survival of 3-months to help select candidates for operative intervention in spinal metastasis. However, this cutoff has been challenged by the development of minimally invasive techniques, novel biologics, and advanced radiotherapy. Recent studies have suggested that a life expectancy of 6 weeks may be enough to achieve significant improvements in postoperative health-related quality of life. PURPOSE: The purpose of this study was to develop a model capable of predicting 6-week mortality in patients with spinal metastases treated with radiation or surgery. STUDY DESIGN/SETTING: A retrospective review was conducted at five large tertiary centers in the United States and Taiwan. PATIENT SAMPLE: The development cohort consisted of 3,001 patients undergoing radiotherapy and/or surgery for spinal metastases from one institution. The validation institutional cohort consisted of 1,303 patients from four independent, external institutions. OUTCOME MEASURES: The primary outcome was 6-week mortality. METHODS: Five models were considered to predict 6-week mortality, and the model with the best performance across discrimination, calibration, decision-curve analysis, and overall performance was integrated into an open access web-based application. RESULTS: The most important variables for prediction of 6-week mortality were albumin, primary tumor histology, absolute lymphocyte, three or more spine metastasis, and ECOG score. The elastic-net penalized logistic model was chosen as the best performing model with AUC 0.84 on evaluation in the independent testing set. On external validation in the 1,303 patients from the four independent institutions, the model retained good discriminative ability with an area under the curve of 0.81. The model is available here: https://sorg-apps.shinyapps.io/spinemetssurvival/. CONCLUSIONS: While this study does not advocate for the use of a 6-week life expectancy as criteria for considering operative management, the algorithm developed and externally validated in this study may be helpful for preoperative planning, multidisciplinary management, and shared decision-making in spinal metastasis patients with shorter life expectancy.

Predictors of reoperation after surgery for spinal epidural abscess.

BACKGROUND CONTEXT: Spinal epidural abscess is a rare but severe condition with high rates of postoperative adverse events. PURPOSE: The objective of the study was to identify independent prognostic factors for reoperation using two datasets: an institutional and national database. STUDY DESIGN/SETTING: Retrospective Review. PATIENT SAMPLE: Database 1: Review of five medical centers from 1993 to 2016. Database 2: The National Surgical Quality Improvement Program (NSQIP) was queried between 2012 and 2016. OUTCOME MEASURES: Thirty-day and ninety-day reoperation rate. METHODS: Two independent datasets were reviewed to identify patients with spinal epidural abscesses undergoing spinal surgery. Multivariate analyses were used to determine independent prognostic factors for reoperation while including factors identified in bivariate analyses. RESULTS: Overall, 642 patients underwent surgery for a spinal epidural abscess in the institutional cohort, with a 90-day unplanned reoperation rate of 19.9%. In the NSQIP database, 951 patients were identified with a 30-day unplanned reoperation rate of 12.3%. On multivariate analysis in the NSQIP database, cervical spine abscess was the only factor that reached significance for 30-day reoperation (OR=1.71, 95% CI=1.11-2.63, p=.02, Area under the curve (AUC)=0.61). On multivariate analysis in the institutional cohort, independent prognostic factors for 30-day reoperation were: preoperative urinary incontinence, ventral location of abscess relative to thecal sac, cervical abscess, preoperative wound infection, and leukocytosis (AUC=0.65). Ninety-day reoperation rate also found hypoalbuminemia as a significant predictor (AUC=0.66). CONCLUSION: Six novel independent prognostic factors were identified for 90-day reoperation after surgery for a spinal epidural abscess. The multivariable analysis fairly predicts reoperation, indicating that there may be additional factors that need to be uncovered in future studies. The risk factors delineated in this study through the use of two large cohorts of spinal epidural abscess patients can be used to improve preoperative risk stratification and patient management.

Does the SORG Orthopaedic Research Group Hip Fracture Delirium Algorithm Perform Well on an Independent Intercontinental Cohort of Patients With Hip Fractures Who Are 60 Years or Older?

BACKGROUND: Postoperative delirium in patients aged 60 years or older with hip fractures adversely affects clinical and functional outcomes. The economic cost of delirium is estimated to be as high as USD 25,000 per patient, with a total budgetary impact between USD 6.6 to USD 82.4 billion annually in the United States alone. Forty percent of delirium episodes are preventable, and accurate risk stratification can decrease the incidence and improve clinical outcomes in patients. A previously developed clinical prediction model (the SORG Orthopaedic Research Group hip fracture delirium machine-learning algorithm) is highly accurate on internal validation (in 28,207 patients with hip fractures aged 60 years or older in a US cohort) in identifying at-risk patients, and it can facilitate the best use of preventive interventions; however, it has not been tested in an independent population. For an algorithm to be useful in real life, it must be valid externally, meaning that it must perform well in a patient cohort different from the cohort used to "train" it. With many promising machine-learning prediction models and many promising delirium models, only few have also been externally validated, and even fewer are international validation studies. QUESTION/PURPOSE: Does the SORG hip fracture delirium algorithm, initially trained on a database from the United States, perform well on external validation in patients aged 60 years or older in Australia and New Zealand? METHODS: We previously developed a model in 2021 for assessing risk of delirium in hip fracture patients using records of 28,207 patients obtained from the American College of Surgeons National Surgical Quality Improvement Program. Variables included in the original model included age, American Society of Anesthesiologists (ASA) class, functional status (independent or partially or totally dependent for any activities of daily living), preoperative dementia, preoperative delirium, and preoperative need for a mobility aid. To assess whether this model could be applied elsewhere, we used records from an international hip fracture registry. Between June 2017 and December 2018, 6672 patients older than 60 years of age in Australia and New Zealand were treated surgically for a femoral neck, intertrochanteric hip, or subtrochanteric hip fracture and entered into the Australian & New Zealand Hip Fracture Registry. Patients were excluded if they had a pathological hip fracture or septic shock. Of all patients, 6% (402 of 6672) did not meet the inclusion criteria, leaving 94% (6270 of 6672) of patients available for inclusion in this retrospective analysis. Seventy-one percent (4249 of 5986) of patients were aged 80 years or older, after accounting for 5% (284 of 6270) of missing values; 68% (4292 of 6266) were female, after accounting for 0.06% (4 of 6270) of missing values, and 83% (4690 of 5661) of patients were classified as ASA III/IV, after accounting for 10% (609 of 6270) of missing values. Missing data were imputed using the missForest methodology. In total, 39% (2467 of 6270) of patients developed postoperative delirium. The performance of the SORG hip fracture delirium algorithm on the validation cohort was assessed by discrimination, calibration, Brier score, and a decision curve analysis. Discrimination, known as the area under the receiver operating characteristic curves (c-statistic), measures the model's ability to distinguish patients who achieved the outcomes from those who did not and ranges from 0.5 to 1.0, with 1.0 indicating the highest discrimination score and 0.50 the lowest. Calibration plots the predicted versus the observed probabilities, a perfect plot has an intercept of 0 and a slope of 1. The Brier score calculates a composite of discrimination and calibration, with 0 indicating perfect prediction and 1 the poorest. RESULTS: The SORG hip fracture algorithm, when applied to an external patient cohort, distinguished between patients at low risk and patients at moderate to high risk of developing postoperative delirium. The SORG hip fracture algorithm performed with a c-statistic of 0.74 (95% confidence interval 0.73 to 0.76). The calibration plot showed high accuracy in the lower predicted probabilities (intercept -0.28, slope 0.52) and a Brier score of 0.22 (the null model Brier score was 0.24). The decision curve analysis showed that the model can be beneficial compared with no model or compared with characterizing all patients as at risk for developing delirium. CONCLUSION: Algorithms developed with machine learning are a potential tool for refining treatment of at-risk patients. If high-risk patients can be reliably identified, resources can be appropriately directed toward their care. Although the current iteration of SORG should not be relied on for patient care, it suggests potential utility in assessing risk. Further assessment in different populations, made easier by international collaborations and standardization of registries, would be useful in the development of universally valid prediction models. The model can be freely accessed at: https://sorg-apps.shinyapps.io/hipfxdelirium/ . LEVEL OF EVIDENCE: Level III, therapeutic study.

What Is the Clinical Benefit of Common Orthopaedic Procedures as Assessed by the PROMIS Versus Other Validated Outcomes Tools?

BACKGROUND: Patient-reported outcome measures (PROMs), including the Patient-reported Outcomes Measurement Information System (PROMIS), are increasingly used to measure healthcare value. The minimum clinically important difference (MCID) is a metric that helps clinicians determine whether a statistically detectable improvement in a PROM after surgical care is likely to be large enough to be important to a patient or to justify an intervention that carries risk and cost. There are two major categories of MCID calculation methods, anchor-based and distribution-based. This variability, coupled with heterogeneous surgical cohorts used for existing MCID values, limits their application to clinical care. QUESTIONS/PURPOSES: In our study, we sought (1) to determine MCID thresholds and attainment percentages for PROMIS after common orthopaedic procedures using distribution-based methods, (2) to use anchor-based MCID values from published studies as a comparison, and (3) to compare MCID attainment percentages using PROMIS scores to other validated outcomes tools such as the Hip Disability and Osteoarthritis Outcome Score (HOOS) and Knee Disability and Osteoarthritis Outcome Score (KOOS). METHODS: This was a retrospective study at two academic medical centers and three community hospitals. The inclusion criteria for this study were patients who were age 18 years or older and who underwent elective THA for osteoarthritis, TKA for osteoarthritis, one-level posterior lumbar fusion for lumbar spinal stenosis or spondylolisthesis, anatomic total shoulder arthroplasty or reverse total shoulder arthroplasty for glenohumeral arthritis or rotator cuff arthropathy, arthroscopic anterior cruciate ligament reconstruction, arthroscopic partial meniscectomy, or arthroscopic rotator cuff repair. This yielded 14,003 patients. Patients undergoing revision operations or surgery for nondegenerative pathologies and patients without preoperative PROMs assessments were excluded, leaving 9925 patients who completed preoperative PROMIS assessments and 9478 who completed other preoperative validated outcomes tools (HOOS, KOOS, numerical rating scale for leg pain, numerical rating scale for back pain, and QuickDASH). Approximately 66% (6529 of 9925) of patients had postoperative PROMIS scores (Physical Function, Mental Health, Pain Intensity, Pain Interference, and Upper Extremity) and were included for analysis. PROMIS scores are population normalized with a mean score of 50 ± 10, with most scores falling between 30 to 70. Approximately 74% (7007 of 9478) of patients had postoperative historical assessment scores and were included for analysis. The proportion who reached the MCID was calculated for each procedure cohort at 6 months of follow-up using distribution-based MCID methods, which included a fraction of the SD (1/2 or 1/3 SD) and minimum detectable change (MDC) using statistical significance (such as the MDC 90 from p < 0.1). Previously published anchor-based MCID thresholds from similar procedure cohorts and analogous PROMs were used to calculate the proportion reaching MCID. RESULTS: Within a given distribution-based method, MCID thresholds for PROMIS assessments were similar across multiple procedures. The MCID threshold ranged between 3.4 and 4.5 points across all procedures using the 1/2 SD method. Except for meniscectomy (3.5 points), the anchor-based PROMIS MCID thresholds (range 4.5 to 8.1 points) were higher than the SD distribution-based MCID values (2.3 to 4.5 points). The difference in MCID thresholds based on the calculation method led to a similar trend in MCID attainment. Using THA as an example, MCID attainment using PROMIS was achieved by 76% of patients using an anchor-based threshold of 7.9 points. However, 82% of THA patients attained MCID using the MDC 95 method (6.1 points), and 88% reached MCID using the 1/2 SD method (3.9 points). Using the HOOS metric (scaled from 0 to 100), 86% of THA patients reached the anchor-based MCID threshold (17.5 points). However, 91% of THA patients attained the MCID using the MDC 90 method (12.5 points), and 93% reached MCID using the 1/2 SD method (8.4 points). In general, the proportion of patients reaching MCID was lower for PROMIS than for other validated outcomes tools; for example, with the 1/2 SD method, 72% of patients who underwent arthroscopic partial meniscectomy reached the MCID on PROMIS Physical Function compared with 86% on KOOS. CONCLUSION: MCID calculations can provide clinical correlation for PROM scores interpretation. The PROMIS form is increasingly used because of its generalizability across diagnoses. However, we found lower proportions of MCID attainment using PROMIS scores compared with historical PROMs. By using historical proportions of attainment on common orthopaedic procedures and a spectrum of MCID calculation techniques, the PROMIS MCID benchmarks are realizable for common orthopaedic procedures. For clinical practices that routinely collect PROMIS scores in the clinical setting, these results can be used by individual surgeons to evaluate personal practice trends and by healthcare systems to quantify whether clinical care initiatives result in meaningful differences. Furthermore, these MCID thresholds can be used by researchers conducting retrospective outcomes research with PROMIS. LEVEL OF EVIDENCE: Level III, therapeutic study.