Predicting short- to medium-term care home admission risk in older adults: a systematic review of externally validated models.

INTRODUCTION: Predicting risk of care home admission could identify older adults for early intervention to support independent living but require external validation in a different dataset before clinical use. We systematically reviewed external validations of care home admission risk prediction models in older adults. METHODS: We searched Medline, Embase and Cochrane Library until 14 August 2023 for external validations of prediction models for care home admission risk in adults aged ≥65 years with up to 3 years of follow-up. We extracted and narratively synthesised data on study design, model characteristics, and model discrimination and calibration (accuracy of predictions). We assessed risk of bias and applicability using Prediction model Risk Of Bias Assessment Tool. RESULTS: Five studies reporting validations of nine unique models were included. Model applicability was fair but risk of bias was mostly high due to not reporting model calibration. Morbidities were used as predictors in four models, most commonly neurological or psychiatric diseases. Physical function was also included in four models. For 1-year prediction, three of the six models had acceptable discrimination (area under the receiver operating characteristic curve (AUC)/c statistic 0.70-0.79) and the remaining three had poor discrimination (AUC < 0.70). No model accounted for competing mortality risk. The only study examining model calibration (but ignoring competing mortality) concluded that it was excellent. CONCLUSIONS: The reporting of models was incomplete. Model discrimination was at best acceptable, and calibration was rarely examined (and ignored competing mortality risk when examined). There is a need to derive better models that account for competing mortality risk and report calibration as well as discrimination.

Multimorbidity analysis with low condition counts: a robust Bayesian approach for small but important subgroups.

BACKGROUND: Robustly examining associations between long-term conditions may be important in identifying opportunities for intervention in multimorbidity but is challenging when evidence is limited. We have developed a Bayesian inference framework that is robust to sparse data and used it to quantify morbidity associations in the oldest old, a population with limited available data. METHODS: We conducted a retrospective cross-sectional study of a representative dataset of primary care patients in Scotland as of March 2007. We included 40 long-term conditions and studied their associations in 12,009 individuals aged 90 and older, stratified by sex (3039 men, 8970 women). We analysed associations obtained with Relative Risk (RR), a standard measure in the literature, and compared them with our proposed measure, Associations Beyond Chance (ABC). To enable a broad exploration of interactions between long-term conditions, we built networks of association and assessed differences in their analysis when associations are estimated by RR or ABC. FINDINGS: Our Bayesian framework was appropriately more cautious in attributing association when evidence is lacking, particularly in uncommon conditions. This caution in reporting association was also present in reporting differences in associations between sex and affected the aggregated measures of multimorbidity and network representations. INTERPRETATION: Incorporating uncertainty into multimorbidity research is crucial to avoid misleading findings when evidence is limited, a problem that particularly affects small but important subgroups. Our proposed framework improves the reliability of estimations of associations and, more in general, of research into disease mechanisms and multimorbidity. FUNDING: National Institute for Health and Care Research.

Performance of models for predicting 1-year to 3-year mortality in older adults: a systematic review of externally validated models.

Mortality prediction models support identifying older adults with short life expectancy for whom clinical care might need modifications. We systematically reviewed external validations of mortality prediction models in older adults (ie, aged 65 years and older) with up to 3 years of follow-up. In March, 2023, we conducted a literature search resulting in 36 studies reporting 74 validations of 64 unique models. Model applicability was fair but validation risk of bias was mostly high, with 50 (68%) of 74 validations not reporting calibration. Morbidities (most commonly cardiovascular diseases) were used as predictors by 45 (70%) of 64 of models. For 1-year prediction, 31 (67%) of 46 models had acceptable discrimination, but only one had excellent performance. Models with more than 20 predictors were more likely to have acceptable discrimination (risk ratio [RR] vs <10 predictors 1·68, 95% CI 1·06-2·66), as were models including sex (RR 1·75, 95% CI 1·12-2·73) or predicting risk during comprehensive geriatric assessment (RR 1·86, 95% CI 1·12-3·07). Development and validation of better-performing mortality prediction models in older people are needed.

Development of prediction models for one-year brain tumour survival using machine learning: a comparison of accuracy and interpretability.

BACKGROUND AND OBJECTIVE: Prediction of survival in patients diagnosed with a brain tumour is challenging because of heterogeneous tumour behaviours and treatment response. Advances in machine learning have led to the development of clinical prognostic models, but due to the lack of model interpretability, integration into clinical practice is almost non-existent. In this retrospective study, we compare five classification models with varying degrees of interpretability for the prediction of brain tumour survival greater than one year following diagnosis. METHODS: 1028 patients aged ≥16 years with a brain tumour diagnosis between April 2012 and April 2020 were included in our study. Three intrinsically interpretable 'glass box' classifiers (Bayesian Rule Lists [BRL], Explainable Boosting Machine [EBM], and Logistic Regression [LR]), and two 'black box' classifiers (Random Forest [RF] and Support Vector Machine [SVM]) were trained on electronic patients records for the prediction of one-year survival. All models were evaluated using balanced accuracy (BAC), F1-score, sensitivity, specificity, and receiver operating characteristics. Black box model interpretability and misclassified predictions were quantified using SHapley Additive exPlanations (SHAP) values and model feature importance was evaluated by clinical experts. RESULTS: The RF model achieved the highest BAC of 78.9%, closely followed by SVM (77.7%), LR (77.5%) and EBM (77.1%). Across all models, age, diagnosis (tumour type), functional features, and first treatment were top contributors to the prediction of one year survival. We used EBM and SHAP to explain model misclassifications and investigated the role of feature interactions in prognosis. CONCLUSION: Interpretable models are a natural choice for the domain of predictive medicine. Intrinsically interpretable models, such as EBMs, may provide an advantage over traditional clinical assessment of brain tumour prognosis by weighting potential risk factors and their interactions that may be unknown to clinicians. An agreement between model predictions and clinical knowledge is essential for establishing trust in the models decision making process, as well as trust that the model will make accurate predictions when applied to new data.