External validation of existing dementia prediction models on observational health data.
BMC Med Res Methodol
; 22(1): 311, 2022 12 05.
Article
in En
| MEDLINE
| ID: mdl-36471238
ABSTRACT
BACKGROUND:
Many dementia prediction models have been developed, but only few have been externally validated, which hinders clinical uptake and may pose a risk if models are applied to actual patients regardless. Externally validating an existing prediction model is a difficult task, where we mostly rely on the completeness of model reporting in a published article. In this study, we aim to externally validate existing dementia prediction models. To that end, we define model reporting criteria, review published studies, and externally validate three well reported models using routinely collected health data from administrative claims and electronic health records.METHODS:
We identified dementia prediction models that were developed between 2011 and 2020 and assessed if they could be externally validated given a set of model criteria. In addition, we externally validated three of these models (Walters' Dementia Risk Score, Mehta's RxDx-Dementia Risk Index, and Nori's ADRD dementia prediction model) on a network of six observational health databases from the United States, United Kingdom, Germany and the Netherlands, including the original development databases of the models.RESULTS:
We reviewed 59 dementia prediction models. All models reported the prediction method, development database, and target and outcome definitions. Less frequently reported by these 59 prediction models were predictor definitions (52 models) including the time window in which a predictor is assessed (21 models), predictor coefficients (20 models), and the time-at-risk (42 models). The validation of the model by Walters (development c-statistic 0.84) showed moderate transportability (0.67-0.76 c-statistic). The Mehta model (development c-statistic 0.81) transported well to some of the external databases (0.69-0.79 c-statistic). The Nori model (development AUROC 0.69) transported well (0.62-0.68 AUROC) but performed modestly overall. Recalibration showed improvements for the Walters and Nori models, while recalibration could not be assessed for the Mehta model due to unreported baseline hazard.CONCLUSION:
We observed that reporting is mostly insufficient to fully externally validate published dementia prediction models, and therefore, it is uncertain how well these models would work in other clinical settings. We emphasize the importance of following established guidelines for reporting clinical prediction models. We recommend that reporting should be more explicit and have external validation in mind if the model is meant to be applied in different settings.Key words
Full text:
1
Collection:
01-internacional
Database:
MEDLINE
Main subject:
Dementia
Type of study:
Diagnostic_studies
/
Etiology_studies
/
Guideline
/
Prognostic_studies
/
Risk_factors_studies
Limits:
Humans
Country/Region as subject:
Europa
Language:
En
Journal:
BMC Med Res Methodol
Journal subject:
MEDICINA
Year:
2022
Document type:
Article
Affiliation country: