Calibration drift in regression and machine learning models for acute kidney injury.
J Am Med Inform Assoc
; 24(6): 1052-1061, 2017 Nov 01.
Article
em En
| MEDLINE
| ID: mdl-28379439
ABSTRACT
OBJECTIVE:
Predictive analytics create opportunities to incorporate personalized risk estimates into clinical decision support. Models must be well calibrated to support decision-making, yet calibration deteriorates over time. This study explored the influence of modeling methods on performance drift and connected observed drift with data shifts in the patient population. MATERIALS ANDMETHODS:
Using 2003 admissions to Department of Veterans Affairs hospitals nationwide, we developed 7 parallel models for hospital-acquired acute kidney injury using common regression and machine learning methods, validating each over 9 subsequent years.RESULTS:
Discrimination was maintained for all models. Calibration declined as all models increasingly overpredicted risk. However, the random forest and neural network models maintained calibration across ranges of probability, capturing more admissions than did the regression models. The magnitude of overprediction increased over time for the regression models while remaining stable and small for the machine learning models. Changes in the rate of acute kidney injury were strongly linked to increasing overprediction, while changes in predictor-outcome associations corresponded with diverging patterns of calibration drift across methods.CONCLUSIONS:
Efficient and effective updating protocols will be essential for maintaining accuracy of, user confidence in, and safety of personalized risk predictions to support decision-making. Model updating protocols should be tailored to account for variations in calibration drift across methods and respond to periods of rapid performance drift rather than be limited to regularly scheduled annual or biannual intervals.Palavras-chave
Texto completo:
1
Coleções:
01-internacional
Base de dados:
MEDLINE
Assunto principal:
Modelos Logísticos
/
Técnicas de Apoio para a Decisão
/
Injúria Renal Aguda
/
Aprendizado de Máquina
Tipo de estudo:
Guideline
/
Prognostic_studies
/
Risk_factors_studies
Limite:
Aged
/
Female
/
Humans
/
Male
/
Middle aged
País/Região como assunto:
America do norte
Idioma:
En
Revista:
J Am Med Inform Assoc
Ano de publicação:
2017
Tipo de documento:
Article