Monitoring Approaches for a Pediatric Chronic Kidney Disease Machine Learning Model.
Appl Clin Inform
; 13(2): 431-438, 2022 03.
Article
em En
| MEDLINE
| ID: mdl-35508197
ABSTRACT
OBJECTIVE:
The purpose of this study is to evaluate the ability of three metrics to monitor for a reduction in performance of a chronic kidney disease (CKD) model deployed at a pediatric hospital.METHODS:
The CKD risk model estimates a patient's risk of developing CKD 3 to 12 months following an inpatient admission. The model was developed on a retrospective dataset of 4,879 admissions from 2014 to 2018, then run silently on 1,270 admissions from April to October, 2019. Three metrics were used to monitor its performance during the silent phase (1) standardized mean differences (SMDs); (2) performance of a "membership model"; and (3) response distribution analysis. Observed patient outcomes for the 1,270 admissions were used to calculate prospective model performance and the ability of the three metrics to detect performance changes.RESULTS:
The deployed model had an area under the receiver-operator curve (AUROC) of 0.63 in the prospective evaluation, which was a significant decrease from an AUROC of 0.76 on retrospective data (p = 0.033). Among the three metrics, SMDs were significantly different for 66/75 (88%) of the model's input variables (p <0.05) between retrospective and deployment data. The membership model was able to discriminate between the two settings (AUROC = 0.71, p <0.0001) and the response distributions were significantly different (p <0.0001) for the two settings.CONCLUSION:
This study suggests that the three metrics examined could provide early indication of performance deterioration in deployed models' performance.
Texto completo:
1
Coleções:
01-internacional
Base de dados:
MEDLINE
Assunto principal:
Simulação por Computador
/
Insuficiência Renal Crônica
/
Aprendizado de Máquina
Tipo de estudo:
Diagnostic_studies
/
Etiology_studies
/
Evaluation_studies
/
Observational_studies
/
Prognostic_studies
/
Risk_factors_studies
Limite:
Child
/
Female
/
Humans
/
Male
Idioma:
En
Revista:
Appl Clin Inform
Ano de publicação:
2022
Tipo de documento:
Article
País de afiliação:
Estados Unidos