Development and Performance of Electronic Pediatric Risk of Mortality and Pediatric Logistic Organ Dysfunction-2 Automated Acuity Scores.

ABSTRACT

OBJECTIVES: Develop and test the performance of electronic version of the Children's Hospital of Pittsburgh Pediatric Risk of Mortality-IV and electronic version of the Children's Hospital of Pittsburgh Pediatric Logistic Organ Dysfunction-2 scores. DESIGN: Retrospective, single-center cohort derived from structured electronic health record data. SETTING: Large, quaternary PICU at a freestanding, university-affiliated children's hospital. PATIENTS All encounters with a PICU admission between January 1, 2009, and December 31, 2017, identified using electronic definitions of inpatient encounter. INTERVENTIONS: None. MEASUREMENTS AND MAIN RESULTS: The main outcome was predictive validity of each score for hospital mortality, assessed as model discrimination and calibration. Discrimination was examined with the area under the receiver operating characteristics curve and the area under the precision-recall curve. Calibration was assessed with the Hosmer-Lemeshow goodness of fit test and calculation of a standardized mortality ratio. Models were recalibrated with new regression coefficients in a training subset of 75% of encounters selected randomly from all years of the cohort and the calibrated models were tested in the remaining 25% of the cohort. Content validity was assessed by examining correlation between electronic versions of the scores and prospectively calculated data (electronic version of the Children's Hospital of Pittsburgh Pediatric Risk of Mortality-IV) and an alternative informatics approach (Children's Hospital of Pittsburgh Pediatric Logistic Organ Dysfunction-2 score). The cohort included 21,335 encounters. Correlation coefficients indicated strong agreement between different methods of score calculation. Uncalibrated area under the receiver operating characteristics curves were 0.96 (95% CI, 0.95-0.97) for electronic version of the Children's Hospital of Pittsburgh Pediatric Logistic Organ Dysfunction-2 score and 0.87 (95% CI, 0.85-0.89) for electronic version of the Children's Hospital of Pittsburgh Pediatric Risk of Mortality-IV for inpatient mortality. The uncalibrated electronic version of the Children's Hospital of Pittsburgh Pediatric Risk of Mortality-IV standardized mortality ratio was 0.63 (0.59-0.66), demonstrating strong agreement with previous, prospective evaluation at the study center. The uncalibrated electronic version of the Children's Hospital of Pittsburgh Pediatric Logistic Organ Dysfunction-2 score standardized mortality ratio was 0.20 (0.18-0.21). All models required recalibrating (all Hosmer-Lemeshow goodness-of-fit, p < 0.001) and subsequently demonstrated acceptable goodness-of-fit when examined in a test subset (n = 5,334) of the cohort. CONCLUSIONS: Electronically derived intensive care acuity scores demonstrate very good to excellent discrimination and can be calibrated to institutional outcomes. This approach can facilitate both performance improvement and research initiatives and may offer a scalable strategy for comparison of interinstitutional PICU outcomes.