Prediction Model Performance With Different Imputation Strategies: A Simulation Study Using a North American ICU Registry.

OBJECTIVES: To evaluate the performance of pragmatic imputation approaches when estimating model coefficients using datasets with varying degrees of data missingness. DESIGN: Performance in predicting observed mortality in a registry dataset was evaluated using simulations of two simple logistic regression models with age-specific criteria for abnormal vital signs (mentation, systolic blood pressure, respiratory rate, WBC count, heart rate, and temperature). Starting with a dataset with complete information, increasing degrees of biased missingness of WBC and mentation were introduced, depending on the values of temperature and systolic blood pressure, respectively. Missing data approaches evaluated included analysis of complete cases only, assuming missing data are normal, and multiple imputation by chained equations. Percent bias and root mean square error, in relation to parameter estimates obtained from the original data, were evaluated as performance indicators. SETTING: Data were obtained from the Virtual Pediatric Systems, LLC, database (Los Angeles, CA), which provides clinical markers and outcomes in prospectively collected records from 117 PICUs in the United States and Canada. PATIENTS: Children admitted to a participating PICU in 2017, for whom all required data were available. INTERVENTIONS: None. MEASUREMENTS AND MAIN RESULTS: Simulations demonstrated that multiple imputation by chained equations is an effective strategy and that even a naive implementation of multiple imputation by chained equations significantly outperforms traditional approaches: the root mean square error for model coefficients was lower using multiple imputation by chained equations in 90 of 99 of all simulations (91%) compared with discarding cases with missing data and lower in 97 of 99 (98%) compared with models assuming missing values are in the normal range. Assuming missing data to be abnormal was inferior to all other approaches. CONCLUSIONS: Analyses of large observational studies are likely to encounter the issue of missing data, which are likely not missing at random. Researchers should always consider multiple imputation by chained equations (or similar imputation approaches) when encountering even only small proportions of missing data in their work.

Postsurgical Pain Risk Stratification to Enhance Pain Management Workflow in Adult Patients: Design, Implementation, and Pilot Evaluation.

BACKGROUND: Exposure to opioids after surgery is the initial contact for some people who develop chronic opioid use disorder. Hence, effective postoperative pain management, with less reliance on opioids, is critical. The Perioperative Opioid Quality Improvement (POQI) program developed (1) a digital health platform leveraging patient-survey-reported risk factors and (2) a postsurgical pain risk stratification algorithm to personalize perioperative care by integrating several commercially available digital health solutions into a combined platform. Development was reduced in scope by the COVID-19 pandemic. OBJECTIVE: This pilot study aims to assess the screening performance of the risk algorithm, quantify the use of the POQI platform, and evaluate clinicians' and patients' perceptions of its utility and benefit. METHODS: A POQI platform prototype was implemented in a quality improvement initiative at a Canadian tertiary care center and evaluated from January to September 2022. After surgical booking, a preliminary risk stratification algorithm was applied to health history questionnaire responses. The estimated risk guided the patient assignment to a care pathway based on low or high risk for persistent pain and opioid use. Demographic, procedural, and medication administration data were extracted retrospectively from the electronic medical record. Postoperative inpatient opioid use of >90 morphine milligram equivalents per day was the outcome used to assess algorithm performance. Data were summarized and compared between the low- and high-risk groups. POQI use was assessed by completed surveys on postoperative days 7, 14, 30, 60, 90, and 120. Semistructured patient and clinician interviews provided qualitative feedback on the platform. RESULTS: Overall, 276 eligible patients were admitted for colorectal procedures. The risk algorithm stratified 203 (73.6%) as the low-risk group and 73 (26.4%) as the high-risk group. Among the 214 (77.5%) patients with available data, high-risk patients were younger than low-risk patients (age: median 53, IQR 40-65 years, vs median 59, IQR 49-69 years, median difference five years, 95% CI 1-9; P=.02) and were more often female patients (45/73, 62% vs 80/203, 39.4%; odds ratio 2.5, 95% CI 1.4-4.5; P=.002). The risk stratification was reasonably specific (true negative rate=144/200, 72%) but not sensitive (true positive rate=10/31, 32%). Only 39.7% (85/214) patients completed any postoperative quality of recovery questionnaires (only 14, 6.5% patients beyond 60 days after surgery), and 22.9% (49/214) completed a postdischarge medication survey. Interviewed participants welcomed the initiative but noted usability issues and poor platform education. CONCLUSIONS: An initial POQI platform prototype was deployed operationally; the risk algorithm had reasonable specificity but poor sensitivity. There was a significant loss to follow-up in postdischarge survey completion. Clinicians and patients appreciated the potential impact of preemptively addressing opioid exposure but expressed shortcomings in the platform's design and implementation. Iterative platform redesign with additional features and reevaluation are required before broader implementation.

Dashboard of Short-Term Postoperative Patient Outcomes for Anesthesiologists: Development and Preliminary Evaluation.

BACKGROUND: Anesthesiologists require an understanding of their patients' outcomes to evaluate their performance and improve their practice. Traditionally, anesthesiologists had limited information about their surgical outpatients' outcomes due to minimal contact post discharge. Leveraging digital health innovations for analyzing personal and population outcomes may improve perioperative care. BC Children's Hospital's postoperative follow-up registry for outpatient surgeries collects short-term outcomes such as pain, nausea, and vomiting. Yet, these data were previously not available to anesthesiologists. OBJECTIVE: This quality improvement study aimed to visualize postoperative outcome data to allow anesthesiologists to reflect on their care and compare their performance with their peers. METHODS: The postoperative follow-up registry contains nurse-reported postoperative outcomes, including opioid and antiemetic administration in the postanesthetic care unit (PACU), and family-reported outcomes, including pain, nausea, and vomiting, within 24 hours post discharge. Dashboards were iteratively co-designed with 5 anesthesiologists, and a department-wide usability survey gathered anesthesiologists' feedback on the dashboards, allowing further design improvements. A final dashboard version has been deployed, with data updated weekly. RESULTS: The dashboard contains three sections: (1) 24-hour outcomes, (2) PACU outcomes, and (3) a practice profile containing individual anesthesiologist's case mix, grouped by age groups, sex, and surgical service. At the time of evaluation, the dashboard included 24-hour data from 7877 cases collected from September 2020 to February 2023 and PACU data from 8716 cases collected from April 2021 to February 2023. The co-design process and usability evaluation indicated that anesthesiologists preferred simpler designs for data summaries but also required the ability to explore details of specific outcomes and cases if needed. Anesthesiologists considered security and confidentiality to be key features of the design and most deemed the dashboard information useful and potentially beneficial for their practice. CONCLUSIONS: We designed and deployed a dynamic, personalized dashboard for anesthesiologists to review their outpatients' short-term postoperative outcomes. This dashboard facilitates personal reflection on individual practice in the context of peer and departmental performance and, hence, the opportunity to evaluate iterative practice changes. Further work is required to establish their effect on improving individual and department performance and patient outcomes.

An Artificial Neural Network-Based Pediatric Mortality Risk Score: Development and Performance Evaluation Using Data From a Large North American Registry.

BACKGROUND: In the pediatric intensive care unit (PICU), quantifying illness severity can be guided by risk models to enable timely identification and appropriate intervention. Logistic regression models, including the pediatric index of mortality 2 (PIM-2) and pediatric risk of mortality III (PRISM-III), produce a mortality risk score using data that are routinely available at PICU admission. Artificial neural networks (ANNs) outperform regression models in some medical fields. OBJECTIVE: In light of this potential, we aim to examine ANN performance, compared to that of logistic regression, for mortality risk estimation in the PICU. METHODS: The analyzed data set included patients from North American PICUs whose discharge diagnostic codes indicated evidence of infection and included the data used for the PIM-2 and PRISM-III calculations and their corresponding scores. We stratified the data set into training and test sets, with approximately equal mortality rates, in an effort to replicate real-world data. Data preprocessing included imputing missing data through simple substitution and normalizing data into binary variables using PRISM-III thresholds. A 2-layer ANN model was built to predict pediatric mortality, along with a simple logistic regression model for comparison. Both models used the same features required by PIM-2 and PRISM-III. Alternative ANN models using single-layer or unnormalized data were also evaluated. Model performance was compared using the area under the receiver operating characteristic curve (AUROC) and the area under the precision recall curve (AUPRC) and their empirical 95% CIs. RESULTS: Data from 102,945 patients (including 4068 deaths) were included in the analysis. The highest performing ANN (AUROC 0.871, 95% CI 0.862-0.880; AUPRC 0.372, 95% CI 0.345-0.396) that used normalized data performed better than PIM-2 (AUROC 0.805, 95% CI 0.801-0.816; AUPRC 0.234, 95% CI 0.213-0.255) and PRISM-III (AUROC 0.844, 95% CI 0.841-0.855; AUPRC 0.348, 95% CI 0.322-0.367). The performance of this ANN was also significantly better than that of the logistic regression model (AUROC 0.862, 95% CI 0.852-0.872; AUPRC 0.329, 95% CI 0.304-0.351). The performance of the ANN that used unnormalized data (AUROC 0.865, 95% CI 0.856-0.874) was slightly inferior to our highest performing ANN; the single-layer ANN architecture performed poorly and was not investigated further. CONCLUSIONS: A simple ANN model performed slightly better than the benchmark PIM-2 and PRISM-III scores and a traditional logistic regression model trained on the same data set. The small performance gains achieved by this two-layer ANN model may not offer clinically significant improvement; however, further research with other or more sophisticated model designs and better imputation of missing data may be warranted.