Recalibrating prognostic models to improve predictions of in-hospital child mortality in resource-limited settings.

BACKGROUND: In an external validation study, model recalibration is suggested once there is evidence of poor model calibration but with acceptable discriminatory abilities. We identified four models, namely RISC-Malawi (Respiratory Index of Severity in Children) developed in Malawi, and three other predictive models developed in Uganda by Lowlaavar et al. (2016). These prognostic models exhibited poor calibration performance in the recent external validation study, hence the need for recalibration. OBJECTIVE: In this study, we aim to recalibrate these models using regression coefficients updating strategy and determine how much their performances improve. METHODS: We used data collected by the Clinical Information Network from paediatric wards of 20 public county referral hospitals. Missing data were multiply imputed using chained equations. Model updating entailed adjustment of the model's calibration performance while the discriminatory ability remained unaltered. We used two strategies to adjust the model: intercept-only and the logistic recalibration method. RESULTS: Eligibility criteria for the RISC-Malawi model were met in 50,669 patients, split into two sets: a model-recalibrating set (n = 30,343) and a test set (n = 20,326). For the Lowlaavar models, 10,782 patients met the eligibility criteria, of whom 6175 were used to recalibrate the models and 4607 were used to test the performance of the adjusted model. The intercept of the recalibrated RISC-Malawi model was 0.12 (95% CI 0.07, 0.17), while the slope of the same model was 1.08 (95% CI 1.03, 1.13). The performance of the recalibrated models on the test set suggested that no model met the threshold of a perfectly calibrated model, which includes a calibration slope of 1 and a calibration-in-the-large/intercept of 0. CONCLUSIONS: Even after model adjustment, the calibration performances of the 4 models did not meet the recommended threshold for perfect calibration. This finding is suggestive of models over/underestimating the predicted risk of in-hospital mortality, potentially harmful clinically. Therefore, researchers may consider other alternatives, such as ensemble techniques to combine these models into a meta-model to improve out-of-sample predictive performance.

Pairwise joint modeling of clustered and high-dimensional outcomes with covariate missingness in pediatric pneumonia care.

Multiple outcomes reflecting different aspects of routine care are a common phenomenon in health care research. A common approach of handling such outcomes is multiple univariate analyses, an approach which does not allow for answering research questions pertaining to joint inference. In this study, we sought to study associations among nine pediatric pneumonia care outcomes spanning assessment, diagnosis and treatment domains of care, while circumventing the computational challenge posed by their clustered and high-dimensional nature and incompletely recorded covariates. We analyzed data from a cluster randomized trial conducted in 12 Kenyan hospitals. There were varying degrees of missingness in the covariates of interest, and these were multiply imputed using latent normal joint modeling. We used the pairwise joint modeling strategy to fit a correlated random effects joint model for the nine outcomes. This entailed fitting 36 bivariate generalized linear mixed models and deriving inference for the joint model using pseudo-likelihood theory. We also analyzed the nine outcomes separately before and after multiple imputation. We observed joint effects of patient-, clinician- and hospital-level factors on pneumonia care indicators before and after multiple imputation of missing covariates. In both pairwise joint modeling and separate univariate analysis methods, enhanced audit and feedback improved documentation and adherence to recommended clinical guidelines over time in six and five pneumonia care indicators, respectively. Additionally, multiple imputation improved precision of parameter estimates compared to complete case analysis. The strength and direction of association among pneumonia outcomes varied within and across the three domains of pneumonia care.

Pediatric prognostic models predicting inhospital child mortality in resource-limited settings: An external validation study.

Background and Aims: Prognostic models provide evidence-based predictions and estimates of future outcomes, facilitating decision-making, patient care, and research. A few of these models have been externally validated, leading to uncertain reliability and generalizability. This study aims to externally validate four models to assess their transferability and usefulness in clinical practice. The models include the respiratory index of severity in children (RISC)-Malawi model and three other models by Lowlavaar et al. Methods: The study used data from the Clinical Information Network (CIN) to validate the four models where the primary outcome was in-hospital mortality. 163,329 patients met eligibility criteria. Missing data were imputed, and the logistic function was used to compute predicted risk of in-hospital mortality. Models' discriminatory ability and calibration were determined using area under the curve (AUC), calibration slope, and intercept. Results: The RISC-Malawi model had 50,669 pneumonia patients who met the eligibility criteria, of which the case-fatality ratio was 4406 (8.7%). Its AUC was 0.77 (95% CI: 0.77-0.78), whereas the calibration slope was 1.04 (95% CI: 1.00 -1.06), and calibration intercept was 0.81 (95% CI: 0.77-0.84). Regarding the external validation of Lowlavaar et al. models, 10,782 eligible patients  were included, with an in-hospital mortality rate of 5.3%. The primary model's AUC was 0.75 (95% CI: 0.72-0.77), the calibration slope was 0.78 (95% CI: 0.71-0.84), and the calibration intercept was 0.37 (95% CI: 0.28-0.46). All models markedly underestimated the risk of mortality. Conclusion: All externally validated models exhibited either underestimation or overestimation of the risk as judged from calibration statistics. Hence, applying these models with confidence in settings other than their original development context may not be advisable. Our findings strongly suggest the need for recalibrating these model to enhance their generalizability.

Handling missing data in a composite outcome with partially observed components: simulation study based on clustered paediatric routine data.

Composite scores are useful in providing insights and trends about complex and multidimensional quality of care processes. However, missing data in subcomponents may hinder the overall reliability of a composite measure. In this study, strategies for handling missing data in Paediatric Admission Quality of Care (PAQC) score, an ordinal composite outcome, were explored through a simulation study. Specifically, the implications of the conventional method employed in addressing missing PAQC score subcomponents, consisting of scoring missing PAQC score components with a zero, and a multiple imputation (MI)-based strategy, were assessed. The latent normal joint modelling MI approach was used for the latter. Across simulation scenarios, MI of missing PAQC score elements at item level produced minimally biased estimates compared to the conventional method. Moreover, regression coefficients were more prone to bias compared to standards errors. Magnitude of bias was dependent on the proportion of missingness and the missing data generating mechanism. Therefore, incomplete composite outcome subcomponents should be handled carefully to alleviate potential for biased estimates and misleading inferences. Further research on other strategies of imputing at the component and composite outcome level and imputing compatibly with the substantive model in this setting, is needed.

A Spatial Survival Model for Risk Factors of Under-Five Child Mortality in Kenya.

Child mortality is high in Sub-Saharan Africa compared to other regions in the world. In Kenya, the risk of mortality is assumed to vary from county to county due to diversity in socio-economic and even climatic factors. Recently, the country was split into 47 different administrative regions called counties, and health care was delegated to those county governments, further aggravating the spatial differences in health care from county to county. The goal of this study is to evaluate the effects of spatial variation in under-five mortality in Kenya. Data from the Kenya Demographic Health Survey (KDHS-2014) consisting the newly introduced counties was used to analyze this risk. Using a spatial Cox Proportional Hazard model, an Intrinsic Conditional Autoregressive Model (ICAR) was fitted to account for the spatial variation among the counties in the country while the Cox model was used to model the risk factors associated with the time to death of a child. Inference regarding the risk factors and the spatial variation was made in a Bayesian setup based on the Markov Chain Monte Carlo (MCMC) technique to provide posterior estimates. The paper indicate the spatial disparities that exist in the country regarding child mortality in Kenya. The specific counties have mortality rates that are county-specific, although neighboring counties have similar hazards for death of a child. Counties in the central Kenya region were shown to have the highest hazard of death, while those from the western region had the lowest hazard of death. Demographic factors such as the sex of the child and sex of the household head, as well as social economic factors, such as the level of education, accounted for the most variation when spatial differences were factored in. The spatial Cox proportional hazard frailty model performed better compared to the non-spatial non-frailty model. These findings can help the country to plan health care interventions at a subnational level and guide social and health policies by ensuring that counties with a higher risk of Under Five Child Mortality (U5CM) are considered differently from counties experiencing a lower risk of death.

Methodological rigor of prognostic models for predicting in-hospital paediatric mortality in low- and middle-income countries: a systematic review protocol.

Introduction: In low- and middle-income countries (LMICs) where healthcare resources are often limited, making decisions on appropriate treatment choices is critical in ensuring reduction of paediatric deaths as well as instilling proper utilisation of the already constrained healthcare resources. Well-developed and validated prognostic models can aid in early recognition of potential risks thus contributing to the reduction of mortality rates. The aim of the planned systematic review is to identify and appraise the methodological rigor of multivariable prognostic models predicting in-hospital paediatric mortality in LMIC in order to identify statistical and methodological shortcomings deserving special attention and to identify models for external validation. Methods and analysis: This protocol has followed the guidelines of the Preferred Reporting Items for Systematic Reviews and Meta-Analyses for Protocols. A search of articles will be conducted in MEDLINE, Google Scholar, and CINAHL (via EbscoHost) from inception to 2019 without any language restriction. We will also perform a search in Web of Science to identify additional reports that cite the identified studies. Data will be extracted from relevant articles in accordance with the Cochrane Prognosis Methods' guidance; the CHecklist for critical Appraisal and data extraction for systematic Reviews of prediction Modelling Studies. Methodological quality assessment will be performed based on prespecified domains of the Prediction study Risk of Bias Assessment Tool. Ethics and dissemination: Ethical permission will not be required as this study will use published data. Findings from this review will be shared through publication in peer-reviewed scientific journals and, presented at conferences. It is our hope that this study will contribute to the development of robust multivariable prognostic models predicting in-hospital paediatric mortality in low- and middle-income countries. Registration: PROSPERO ID CRD42018088599; registered on 13 February 2018.

Prognostic models for predicting in-hospital paediatric mortality in resource-limited countries: a systematic review.

OBJECTIVES: To identify and appraise the methodological rigour of multivariable prognostic models predicting in-hospital paediatric mortality in low-income and middle-income countries (LMICs). DESIGN: Systematic review of peer-reviewed journals. DATA SOURCES: MEDLINE, CINAHL, Google Scholar and Web of Science electronic databases since inception to August 2019. ELIGIBILITY CRITERIA: We included model development studies predicting in-hospital paediatric mortality in LMIC. DATA EXTRACTION AND SYNTHESIS: This systematic review followed the Checklist for critical Appraisal and data extraction for systematic Reviews of prediction Modelling Studies framework. The risk of bias assessment was conducted using Prediction model Risk of Bias Assessment Tool (PROBAST). No quantitative summary was conducted due to substantial heterogeneity that was observed after assessing the studies included. RESULTS: Our search strategy identified a total of 4054 unique articles. Among these, 3545 articles were excluded after review of titles and abstracts as they covered non-relevant topics. Full texts of 509 articles were screened for eligibility, of which 15 studies reporting 21 models met the eligibility criteria. Based on the PROBAST tool, risk of bias was assessed in four domains; participant, predictors, outcome and analyses. The domain of statistical analyses was the main area of concern where none of the included models was judged to be of low risk of bias. CONCLUSION: This review identified 21 models predicting in-hospital paediatric mortality in LMIC. However, most reports characterising these models are of poor quality when judged against recent reporting standards due to a high risk of bias. Future studies should adhere to standardised methodological criteria and progress from identifying new risk scores to validating or adapting existing scores. PROSPERO REGISTRATION NUMBER: CRD42018088599.