Don't be misled: Three misconceptions about external validation of clinical prediction models.

Clinical prediction models provide risks of health outcomes that can inform patients and support medical decisions. However, most models never make it to actual implementation in practice. A commonly heard reason for this lack of implementation is that prediction models are often not externally validated. While we generally encourage external validation, we argue that an external validation is often neither sufficient nor required as an essential step before implementation. As such, any available external validation should not be perceived as a license for model implementation. We clarify this argument by discussing three common misconceptions about external validation. We argue that there is not one type of recommended validation design, not always a necessity for external validation, and sometimes a need for multiple external validations. The insights from this paper can help readers to consider, design, interpret, and appreciate external validation studies.

[When to suspect pulmonary embolism: a balancing act]. / Wanneer moet je aan een longembolie denken?

Pulmonary embolism (PE) is a common disease, which can present with a variety of symptoms. Optimal use of diagnostics is challenging given the tight and delicate balance between underdiagnosis and over-testing or overdiagnosis. Diagnostic delay occurs in a substantial part of patients, and seems more common in those with known cardiopulmonary disease or non-specific signs and symptoms. At the other end of the spectrum, the amount of diagnostic imaging increases. Increased use of diagnostic imaging in general leads to more harmful exposures and might result in overtreatment, as may be the case in subsegmental PE. Correct use of clinical prediction rules reduces the need for diagnostic imaging while PE can still be ruled out safely. This clinical lesson describes three cases of PE and provides an overview of factors that contribute to underdiagnosis or overdiagnosis. We provide recommendations to improve our balancing act for this challenging disease.

External validation of six COVID-19 prognostic models for predicting mortality risk in older populations in a hospital, primary care, and nursing home setting.

OBJECTIVES: To systematically evaluate the performance of COVID-19 prognostic models and scores for mortality risk in older populations across three health-care settings: hospitals, primary care, and nursing homes. STUDY DESIGN AND SETTING: This retrospective external validation study included 14,092 older individuals of ≥70 years of age with a clinical or polymerase chain reaction-confirmed COVID-19 diagnosis from March 2020 to December 2020. The six validation cohorts include three hospital-based (CliniCo, COVID-OLD, COVID-PREDICT), two primary care-based (Julius General Practitioners Network/Academisch network huisartsgeneeskunde/Network of Academic general Practitioners, PHARMO), and one nursing home cohort (YSIS) in the Netherlands. Based on a living systematic review of COVID-19 prediction models using Prediction model Risk Of Bias ASsessment Tool for quality and risk of bias assessment and considering predictor availability in validation cohorts, we selected six prognostic models predicting mortality risk in adults with COVID-19 infection (GAL-COVID-19 mortality, 4C Mortality Score, National Early Warning Score 2-extended model, Xie model, Wang clinical model, and CURB65 score). All six prognostic models were validated in the hospital cohorts and the GAL-COVID-19 mortality model was validated in all three healthcare settings. The primary outcome was in-hospital mortality for hospitals and 28-day mortality for primary care and nursing home settings. Model performance was evaluated in each validation cohort separately in terms of discrimination, calibration, and decision curves. An intercept update was performed in models indicating miscalibration followed by predictive performance re-evaluation. MAIN OUTCOME MEASURE: In-hospital mortality for hospitals and 28-day mortality for primary care and nursing home setting. RESULTS: All six prognostic models performed poorly and showed miscalibration in the older population cohorts. In the hospital settings, model performance ranged from calibration-in-the-large -1.45 to 7.46, calibration slopes 0.24-0.81, and C-statistic 0.55-0.71 with 4C Mortality Score performing as the most discriminative and well-calibrated model. Performance across health-care settings was similar for the GAL-COVID-19 model, with a calibration-in-the-large in the range of -2.35 to -0.15 indicating overestimation, calibration slopes of 0.24-0.81 indicating signs of overfitting, and C-statistic of 0.55-0.71. CONCLUSION: Our results show that most prognostic models for predicting mortality risk performed poorly in the older population with COVID-19, in each health-care setting: hospital, primary care, and nursing home settings. Insights into factors influencing predictive model performance in the older population are needed for pandemic preparedness and reliable prognostication of health-related outcomes in this demographic.

Limited incremental predictive value of the frailty index and other vulnerability measures from routine care data for mortality risk prediction in older patients with COVID-19 in primary care.

BACKGROUND: During the COVID-19 pandemic, older patients in primary care were triaged based on their frailty or assumed vulnerability for poor outcomes, while evidence on the prognostic value of vulnerability measures in COVID-19 patients in primary care was lacking. Still, knowledge on the role of vulnerability is pivotal in understanding the resilience of older people during acute illness, and hence important for future pandemic preparedness. Therefore, we assessed the predictive value of different routine care-based vulnerability measures in addition to age and sex for 28-day mortality in an older primary care population of patients with COVID-19. METHODS: From primary care medical records using three routinely collected Dutch primary care databases, we included all patients aged 70 years or older with a COVID-19 diagnosis registration in 2020 and 2021. All-cause mortality was predicted using logistic regression based on age and sex only (basic model), and separately adding six vulnerability measures: renal function, cognitive impairment, number of chronic drugs, Charlson Comorbidity Index, Chronic Comorbidity Score, and a Frailty Index. Predictive performance of the basic model and the six vulnerability models was compared in terms of area under the receiver operator characteristic curve (AUC), index of prediction accuracy and the distribution of predicted risks. RESULTS: Of the 4,065 included patients, 9% died within 28 days after COVID-19 diagnosis. Predicted mortality risk ranged between 7-26% for the basic model including age and sex, changing to 4-41% by addition of comorbidity-based vulnerability measures (Charlson Comorbidity Index, Chronic Comorbidity Score), more reflecting impaired organ functioning. Similarly, the AUC of the basic model slightly increased from 0.69 (95%CI 0.66 - 0.72) to 0.74 (95%CI 0.71 - 0.76) by addition of either of these comorbidity scores. Addition of a Frailty Index, renal function, the number of chronic drugs or cognitive impairment yielded no substantial change in predictions. CONCLUSION: In our dataset of older COVID-19 patients in primary care, the 28-day mortality fraction was substantial at 9%. Six different vulnerability measures had little incremental predictive value in addition to age and sex in predicting short-term mortality.

Incidence and individual risk prediction of post-COVID-19 cardiovascular disease in the general population: a multivariable prediction model development and validation study.

Aims: Previous studies suggest relatively increased cardiovascular risk after COVID-19 infection. This study assessed incidence and explored individual risk and timing of cardiovascular disease occurring post-COVID-19 in a large primary care database. Methods and results: Data were extracted from the UK's Clinical Practice Research Datalink. Incidence rates within 180 days post-infection were estimated for arterial or venous events, inflammatory heart disease, and new-onset atrial fibrillation or heart failure. Next, multivariable logistic regression models were developed on 220 751 adults with COVID-19 infection before 1 December 2020 using age, sex and traditional cardiovascular risk factors. All models were externally validated in (i) 138 034 vaccinated and (ii) 503 404 unvaccinated adults with a first COVID-19 infection after 1 December 2020. Discriminative performance and calibration were evaluated with internal and external validation. Increased incidence rates were observed up to 60 days after COVID-19 infection for venous and arterial cardiovascular events and new-onset atrial fibrillation, but not for inflammatory heart disease or heart failure, with the highest rate for venous events (13 per 1000 person-years). The best prediction models had c-statistics of 0.90 or higher. However, <5% of adults had a predicted 180-day outcome-specific risk larger than 1%. These rare outcomes complicated calibration. Conclusion: Risks of arterial and venous cardiovascular events and new-onset atrial fibrillation are increased within the first 60 days after COVID-19 infection in the general population. Models' c-statistics suggest high discrimination, but because of the very low absolute risks, they are insufficient to inform individual risk management.

Estimated causal effects of common respiratory infections on cardiovascular risk: a meta-analysis.

OBJECTIVE: Literature supports associations between common respiratory tract infections (RTIs) and risk of cardiovascular diseases, yet the importance of RTIs for cardiovascular risk management remains less understood. This systematic review and meta-analysis aimed to estimate the causal effects of RTIs on occurrence of cardiovascular diseases in the general population. METHODS: MEDLINE and EMBASE were systematically searched up to 4 November 2022. Eligible were all aetiological studies evaluating risk of cardiovascular outcomes after exposure to common RTIs within any follow-up duration. Evidence was pooled using random-effects models if data allowed. The ROBINS-E and GRADE approaches were used to rate risk of bias and certainty of evidence, respectively. All assessments were performed in duplicate. RESULTS: We included 34 studies (65 678 650 individuals). Most studies had a high risk of bias. COVID-19 likely increases relative risk (RR (95% CI)) of myocardial infarction (3.3 (1.0 to 11.0)), stroke (3.5 (1.2 to 10)), pulmonary embolism (24.6 (13.5 to 44.9)) and deep venous thrombosis (7.8 (4.3 to 14.4)) within 30 days after infection (GRADE: moderate) and about twofold within 1 year (GRADE: low to moderate). Other RTIs also likely increase the RR of myocardial infarction (2.9 (95% CI 1.8 to 4.9)) and stroke (2.6 (95% CI 1.1 to 6.4)) within 30 days (GRADE: moderate), and to a lesser extent with longer follow-up. CONCLUSIONS: RTIs likely increase the risk of cardiovascular diseases about 1.5-5 fold within 1 month after infection. RTIs may, therefore, have clinical relevance as target for cardiovascular risk management, especially in high-risk populations. PROSPERO REGISTRATION NUMBER: CRD42023416277.

Distinct clinical phenotypes in paediatric cancer patients with sepsis are associated with different outcomes-an international multicentre retrospective study.

Background: Identifying phenotypes in sepsis patients may enable precision medicine approaches. However, the generalisability of these phenotypes to specific patient populations is unclear. Given that paediatric cancer patients with sepsis have different host response and pathogen profiles and higher mortality rates when compared to non-cancer patients, we determined whether unique, reproducible, and clinically-relevant sepsis phenotypes exist in this specific patient population. Methods: We studied patients with underlying malignancies admitted with sepsis to one of 25 paediatric intensive care units (PICUs) participating in two large, multi-centre, observational cohorts from the European SCOTER study (n = 383 patients; study period between January 1, 2018 and January 1, 2020) and the U.S. Novel Data-Driven Sepsis Phenotypes in Children study (n = 1898 patients; study period between January 1, 2012 and January 1, 2018). We independently used latent class analysis (LCA) in both cohorts to identify phenotypes using demographic, clinical, and laboratory data from the first 24 h of PICU admission. We then tested the association of the phenotypes with clinical outcomes in both cohorts. Findings: LCA identified two distinct phenotypes that were comparable across both cohorts. Phenotype 1 was characterised by lower serum bicarbonate and albumin, markedly increased lactate and hepatic, renal, and coagulation abnormalities when compared to phenotype 2. Patients with phenotype 1 had a higher 90-day mortality (European cohort 29.2% versus 13.4%, U.S. cohort 27.3% versus 11.4%, p < 0.001) and received more vasopressor and renal replacement therapy than patients with phenotype 2. After adjusting for severity of organ dysfunction, haematological cancer, prior stem cell transplantation and age, phenotype 1 was associated with an adjusted OR of death at 90-day of 1.9 (1.04-3.34) in the European cohort and 1.6 (1.2-2.2) in the U.S. cohort. Interpretation: We identified two clinically-relevant sepsis phenotypes in paediatric cancer patients that are reproducible across two international, multicentre cohorts with prognostic implications. These results may guide further research regarding therapeutic approaches for these specific phenotypes. Funding: Part of this study is funded by the Eunice Kennedy Shriver National Institute of Child Health and Human Development.

A study protocol of external validation of eight COVID-19 prognostic models for predicting mortality risk in older populations in a hospital, primary care, and nursing home setting.

BACKGROUND: The COVID-19 pandemic has a large impact worldwide and is known to particularly affect the older population. This paper outlines the protocol for external validation of prognostic models predicting mortality risk after presentation with COVID-19 in the older population. These prognostic models were originally developed in an adult population and will be validated in an older population (≥ 70 years of age) in three healthcare settings: the hospital setting, the primary care setting, and the nursing home setting. METHODS: Based on a living systematic review of COVID-19 prediction models, we identified eight prognostic models predicting the risk of mortality in adults with a COVID-19 infection (five COVID-19 specific models: GAL-COVID-19 mortality, 4C Mortality Score, NEWS2 + model, Xie model, and Wang clinical model and three pre-existing prognostic scores: APACHE-II, CURB65, SOFA). These eight models will be validated in six different cohorts of the Dutch older population (three hospital cohorts, two primary care cohorts, and a nursing home cohort). All prognostic models will be validated in a hospital setting while the GAL-COVID-19 mortality model will be validated in hospital, primary care, and nursing home settings. The study will include individuals ≥ 70 years of age with a highly suspected or PCR-confirmed COVID-19 infection from March 2020 to December 2020 (and up to December 2021 in a sensitivity analysis). The predictive performance will be evaluated in terms of discrimination, calibration, and decision curves for each of the prognostic models in each cohort individually. For prognostic models with indications of miscalibration, an intercept update will be performed after which predictive performance will be re-evaluated. DISCUSSION: Insight into the performance of existing prognostic models in one of the most vulnerable populations clarifies the extent to which tailoring of COVID-19 prognostic models is needed when models are applied to the older population. Such insight will be important for possible future waves of the COVID-19 pandemic or future pandemics.