Don't be misled: Three misconceptions about external validation of clinical prediction models.

Clinical prediction models provide risks of health outcomes that can inform patients and support medical decisions. However, most models never make it to actual implementation in practice. A commonly heard reason for this lack of implementation is that prediction models are often not externally validated. While we generally encourage external validation, we argue that an external validation is often neither sufficient nor required as an essential step before implementation. As such, any available external validation should not be perceived as a license for model implementation. We clarify this argument by discussing three common misconceptions about external validation. We argue that there is not one type of recommended validation design, not always a necessity for external validation, and sometimes a need for multiple external validations. The insights from this paper can help readers to consider, design, interpret, and appreciate external validation studies.

Diabetes and risk of acute coronary syndrome in callers with chest discomfort: Cross-sectional study in out-of-hours primary care.

AIMS: We investigated the differences in prevalence of acute coronary syndrome (ACS) by presence versus absence of diabetes in males and females with chest discomfort who called out-of-hours primary care (OHS-PC). METHODS: A cross-sectional study performed in the Netherlands. Patients who called the OHS-PC in the Utrecht region, the Netherlands between 2014 and 2017 with acute chest discomfort were included. We compared those with diabetes with those without diabetes. Multivariable logistic regression was used to determine the relation between diabetes and (i) high urgency allocation and (ii) ACS. RESULTS: Of the 2,195 callers with acute chest discomfort, 180 (8.2%) reported having diabetes. ACS was present in 15.3% of males (22.0% in those with diabetes) and 8.4% of females (18.8% in those with diabetes). Callers with diabetes did not receive a high urgency more frequently (74.4% vs. 67.8% (OR: 1.38; 95% CI 0.98-1.96). However, such callers had a higher odds for ACS (OR: 2.17; 95% CI 1.47-3.19). These differences were similar for females and males. CONCLUSIONS: Diabetes holds promise as diagnostic factor in callers to OHS-PC with chest discomfort. It might help triage in this setting given the increased risk of ACS in those with diabetes.

It's not all about residual confounding: a plea for QBA for epidemiologic researchers and educators.

We spend a great deal of time on confounding in our teaching, in our methods development and in our assessment of study results. This may give the impression that uncontrolled confounding is the biggest problem that observational epidemiology faces, when in fact, other sources of bias such as selection bias, measurement error, missing data, and misalignment of zero time may often (especially if they are all present in a single study) lead to a stronger deviation from the truth. Compared to the amount of time we spend teaching how to address confounding in a data analysis, we spend relatively little time teaching methods for simulating confounding (and other sources of bias) to learn their impact and develop plans to mitigate or quantify the bias. We review a paper by Desai et al that uses simulation methods to quantify the impact of an unmeasured confounder when it is completely missing or when a proxy of the confounder is measured. We use this article to discuss how we can use simulations of sources of bias to ensure we generate better and more valid study estimates, and we discuss the importance of simulating realistic datasets with plausible bias structures to guide data collection. If an advanced life form exists outside of our current universe and they came to earth with the goal of scouring the published epidemiologic literature to understand what the biggest problem epidemiologists have, they would quickly discover that the limitations section of publications would provide them with all the information they needed. And most likely what they would conclude is that the biggest problem that we face is uncontrolled confounding. It seems to be an obsession of ours.

Predictive factors of clot propagation in patients with superficial venous thrombosis towards deep venous thrombosis and pulmonary embolism: a systematic review and meta-analysis.

OBJECTIVE: A subset of patients with superficial venous thrombosis (SVT) experiences clot propagation towards deep venous thrombosis (DVT) and/or pulmonary embolism (PE). The aim of this systematic review is to identify all clinically relevant cross-sectional and prognostic factors for predicting thrombotic complications in patients with SVT. DESIGN: Systematic review. DATA SOURCES: PubMed/MEDLINE and Embase were systematically searched until 3 March 2023. ELIGIBILITY CRITERIA: Original research studies with patients with SVT, DVT and/or PE as the outcome and presenting cross-sectional or prognostic predictive factors. DATA EXTRACTION AND SYNTHESIS OF RESULTS: The CHecklist for critical Appraisal and data extraction for systematic Reviews of prediction Modelling (CHARMS) checklist for prognostic factor studies was used for systematic extraction of study characteristics. Per identified predictive factor, relevant estimates of univariable and multivariable predictor-outcome associations were extracted, such as ORs and HRs. Estimates of association for the most frequently reported predictors were summarised in forest plots, and meta-analyses with heterogeneity were presented. The Quality in Prognosis Studies (QUIPS) tool was used for risk of bias assessment and Grading of Recommendations, Assessment, Development and Evaluations (GRADE) for assessing the certainty of evidence. RESULTS: Twenty-two studies were included (n=10 111 patients). The most reported predictive factors were high age, male sex, history of venous thromboembolism (VTE), absence of varicose veins and cancer. Pooled effect estimates were heterogenous and ranged from OR 3.12 (95% CI 1.75 to 5.59) for the cross-sectional predictor cancer to OR 0.92 (95% CI 0.56 to 1.53) for the prognostic predictor high age. The level of evidence was rated very low to low. Most studies were scored high or moderate risk of bias. CONCLUSIONS: Although the pooled estimates of the predictors high age, male sex, history of VTE, cancer and absence of varicose veins showed predictive potential in isolation, variability in study designs, lack of multivariable adjustment and high risk of bias prevent firm conclusions. High-quality, multivariable studies are necessary to be able to identify individual SVT risk profiles. PROSPERO REGISTRATION NUMBER: CRD42021262819.

SPIN-PM: a consensus framework to evaluate the presence of spin in studies on prediction models.

OBJECTIVES: To develop a framework to identify and evaluate spin practices and its facilitators in studies on clinical prediction model regardless of the modeling technique. STUDY DESIGN AND SETTING: We followed a three-phase consensus process: (1) premeeting literature review to generate items to be included; (2) a series of structured meetings to provide comments discussed and exchanged viewpoints on items to be included with a panel of experienced researchers; and (3) postmeeting review on final list of items and examples to be included. Through this iterative consensus process, a framework was derived after all panel's researchers agreed. RESULTS: This consensus process involved a panel of eight researchers and resulted in SPIN-Prediction Models which consists of two categories of spin (misleading interpretation and misleading transportability), and within these categories, two forms of spin (spin practices and facilitators of spin). We provide criteria and examples. CONCLUSION: We proposed this guidance aiming to facilitate not only the accurate reporting but also an accurate interpretation and extrapolation of clinical prediction models which will likely improve the reporting quality of subsequent research, as well as reduce research waste.

Accuracy of urgency allocation in patients with shortness of breath calling out-of-hours primary care: a cross-sectional study.

BACKGROUND: In out-of-hours primary care (OHS-PC), semi-automatic decision support tools are often used during telephone triage. In the Netherlands, the Netherlands Triage Standard (NTS) is used. The NTS is mainly expert-based and evidence on the diagnostic accuracy of the NTS' urgency allocation against clinically relevant outcomes for patients calling with shortness of breath (SOB) is lacking. METHODS: We included data from adults (≥18 years) who contacted two large Dutch OHS-PC centres for SOB between 1 September 2020 and 31 August 2021 and whose follow-up data about final diagnosis could be retrieved from their own general practitioner (GP). The diagnostic accuracy (sensitivity and specificity with corresponding 95% confidence intervals (CI)) of the NTS' urgency levels (high (U1/U2) versus low (U3/U4/U5) and 'final' urgency levels (including overruling of the urgency by triage nurses or supervising general practitioners (GPs)) was determined with life-threatening events (LTEs) as the reference. LTEs included, amongst others, acute coronary syndrome, pulmonary embolism, acute heart failure and severe pneumonia. RESULTS: Out of 2012 eligible triage calls, we could include 1833 adults with SOB who called the OHS-PC, mean age 53.3 (SD 21.5) years, 55.5% female, and 16.6% showed to have had a LTE. Most often severe COVID-19 infection (6.0%), acute heart failure (2.6%), severe COPD exacerbation (2.1%) or severe pneumonia (1.9%). The NTS urgency level had a sensitivity of 0.56 (95% CI 0.50-0.61) and specificity of 0.61 (95% CI 0.58-0.63). Overruling of the NTS' urgency allocation by triage nurses and/or supervising GPs did not impact sensitivity (0.56 vs. 0.54, p = 0.458) but slightly improved specificity (0.61 vs. 0.65, p < 0.001). CONCLUSIONS: The semi-automatic decision support tool NTS performs poorly with respect to safety (sensitivity) and efficiency (specificity) of urgency allocation in adults calling Dutch OHS-PC with SOB. There is room for improvement of telephone triage in patients calling OHS-PC with SOB. TRIAL REGISTRATION: The Netherlands Trial Register, number: NL9682 .

Prognostic factors and prediction models for hospitalisation and all-cause mortality in adults presenting to primary care with a lower respiratory tract infection: a systematic review.

OBJECTIVE: To identify and synthesise relevant existing prognostic factors (PF) and prediction models (PM) for hospitalisation and all-cause mortality within 90 days in primary care patients with acute lower respiratory tract infections (LRTI). DESIGN: Systematic review. METHODS: Systematic searches of MEDLINE, Embase and the Cochrane Library were performed. All PF and PM studies on the risk of hospitalisation or all-cause mortality within 90 days in adult primary care LRTI patients were included. The risk of bias was assessed using the Quality in Prognostic Studies tool and Prediction Model Risk Of Bias Assessment Tool tools for PF and PM studies, respectively. The results of included PF and PM studies were descriptively summarised. RESULTS: Of 2799 unique records identified, 16 were included: 9 PF studies, 6 PM studies and 1 combination of both. The risk of bias was judged high for all studies, mainly due to limitations in the analysis domain. Based on reported multivariable associations in PF studies, increasing age, sex, current smoking, diabetes, a history of stroke, cancer or heart failure, previous hospitalisation, influenza vaccination (negative association), current use of systemic corticosteroids, recent antibiotic use, respiratory rate ≥25/min and diagnosis of pneumonia were identified as most promising candidate predictors. One newly developed PM was externally validated (c statistic 0.74, 95% CI 0.71 to 0.78) whereas the previously hospital-derived CRB-65 was externally validated in primary care in five studies (c statistic ranging from 0.72 (95% CI 0.63 to 0.81) to 0.79 (95% CI 0.65 to 0.92)). None of the PM studies reported measures of model calibration. CONCLUSIONS: Implementation of existing models for individualised risk prediction of 90-day hospitalisation or mortality in primary care LRTI patients in everyday practice is hampered by incomplete assessment of model performance. The identified candidate predictors provide useful information for clinicians and warrant consideration when developing or updating PMs using state-of-the-art development and validation techniques. PROSPERO REGISTRATION NUMBER: CRD42022341233.

External validation of six COVID-19 prognostic models for predicting mortality risk in older populations in a hospital, primary care, and nursing home setting.

OBJECTIVES: To systematically evaluate the performance of COVID-19 prognostic models and scores for mortality risk in older populations across three health-care settings: hospitals, primary care, and nursing homes. STUDY DESIGN AND SETTING: This retrospective external validation study included 14,092 older individuals of ≥70 years of age with a clinical or polymerase chain reaction-confirmed COVID-19 diagnosis from March 2020 to December 2020. The six validation cohorts include three hospital-based (CliniCo, COVID-OLD, COVID-PREDICT), two primary care-based (Julius General Practitioners Network/Academisch network huisartsgeneeskunde/Network of Academic general Practitioners, PHARMO), and one nursing home cohort (YSIS) in the Netherlands. Based on a living systematic review of COVID-19 prediction models using Prediction model Risk Of Bias ASsessment Tool for quality and risk of bias assessment and considering predictor availability in validation cohorts, we selected six prognostic models predicting mortality risk in adults with COVID-19 infection (GAL-COVID-19 mortality, 4C Mortality Score, National Early Warning Score 2-extended model, Xie model, Wang clinical model, and CURB65 score). All six prognostic models were validated in the hospital cohorts and the GAL-COVID-19 mortality model was validated in all three healthcare settings. The primary outcome was in-hospital mortality for hospitals and 28-day mortality for primary care and nursing home settings. Model performance was evaluated in each validation cohort separately in terms of discrimination, calibration, and decision curves. An intercept update was performed in models indicating miscalibration followed by predictive performance re-evaluation. MAIN OUTCOME MEASURE: In-hospital mortality for hospitals and 28-day mortality for primary care and nursing home setting. RESULTS: All six prognostic models performed poorly and showed miscalibration in the older population cohorts. In the hospital settings, model performance ranged from calibration-in-the-large -1.45 to 7.46, calibration slopes 0.24-0.81, and C-statistic 0.55-0.71 with 4C Mortality Score performing as the most discriminative and well-calibrated model. Performance across health-care settings was similar for the GAL-COVID-19 model, with a calibration-in-the-large in the range of -2.35 to -0.15 indicating overestimation, calibration slopes of 0.24-0.81 indicating signs of overfitting, and C-statistic of 0.55-0.71. CONCLUSION: Our results show that most prognostic models for predicting mortality risk performed poorly in the older population with COVID-19, in each health-care setting: hospital, primary care, and nursing home settings. Insights into factors influencing predictive model performance in the older population are needed for pandemic preparedness and reliable prognostication of health-related outcomes in this demographic.

Limited incremental predictive value of the frailty index and other vulnerability measures from routine care data for mortality risk prediction in older patients with COVID-19 in primary care.

BACKGROUND: During the COVID-19 pandemic, older patients in primary care were triaged based on their frailty or assumed vulnerability for poor outcomes, while evidence on the prognostic value of vulnerability measures in COVID-19 patients in primary care was lacking. Still, knowledge on the role of vulnerability is pivotal in understanding the resilience of older people during acute illness, and hence important for future pandemic preparedness. Therefore, we assessed the predictive value of different routine care-based vulnerability measures in addition to age and sex for 28-day mortality in an older primary care population of patients with COVID-19. METHODS: From primary care medical records using three routinely collected Dutch primary care databases, we included all patients aged 70 years or older with a COVID-19 diagnosis registration in 2020 and 2021. All-cause mortality was predicted using logistic regression based on age and sex only (basic model), and separately adding six vulnerability measures: renal function, cognitive impairment, number of chronic drugs, Charlson Comorbidity Index, Chronic Comorbidity Score, and a Frailty Index. Predictive performance of the basic model and the six vulnerability models was compared in terms of area under the receiver operator characteristic curve (AUC), index of prediction accuracy and the distribution of predicted risks. RESULTS: Of the 4,065 included patients, 9% died within 28 days after COVID-19 diagnosis. Predicted mortality risk ranged between 7-26% for the basic model including age and sex, changing to 4-41% by addition of comorbidity-based vulnerability measures (Charlson Comorbidity Index, Chronic Comorbidity Score), more reflecting impaired organ functioning. Similarly, the AUC of the basic model slightly increased from 0.69 (95%CI 0.66 - 0.72) to 0.74 (95%CI 0.71 - 0.76) by addition of either of these comorbidity scores. Addition of a Frailty Index, renal function, the number of chronic drugs or cognitive impairment yielded no substantial change in predictions. CONCLUSION: In our dataset of older COVID-19 patients in primary care, the 28-day mortality fraction was substantial at 9%. Six different vulnerability measures had little incremental predictive value in addition to age and sex in predicting short-term mortality.

A comparison of hyperparameter tuning procedures for clinical prediction models: A simulation study.

Tuning hyperparameters, such as the regularization parameter in Ridge or Lasso regression, is often aimed at improving the predictive performance of risk prediction models. In this study, various hyperparameter tuning procedures for clinical prediction models were systematically compared and evaluated in low-dimensional data. The focus was on out-of-sample predictive performance (discrimination, calibration, and overall prediction error) of risk prediction models developed using Ridge, Lasso, Elastic Net, or Random Forest. The influence of sample size, number of predictors and events fraction on performance of the hyperparameter tuning procedures was studied using extensive simulations. The results indicate important differences between tuning procedures in calibration performance, while generally showing similar discriminative performance. The one-standard-error rule for tuning applied to cross-validation (1SE CV) often resulted in severe miscalibration. Standard non-repeated and repeated cross-validation (both 5-fold and 10-fold) performed similarly well and outperformed the other tuning procedures. Bootstrap showed a slight tendency to more severe miscalibration than standard cross-validation-based tuning procedures. Differences between tuning procedures were larger for smaller sample sizes, lower events fractions and fewer predictors. These results imply that the choice of tuning procedure can have a profound influence on the predictive performance of prediction models. The results support the application of standard 5-fold or 10-fold cross-validation that minimizes out-of-sample prediction error. Despite an increased computational burden, we found no clear benefit of repeated over non-repeated cross-validation for hyperparameter tuning. We warn against the potentially detrimental effects on model calibration of the popular 1SE CV rule for tuning prediction models in low-dimensional settings.

Cross-institution natural language processing for reliable clinical association studies: a methodological exploration.

OBJECTIVES: Natural language processing (NLP) of clinical notes in electronic medical records is increasingly used to extract otherwise sparsely available patient characteristics, to assess their association with relevant health outcomes. Manual data curation is resource intensive and NLP methods make these studies more feasible. However, the methodology of using NLP methods reliably in clinical research is understudied. The objective of this study is to investigate how NLP models could be used to extract study variables (specifically exposures) to reliably conduct exposure-outcome association studies. STUDY DESIGN AND SETTING: In a convenience sample of patients admitted to the intensive care unit of a US academic health system, multiple association studies are conducted, comparing the association estimates based on NLP-extracted vs. manually extracted exposure variables. The association studies varied in NLP model architecture (Bidirectional Encoder Decoder from Transformers, Long Short-Term Memory), training paradigm (training a new model, fine-tuning an existing external model), extracted exposures (employment status, living status, and substance use), health outcomes (having a do-not-resuscitate/intubate code, length of stay, and in-hospital mortality), missing data handling (multiple imputation vs. complete case analysis), and the application of measurement error correction (via regression calibration). RESULTS: The study was conducted on 1,174 participants (median [interquartile range] age, 61 [50, 73] years; 60.6% male). Additionally, up to 500 discharge reports of participants from the same health system and 2,528 reports of participants from an external health system were used to train the NLP models. Substantial differences were found between the associations based on NLP-extracted and manually extracted exposures under all settings. The error in association was only weakly correlated with the overall F1 score of the NLP models. CONCLUSION: Associations estimated using NLP-extracted exposures should be interpreted with caution. Further research is needed to set conditions for reliable use of NLP in medical association studies.

Incomplete and possibly selective recording of signs, symptoms, and measurements in free text fields of primary care electronic health records of adults with lower respiratory tract infections.

OBJECTIVES: To assess the completeness of recording of relevant signs, symptoms, and measurements in Dutch free text fields of primary care electronic health records (EHR) of adults with lower respiratory tract infections (LRTI). STUDY DESIGN AND SETTING: Retrospective cohort study embedded in a prediction modeling project using routine health care data of the Julius General Practitioners' Network of adult patients with LRTI. Free text fields of 1,000 primary care consultations of LRTI episodes between 2016 and 2019 were manually annotated to retrieve data on the recording of sixteen relevant signs, symptoms, and measurements. RESULTS: For 12/16 (75%) of the relevant signs, symptoms, and measurements, more than 50% of the values was not recorded. The patterns of recorded values indicated selective recording of positive or abnormal values. Recording rates varied across consultation type (physical consultation vs. home visit), diagnosis (acute bronchitis vs. pneumonia), antibiotic prescription issued (yes vs. no), and between practices. CONCLUSION: In EHR of primary care LRTI patients, recording of signs, symptoms, and measurements in free text fields is incomplete and possibly selective. When using free text data in EHR-based research, careful consideration of its recording patterns and appropriate missing data handling techniques is therefore required.

Comparison of likelihood penalization and variance decomposition approaches for clinical prediction models: A simulation study.

Logistic regression is one of the most commonly used approaches to develop clinical risk prediction models. Developers of such models often rely on approaches that aim to minimize the risk of overfitting and improve predictive performance of the logistic model, such as through likelihood penalization and variance decomposition techniques. We present an extensive simulation study that compares the out-of-sample predictive performance of risk prediction models derived using the elastic net, with Lasso and ridge as special cases, and variance decomposition techniques, namely, incomplete principal component regression and incomplete partial least squares regression. We varied the expected events per variable, event fraction, number of candidate predictors, presence of noise predictors, and the presence of sparse predictors in a full-factorial design. Predictive performance was compared on measures of discrimination, calibration, and prediction error. Simulation metamodels were derived to explain the performance differences within model derivation approaches. Our results indicate that, on average, prediction models developed using penalization and variance decomposition approaches outperform models developed using ordinary maximum likelihood estimation, with penalization approaches being consistently superior over the variance decomposition approaches. Differences in performance were most pronounced on the calibration of the model. Performance differences regarding prediction error and concordance statistic outcomes were often small between approaches. The use of likelihood penalization and variance decomposition techniques methods was illustrated in the context of peripheral arterial disease.

Shortness of breath as a diagnostic factor for acute coronary syndrome in male and female callers to out-of-hours primary care.

OBJECTIVE: Chest discomfort and shortness of breath (SOB) are key symptoms in patients with acute coronary syndrome (ACS). It is, however, unknown whether SOB is valuable for recognising ACS during telephone triage in the out-of-hours primary care (OHS-PC) setting. METHODS: A cross-sectional study performed in the Netherlands. Telephone triage conversations were analysed of callers with chest discomfort who contacted the OHS-PC between 2014 and 2017, comparing patients with SOB with those who did not report SOB. We determine the relation between SOB and (1) High urgency allocation, (2) ACS and (3) ACS or other life-threatening diseases. RESULTS: Of the 2195 callers with chest discomfort, 1096 (49.9%) reported SOB (43.7% men, 56.3% women). In total, 15.3% men (13.2% in those with SOB) and 8.4% women (9.2% in those with SOB) appeared to have ACS. SOB compared with no SOB was associated with high urgency allocation (75.9% vs 60.8%, OR: 2.03; 95% CI 1.69 to 2.44, multivariable OR (mOR): 2.03; 95% CI 1.69 to 2.44), but not with ACS (10.9% vs 12.0%; OR: 0.90; 95% CI 0.69 to 1.17, mOR: 0.91; 95% CI 0.70 to 1.19) or 'ACS or other life-threatening diseases' (15.0% vs 14.1%; OR: 1.07; 95% CI 0.85 to 1.36, mOR: 1.09; 95% CI 0.86 to 1.38). For women the relation with ACS was 9.2% vs 7.5%, OR: 1.25; 95% CI 0.83 to 1.88, and for men 13.2% vs 17.4%, OR: 0.72; 95% CI 0.51 to 1.02. For 'ACS or other life-threatening diseases', this was 13.0% vs 8.5%, OR: 1.60; 95% CI 1.10 to 2.32 for women, and 7.5% vs 20.8%, OR: 0.81; 95% CI 0.59 to 1.12 for men. CONCLUSIONS: Men and women with chest discomfort and SOB who contact the OHS-PC more often receive high urgency than those without SOB. This seems to be adequate in women, but not in men when considering the risk of ACS or other life-threatening diseases.

Predicting adverse outcomes in adults with a community-acquired lower respiratory tract infection: a protocol for the development and validation of two prediction models for (i) all-cause hospitalisation and mortality and (ii) cardiovascular outcomes.

BACKGROUND: Community-acquired lower respiratory tract infections (LRTI) are common in primary care and patients at particular risk of adverse outcomes, e.g., hospitalisation and mortality, are challenging to identify. LRTIs are also linked to an increased incidence of cardiovascular diseases (CVD) following the initial infection, whereas concurrent CVD might negatively impact overall prognosis in LRTI patients. Accurate risk prediction of adverse outcomes in LRTI patients, while considering the interplay with CVD, can aid general practitioners (GP) in the clinical decision-making process, and may allow for early detection of deterioration. This paper therefore presents the design of the development and external validation of two models for predicting individual risk of all-cause hospitalisation or mortality (model 1) and short-term incidence of CVD (model 2) in adults presenting to primary care with LRTI. METHODS: Both models will be developed using linked routine electronic health records (EHR) data from Dutch primary and secondary care, and the mortality registry. Adults aged ≥ 40 years with a GP-diagnosis of LRTI between 2016 and 2019 are eligible for inclusion. Relevant patient demographics, medical history, medication use, presenting signs and symptoms, and vital and laboratory measurements will be considered as candidate predictors. Outcomes of interest include 30-day all-cause hospitalisation or mortality (model 1) and 90-day CVD (model 2). Multivariable elastic net regression techniques will be used for model development. During the modelling process, the incremental predictive value of CVD for hospitalisation or all-cause mortality (model 1) will also be assessed. The models will be validated through internal-external cross-validation and external validation in an equivalent cohort of primary care LRTI patients. DISCUSSION: Implementation of currently available prediction models for primary care LRTI patients is hampered by limited assessment of model performance. While considering the role of CVD in LRTI prognosis, we aim to develop and externally validate two models that predict clinically relevant outcomes to aid GPs in clinical decision-making. Challenges that we anticipate include the possibility of low event rates and common problems related to the use of EHR data, such as candidate predictor measurement and missingness, how best to retrieve information from free text fields, and potential misclassification of outcome events.

Incidence and individual risk prediction of post-COVID-19 cardiovascular disease in the general population: a multivariable prediction model development and validation study.

Aims: Previous studies suggest relatively increased cardiovascular risk after COVID-19 infection. This study assessed incidence and explored individual risk and timing of cardiovascular disease occurring post-COVID-19 in a large primary care database. Methods and results: Data were extracted from the UK's Clinical Practice Research Datalink. Incidence rates within 180 days post-infection were estimated for arterial or venous events, inflammatory heart disease, and new-onset atrial fibrillation or heart failure. Next, multivariable logistic regression models were developed on 220 751 adults with COVID-19 infection before 1 December 2020 using age, sex and traditional cardiovascular risk factors. All models were externally validated in (i) 138 034 vaccinated and (ii) 503 404 unvaccinated adults with a first COVID-19 infection after 1 December 2020. Discriminative performance and calibration were evaluated with internal and external validation. Increased incidence rates were observed up to 60 days after COVID-19 infection for venous and arterial cardiovascular events and new-onset atrial fibrillation, but not for inflammatory heart disease or heart failure, with the highest rate for venous events (13 per 1000 person-years). The best prediction models had c-statistics of 0.90 or higher. However, <5% of adults had a predicted 180-day outcome-specific risk larger than 1%. These rare outcomes complicated calibration. Conclusion: Risks of arterial and venous cardiovascular events and new-onset atrial fibrillation are increased within the first 60 days after COVID-19 infection in the general population. Models' c-statistics suggest high discrimination, but because of the very low absolute risks, they are insufficient to inform individual risk management.