Prognostic Models in Nephrology: Where Do We Stand and Where Do We Go from Here? Mapping Out the Evidence in a Scoping Review.

Prognostic models can strongly support individualized care provision and well-informed shared decision making. There has been an upsurge of prognostic research in the field of nephrology, but the uptake of prognostic models in clinical practice remains limited. Therefore, we map out the research field of prognostic models for kidney patients and provide directions on how to proceed from here. We performed a scoping review of studies developing, validating, or updating a prognostic model for patients with CKD. We searched all published models in PubMed and Embase and report predicted outcomes, methodological quality, and validation and/or updating efforts. We found 602 studies, of which 30.1% concerned CKD populations, 31.6% dialysis populations, and 38.4% kidney transplantation populations. The most frequently predicted outcomes were mortality ( n =129), kidney disease progression ( n =75), and kidney graft survival ( n =54). Most studies provided discrimination measures (80.4%), but much less showed calibration results (43.4%). Of the 415 development studies, 28.0% did not perform any validation and 57.6% performed only internal validation. Moreover, only 111 models (26.7%) were externally validated either in the development study itself or in an independent external validation study. Finally, in 45.8% of development studies no useable version of the model was reported. To conclude, many prognostic models have been developed for patients with CKD, mainly for outcomes related to kidney disease progression and patient/graft survival. To bridge the gap between prediction research and kidney patient care, patient-reported outcomes, methodological rigor, complete reporting of prognostic models, external validation, updating, and impact assessment urgently need more attention.

Clinical impact of the Kidney Failure Risk Equation for vascular access planning.

BACKGROUND: Risk-based thresholds for arteriovenous (AV) access creation has been proposed to aid vascular access planning. We aimed to assess the clinical impact of implementing the kidney failure risk equation (KFRE) for vascular access referral. METHODS: 16,102 nephrology-referred chronic kidney disease (CKD) patients from the Swedish Renal Registry 2008-2018 were included. The KFRE was calculated repeatedly, and the timing was identified for when the KFRE risk exceeded several pre-defined thresholds and/or the estimated glomerular filtration rate <15 ml/min/1.73m2 (eGFR15). To assess the utility of the KFRE/eGFR thresholds, cumulative incidence curves of kidney replacement therapy (KRT) or death, and decision-curve analyses were computed at 6, 12 months, and 2 years. The potential impact of using the different thresholds was illustrated by an example from the Swedish access registry. RESULTS: The 12-month specificity for KRT initiation was highest for KFRE>50% 94.5 (95% Confidence interval [CI] 94.3-94.7), followed by KFRE>40% 90.0 (95% CI 89.7-90.3), while sensitivity was highest for KFRE>30% 79.3 (95% CI 78.2-80.3) and eGFR<15 ml/min/1.73m2 81.2 (95% CI 80.2-82.2). The 2-year positive predictive value was 71.5 (95% CI 70.2-72.8), 61.7 (95% CI 60.4-63.0) and 47.2 (95% CI 46.1-48.3) for KFRE>50%, KFRE>40%, and eGFR<15 respectively. Decision curve analyses suggested the largest net benefit for KFRE>40% over two years and KFRE>50% over 12 months when it is important to avoid the harm of possibly unnecessary surgery. In Sweden, 54% of nephrology-referred patients started hemodialysis in a central venous catheter (CVC) of which only 5% had AV access surgery >6 months before initiation. 60% of the CVC patients exceeded KFRE>40% a median of 0.8 years (interquartile range 0.4-1.5) before KRT initiation. CONCLUSIONS: The utility of using KFRE>40% and KFRE>50% is higher compared to the more traditionally used eGFR threshold <15 ml/min/1.73m2 for vascular access planning.

Removing race from the CKD-EPI equation and its impact on prognosis in a predominantly White European population.

BACKGROUND: While American nephrology societies recommend using the 2021 Chronic Kidney Disease Epidemiology Collaboration (CKD-EPI) estimated glomerular filtration rate (eGFR) equation without a Black race coefficient, it is unknown how this would impact disease distribution, prognosis and kidney failure risk prediction in predominantly White non-US populations. METHODS: We studied 1.6 million Stockholm adults with serum/plasma creatinine measurements between 2007 and 2019. We calculated changes in eGFR and reclassification across KDIGO GFR categories when changing from the 2009 to 2021 CKD-EPI equation; estimated associations between eGFR and the clinical outcomes kidney failure with replacement therapy (KFRT), (cardiovascular) mortality and major adverse cardiovascular events using Cox regression; and investigated prognostic accuracy (discrimination and calibration) of both equations within the Kidney Failure Risk Equation. RESULTS: Compared with the 2009 equation, the 2021 equation yielded a higher eGFR by a median [interquartile range (IQR)] of 3.9 (2.9-4.8) mL/min/1.73 m2, which was larger at older age and for men. Consequently, 9.9% of the total population and 36.2% of the population with CKD G3a-G5 was reclassified to a higher eGFR category. Reclassified individuals exhibited a lower risk of KFRT, but higher risks of all-cause/cardiovascular death and major adverse cardiovascular events, compared with non-reclassified participants of similar eGFR. eGFR by both equations strongly predicted study outcomes, with equal discrimination and calibration for the Kidney Failure Risk Equation. CONCLUSIONS: Implementing the 2021 CKD-EPI equation in predominantly White European populations would raise eGFR by a modest amount (larger at older age and in men) and shift a major proportion of CKD patients to a higher eGFR category. eGFR by both equations strongly predicted outcomes.

Kidney Failure Prediction Models: A Comprehensive External Validation Study in Patients with Advanced CKD.

BACKGROUND: Various prediction models have been developed to predict the risk of kidney failure in patients with CKD. However, guideline-recommended models have yet to be compared head to head, their validation in patients with advanced CKD is lacking, and most do not account for competing risks. METHODS: To externally validate 11 existing models of kidney failure, taking the competing risk of death into account, we included patients with advanced CKD from two large cohorts: the European Quality Study (EQUAL), an ongoing European prospective, multicenter cohort study of older patients with advanced CKD, and the Swedish Renal Registry (SRR), an ongoing registry of nephrology-referred patients with CKD in Sweden. The outcome of the models was kidney failure (defined as RRT-treated ESKD). We assessed model performance with discrimination and calibration. RESULTS: The study included 1580 patients from EQUAL and 13,489 patients from SRR. The average c statistic over the 11 validated models was 0.74 in EQUAL and 0.80 in SRR, compared with 0.89 in previous validations. Most models with longer prediction horizons overestimated the risk of kidney failure considerably. The 5-year Kidney Failure Risk Equation (KFRE) overpredicted risk by 10%-18%. The four- and eight-variable 2-year KFRE and the 4-year Grams model showed excellent calibration and good discrimination in both cohorts. CONCLUSIONS: Some existing models can accurately predict kidney failure in patients with advanced CKD. KFRE performed well for a shorter time frame (2 years), despite not accounting for competing events. Models predicting over a longer time frame (5 years) overestimated risk because of the competing risk of death. The Grams model, which accounts for the latter, is suitable for longer-term predictions (4 years).

Development and external validation study combining existing models and recent data into an up-to-date prediction model for evaluating kidneys from older deceased donors for transplantation.

With a rising demand for kidney transplantation, reliable pre-transplant assessment of organ quality becomes top priority. In clinical practice, physicians are regularly in doubt whether suboptimal kidney offers from older donors should be accepted. Here, we externally validate existing prediction models in a European population of older deceased donors, and subsequently developed and externally validated an adverse outcome prediction tool. Recipients of kidney grafts from deceased donors 50 years of age and older were included from the Netherlands Organ Transplant Registry (NOTR) and United States organ transplant registry from 2006-2018. The predicted adverse outcome was a composite of graft failure, death or chronic kidney disease stage 4 plus within one year after transplantation, modelled using logistic regression. Discrimination and calibration were assessed in internal, temporal and external validation. Seven existing models were validated with the same cohorts. The NOTR development cohort contained 2510 patients and 823 events. The temporal validation within NOTR had 837 patients and the external validation used 31987 patients in the United States organ transplant registry. Discrimination of our full adverse outcome model was moderate in external validation (C-statistic 0.63), though somewhat better than discrimination of the seven existing prediction models (average C-statistic 0.57). The model's calibration was highly accurate. Thus, since existing adverse outcome kidney graft survival models performed poorly in a population of older deceased donors, novel models were developed and externally validated, with maximum achievable performance in a population of older deceased kidney donors. These models could assist transplant clinicians in deciding whether to accept a kidney from an older donor.

Renal function decline in older men and women with advanced chronic kidney disease-results from the EQUAL study.

INTRODUCTION: Understanding the mechanisms underlying the differences in renal decline between men and women may improve sex-specific clinical monitoring and management. To this end, we aimed to compare the slope of renal function decline in older men and women in chronic kidney disease (CKD) Stages 4 and 5, taking into account informative censoring related to the sex-specific risks of mortality and dialysis initiation. METHODS: The European QUALity Study on treatment in advanced CKD (EQUAL) study is an observational prospective cohort study in Stages 4 and 5 CKD patients ≥65 years not on dialysis. Data on clinical and demographic patient characteristics were collected between April 2012 and December 2018. Estimated glomerular filtration rate (eGFR) was calculated using the CKD Epidemiology Collaboration equation. eGFR trajectory by sex was modelled using linear mixed models, and joint models were applied to deal with informative censoring. RESULTS: We included 7801 eGFR measurements in 1682 patients over a total of 2911 years of follow-up. Renal function declined by 14.0% [95% confidence interval (CI) 12.9-15.1%] on average each year. Renal function declined faster in men (16.2%/year, 95% CI 15.9-17.1%) compared with women (9.6%/year, 95% CI 6.3-12.1%), which remained largely unchanged after accounting for various mediators and for informative censoring due to mortality and dialysis initiation. Diabetes was identified as an important determinant of renal decline specifically in women. CONCLUSION: In conclusion, renal function declines faster in men compared with women, which remained similar after adjustment for mediators and despite a higher risk of informative censoring in men. We demonstrate a disproportional negative impact of diabetes specifically in women.

Prediction or causality? A scoping review of their conflation within current observational research.

Etiological research aims to uncover causal effects, whilst prediction research aims to forecast an outcome with the best accuracy. Causal and prediction research usually require different methods, and yet their findings may get conflated when reported and interpreted. The aim of the current study is to quantify the frequency of conflation between etiological and prediction research, to discuss common underlying mistakes and provide recommendations on how to avoid these. Observational cohort studies published in January 2018 in the top-ranked journals of six distinct medical fields (Cardiology, Clinical Epidemiology, Clinical Neurology, General and Internal Medicine, Nephrology and Surgery) were included for the current scoping review. Data on conflation was extracted through signaling questions. In total, 180 studies were included. Overall, 26% (n = 46) contained conflation between etiology and prediction. The frequency of conflation varied across medical field and journal impact factor. From the causal studies 22% was conflated, mainly due to the selection of covariates based on their ability to predict without taking the causal structure into account. Within prediction studies 38% was conflated, the most frequent reason was a causal interpretation of covariates included in a prediction model. Conflation of etiology and prediction is a common methodological error in observational medical research and more frequent in prediction studies. As this may lead to biased estimations and erroneous conclusions, researchers must be careful when designing, interpreting and disseminating their research to ensure this conflation is avoided.

Appraising prediction research: a guide and meta-review on bias and applicability assessment using the Prediction model Risk Of Bias ASsessment Tool (PROBAST).

Over the past few years, a large number of prediction models have been published, often of poor methodological quality. Seemingly objective and straightforward, prediction models provide a risk estimate for the outcome of interest, usually based on readily available clinical information. Yet, using models of substandard methodological rigour, especially without external validation, may result in incorrect risk estimates and consequently misclassification. To assess and combat bias in prediction research the prediction model risk of bias assessment tool (PROBAST) was published in 2019. This risk of bias (ROB) tool includes four domains and 20 signalling questions highlighting methodological flaws, and provides guidance in assessing the applicability of the model. In this paper, the PROBAST will be discussed, along with an in-depth review of two commonly encountered pitfalls in prediction modelling that may induce bias: overfitting and composite endpoints. We illustrate the prevalence of potential bias in prediction models with a meta-review of 50 systematic reviews that used the PROBAST to appraise their included studies, thus including 1510 different studies on 2104 prediction models. All domains showed an unclear or high ROB; these results were markedly stable over time, highlighting the urgent need for attention on bias in prediction research. This article aims to do just that by providing (1) the clinician with tools to evaluate the (methodological) quality of a clinical prediction model, (2) the researcher working on a review with methods to appraise the included models, and (3) the researcher developing a model with suggestions to improve model quality.

Towards the best kidney failure prediction tool: a systematic review and selection aid.

BACKGROUND: Prediction tools that identify chronic kidney disease (CKD) patients at a high risk of developing kidney failure have the potential for great clinical value, but limited uptake. The aim of the current study is to systematically review all available models predicting kidney failure in CKD patients, organize empirical evidence on their validity and ultimately provide guidance in the interpretation and uptake of these tools. METHODS: PubMed and EMBASE were searched for relevant articles. Titles, abstracts and full-text articles were sequentially screened for inclusion by two independent researchers. Data on study design, model development and performance were extracted. The risk of bias and clinical usefulness were assessed and combined in order to provide recommendations on which models to use. RESULTS: Of 2183 screened studies, a total of 42 studies were included in the current review. Most studies showed high discriminatory capacity and the included predictors had large overlap. Overall, the risk of bias was high. Slightly less than half the studies (48%) presented enough detail for the use of their prediction tool in practice and few models were externally validated. CONCLUSIONS: The current systematic review may be used as a tool to select the most appropriate and robust prognostic model for various settings. Although some models showed great potential, many lacked clinical relevance due to being developed in a prevalent patient population with a wide range of disease severity. Future research efforts should focus on external validation and impact assessment in clinically relevant patient populations.

Prediction meets causal inference: the role of treatment in clinical prediction models.

In this paper we study approaches for dealing with treatment when developing a clinical prediction model. Analogous to the estimand framework recently proposed by the European Medicines Agency for clinical trials, we propose a 'predictimand' framework of different questions that may be of interest when predicting risk in relation to treatment started after baseline. We provide a formal definition of the estimands matching these questions, give examples of settings in which each is useful and discuss appropriate estimators including their assumptions. We illustrate the impact of the predictimand choice in a dataset of patients with end-stage kidney disease. We argue that clearly defining the estimand is equally important in prediction research as in causal inference.

Con: Most clinical risk scores are useless.

While developing prediction models has become quite popular both in nephrology and in medicine in general, most models have not been implemented in clinical practice on a larger scale. This should be no surprise, as the majority of published models has been shown to be poorly reported and often developed using inappropriate methods. The main problems identified relate to either using too few candidate predictors (based on univariable P < 0.05) or too many (for the number of events), resulting in poorly performing prediction models. Guidelines on how to develop and test a prediction model all stress the importance of external validation to test discrimination and calibration in other populations, as prediction models usually perform less well in new subjects. However, external validity has not often been tested for prediction models in renal patients. Moreover, impact studies showing improved clinical outcomes when using a prediction model in routine clinical practice have been reported rarely. By and large, notwithstanding a few notable exceptions like the kidney failure risk equation prediction model, most models have not been validated externally or are at best inadequately reported, preventing them from be used in clinical practice. Therefore, we recommend researchers to spend more energy on validation and assessing the impact of existing models, instead of merely developing more models that will most likely never be used in clinical practice as well.

Prediction versus aetiology: common pitfalls and how to avoid them.

Prediction research is a distinct field of epidemiologic research, which should be clearly separated from aetiological research. Both prediction and aetiology make use of multivariable modelling, but the underlying research aim and interpretation of results are very different. Aetiology aims at uncovering the causal effect of a specific risk factor on an outcome, adjusting for confounding factors that are selected based on pre-existing knowledge of causal relations. In contrast, prediction aims at accurately predicting the risk of an outcome using multiple predictors collectively, where the final prediction model is usually based on statistically significant, but not necessarily causal, associations in the data at hand.In both scientific and clinical practice, however, the two are often confused, resulting in poor-quality publications with limited interpretability and applicability. A major problem is the frequently encountered aetiological interpretation of prediction results, where individual variables in a prediction model are attributed causal meaning. This article stresses the differences in use and interpretation of aetiological and prediction studies, and gives examples of common pitfalls.

Pre-dialysis decline of measured glomerular filtration rate but not serum creatinine-based estimated glomerular filtration rate is a risk factor for mortality on dialysis.

Background: Monitoring of renal function is important in patients with chronic kidney disease progressing towards end-stage renal failure, both for timing the start of renal replacement therapy and for determining the prognosis on dialysis. Thus far, studies on associations between estimated glomerular filtration rate (eGFR) measurements in the pre-dialysis stage and mortality on dialysis have shown no or even inverse relations, which may result from the poor validity of serum creatinine-based estimation equations for renal function in pre-dialysis patients. As decline in renal function may be better reflected by the mean of the measured creatinine and urea clearance based on 24-h urine collections (mGFR by C Cr-U ), we hypothesize that in patients with low kidney function, a fast mGFR decline is a risk factor for mortality on dialysis, in contrast to a fast eGFR decline. Methods: For 197 individuals, included from the multicentre NECOSAD cohort, pre-dialysis annual decline of mGFR and eGFR was estimated with linear regression, and classified according to KDOQI as fast (>4 mL/min/1.73 m 2 /year) or slow (≤4 mL/min/1.73 m 2 /year). Cox regression was used to adjust for potential confounders. Results: Patients with a fast mGFR decline had an increased risk of mortality on dialysis: crude hazard ratio (HR) 1.84 (95% confidence interval: 1.13-2.98), adjusted HR 1.94 (1.11-3.36). In contrast, no association was found between a fast eGFR decline in the pre-dialysis phase and mortality on dialysis: crude HR 1.20 (0.75-1.89), adjusted HR 1.14 (0.67-1.94). Conclusions: This study demonstrates the importance of mGFR decline (by C Cr-U ) as opposed to eGFR decline in patients with low kidney function, and gives incentive for repeated mGFR measurements in patients on pre-dialysis care.

Dealing with an uncertain future: a survey study on what patients with chronic kidney disease actually want to know.

Background: Prognostic uncertainty is a recurring theme among patients with chronic kidney disease (CKD). We developed a survey to explore whether CKD patients want to know more about their future, and if so, which topics they prioritize. In addition, we explored differences between several subgroups. Methods: A survey was constructed and tested in collaboration with the Dutch Kidney Patients Association. The survey consisted of three parts: (i) demographics, (ii) considerations about the future, and (iii) prognostic information. The survey was distributed among CKD patients (all stages) through patient associations and via healthcare professionals in two Dutch hospitals. Descriptive statistics were used to summarize the results. All results were stratified by population, sex, and age. Results: A total of 163 patients (45 CKD, 26 dialysis, and 92 kidney transplantation) participated in the survey. The mean age was 63.9 (SD 12.0) and 48.5% was male. Most patients think about their future with CKD occasionally (56.4%) or often (35.0%). Nearly half of the patients (49.7%) discuss the future with their nephrologist, some (19.6%) do not but would like to, and 20 (15.3%) prefer not to. Most patients (73.6%) want more prognostic information, regardless of it being positive or negative. Key topics to receive prognostic information about were laboratory values, symptoms, and physical well-being. Dialysis patients prioritized mental over physical well-being. CKD patients without kidney replacement therapy (KRT) indicated thinking about, and discussing their future more regularly than KRT patients. Conclusions: Patients with CKD contemplate their future regularly and express interest in receiving prognostic information on a variety of topics. One in five patients currently do not discuss their future with CKD with their nephrologist, despite wanting to do so. These findings underline the need to tailor prognostic information provision to patients' preferences, advocating more attention to this subject both in research and clinical practice.

Considerable Variability Among Transplant Nephrologists in Judging Deceased Donor Kidney Offers.

Introduction: Transplant clinicians may disagree on whether or not to accept a deceased donor kidney offer. We investigated the interobserver variability between transplant nephrologists regarding organ acceptance and whether the use of a prediction model impacted their decisions. Methods: We developed an observational online survey with 6 real-life cases of deceased donor kidneys offered to a waitlisted recipient. Per case, nephrologists were asked to estimate the risk of adverse outcome and whether they would accept the offer for this patient, or for a patient of their own choice, and how certain they felt. These questions were repeated after revealing the risk of adverse outcome, calculated by a validated prediction model. Results: Sixty Dutch nephrologists completed the survey. The intraclass correlation coefficient of their estimated risk of adverse outcome was poor (0.20, 95% confidence interval [CI] 0.08-0.62). Interobserver agreement of the decision on whether or not to accept the kidney offer was also poor (Fleiss kappa 0.13, 95% CI 0.129-0.130). The acceptance rate before and after providing the outcome of the prediction model was significantly influenced in 2 of 6 cases. Acceptance rates varied considerably among transplant centers. Conclusion: In this study, the estimated risk of adverse outcome and subsequent decision to accept a suboptimal donor kidney varied greatly among transplant nephrologists. The use of a prediction model could influence this decision and may enhance nephrologists' certainty about their decision.

Lessons learnt when accounting for competing events in the external validation of time-to-event prognostic models.

BACKGROUND: External validation of prognostic models is necessary to assess the accuracy and generalizability of the model to new patients. If models are validated in a setting in which competing events occur, these competing risks should be accounted for when comparing predicted risks to observed outcomes. METHODS: We discuss existing measures of calibration and discrimination that incorporate competing events for time-to-event models. These methods are illustrated using a clinical-data example concerning the prediction of kidney failure in a population with advanced chronic kidney disease (CKD), using the guideline-recommended Kidney Failure Risk Equation (KFRE). The KFRE was developed using Cox regression in a diverse population of CKD patients and has been proposed for use in patients with advanced CKD in whom death is a frequent competing event. RESULTS: When validating the 5-year KFRE with methods that account for competing events, it becomes apparent that the 5-year KFRE considerably overestimates the real-world risk of kidney failure. The absolute overestimation was 10%age points on average and 29%age points in older high-risk patients. CONCLUSIONS: It is crucial that competing events are accounted for during external validation to provide a more reliable assessment the performance of a model in clinical settings in which competing risks occur.

Predicting Kidney Failure, Cardiovascular Disease and Death in Advanced CKD Patients.

Introduction: Predicting the timing and occurrence of kidney replacement therapy (KRT), cardiovascular events, and death among patients with advanced chronic kidney disease (CKD) is clinically useful and relevant. We aimed to externally validate a recently developed CKD G4+ risk calculator for these outcomes and to assess its potential clinical impact in guiding vascular access placement. Methods: We included 1517 patients from the European Quality (EQUAL) study, a European multicentre prospective cohort study of nephrology-referred advanced CKD patients aged ≥65 years. Model performance was assessed based on discrimination and calibration. Potential clinical utility for timing of referral for vascular access placement was studied with diagnostic measures and decision curve analysis (DCA). Results: The model showed a good discrimination for KRT and "death after KRT," with 2-year concordance (C) statistics of 0.74 and 0.76, respectively. Discrimination for cardiovascular events (2-year C-statistic: 0.70) and overall death (2-year C-statistic: 0.61) was poorer. Calibration was fairly accurate. Decision curves illustrated that using the model to guide vascular access referral would generally lead to less unused arteriovenous fistulas (AVFs) than following estimated glomerular filtration rate (eGFR) thresholds. Conclusion: This study shows moderate to good predictive performance of the model in an older cohort of nephrology-referred patients with advanced CKD. Using the model to guide referral for vascular access placement has potential in combating unnecessary vascular surgeries.

External validation of risk scores to predict in-hospital mortality in patients hospitalized due to coronavirus disease 2019.

BACKGROUND: The coronavirus disease 2019 (COVID-19) presents an urgent threat to global health. Prediction models that accurately estimate mortality risk in hospitalized patients could assist medical staff in treatment and allocating limited resources. AIMS: To externally validate two promising previously published risk scores that predict in-hospital mortality among hospitalized COVID-19 patients. METHODS: Two prospective cohorts were available; a cohort of 1028 patients admitted to one of nine hospitals in Lombardy, Italy (the Lombardy cohort) and a cohort of 432 patients admitted to a hospital in Leiden, the Netherlands (the Leiden cohort). The endpoint was in-hospital mortality. All patients were adult and tested COVID-19 PCR-positive. Model discrimination and calibration were assessed. RESULTS: The C-statistic of the 4C mortality score was good in the Lombardy cohort (0.85, 95CI: 0.82-0.89) and in the Leiden cohort (0.87, 95CI: 0.80-0.94). Model calibration was acceptable in the Lombardy cohort but poor in the Leiden cohort due to the model systematically overpredicting the mortality risk for all patients. The C-statistic of the CURB-65 score was good in the Lombardy cohort (0.80, 95CI: 0.75-0.85) and in the Leiden cohort (0.82, 95CI: 0.76-0.88). The mortality rate in the CURB-65 development cohort was much lower than the mortality rate in the Lombardy cohort. A similar but less pronounced trend was found for patients in the Leiden cohort. CONCLUSION: Although performances did not differ greatly, the 4C mortality score showed the best performance. However, because of quickly changing circumstances, model recalibration may be necessary before using the 4C mortality score.