Prediction of pre-eclampsia-related complications in women with suspected or confirmed pre-eclampsia: development and internal validation of clinical prediction model.

OBJECTIVE: A model that can predict reliably the risk of pre-eclampsia (PE)-related pregnancy complications does not exist. The aim of this study was to develop and validate internally a clinical prediction model to predict the risk of a composite outcome of PE-related maternal and fetal complications within 7, 14 and 30 days of testing in women with suspected or confirmed PE. METHODS: The data for this study were derived from a prospective, multicenter, observational cohort study on women with a singleton pregnancy and suspected or confirmed PE at 20 to < 37 weeks' gestation. For the development of the prediction model, the possible contribution of clinical and standard laboratory variables, as well as the biomarkers soluble fms-like tyrosine kinase-1 (sFlt-1), placental growth factor (PlGF) and their ratio, in the prediction of a composite outcome of PE-related complications, consisting of maternal and fetal adverse events within 7, 14 and 30 days, was explored using multivariable competing-risks regression analysis. The discriminative ability of the model was assessed using the concordance (c-) statistic. A bootstrap validation procedure with 500 replications was used to correct the estimate of the prediction model performance for optimism and to compute a shrinkage factor for the regression coefficients to correct for overfitting. RESULTS: Among 384 women with suspected or confirmed PE, 96 (25%) had an adverse PE-related outcome at any time after hospital admission. Important predictors of adverse PE-related outcome included sFlt-1/PlGF ratio, gestational age at the time of biomarker measurement and protein-to-creatinine ratio as continuous variables. The c-statistics (corrected for optimism) for developing a PE-related complication within 7, 14 and 30 days were 0.89, 0.88 and 0.87, respectively. There was limited overfitting, as indicated by a shrinkage factor of 0.91. CONCLUSIONS: We propose a simple clinical prediction model with good discriminative performance to predict PE-related complications. Determination of its usefulness in clinical practice awaits further investigation and external validation. © 2020 The Authors. Ultrasound in Obstetrics & Gynecology published by John Wiley & Sons Ltd on behalf of International Society of Ultrasound in Obstetrics and Gynecology.

sIgE Ana o 1, 2 and 3 accurately distinguish tolerant from allergic children sensitized to cashew nuts.

BACKGROUND: The double-blind, placebo-controlled food challenge test (DBPCFC) is the gold standard in cashew nut allergy. This test is costly, time consuming and not without side effects. Analysis of IgE reactivity to cashew nut components may reduce the need for food challenge tests. METHODS: In a prospective and multicentre study, children with suspected cashew nut allergy underwent a DBPCFC with cashew nut. Specific IgE to cashew nut and to the components Ana o 1, 2 and 3 were determined. A skin prick test (SPT) with cashew nut extract was performed. The association between the outcome of the food challenge test and specific IgE to Ana o 1, 2 and 3 was assessed with logistic regression analyses, unadjusted and adjusted for other diagnostic variables. Discriminative ability was quantified with a concordance index (c). RESULTS: A total of 173 children (103 boys, 60%) with a median age of 9 years were included. About 79% had a positive challenge test outcome. A steep rise in the risk of a positive challenge was observed for specific IgE to each individual component Ana o 1, 2 and 3 with estimated risks up to approximately 100%. Median values of Ana o 1, 2, 3 were 1.29 kU/l (range 0-100 kU/l), 4.77 kU/l (range 0-100 kU/l) and 8.33 kU/l (range 0-100 kU/l) respectively and varied significantly (p < 0.001). Specific IgE to Ana o 1, 2 and 3 was better distinguished between cashew-allergic and tolerant children (c = 0.87, 0.85 and 0.89, respectively) than specific IgE to cashew nut or SPT (c = 0.76 and 0.83, respectively). CONCLUSION: The major cashew nut allergens Ana o 1, 2 and 3 are each individually predictive for the outcome of food challenge tests in cashew-allergic children.

Occurrence of metachronous basal cell carcinomas: a prognostic model.

BACKGROUND: A third of patients with a first basal cell carcinoma (BCC) will develop subsequent (metachronous) BCCs. OBJECTIVES: To study the prognostic effect of the number of previous BCC diagnosis dates a patient has experienced to derive a prediction model to assess the risk of metachronous BCCs that may inform individualized decision making on surveillance. METHODS: We considered participants of north-western European ancestry from a prospective population-based cohort study (Rotterdam Study). After linkage with the Dutch Pathology Registry, 1077 patients with a first BCC were included. Candidate predictors for metachronous BCCs included patient, lifestyle and tumour characteristics. The prognostic model was developed with Fine and Gray regression analysis to account for competing risk of death. We used bootstrapping to correct for within-patient correlation and statistical optimism in predictive performance. RESULTS: Second to fifth BCCs occurred in 293, 122, 58 and 36 patients, with median follow-up times of 3·0, 2·1, 1·7 and 1·8 years after the previous BCC, respectively. The risk of a new BCC was higher for patients with more metachronous BCCs. Having more than one BCC at diagnosis was another strong predictor of metachronous BCCs. Discriminative ability of the model was reasonable with an optimism-corrected c-index of 0·70 at 3 years. CONCLUSIONS: The number of previous BCC diagnosis dates was a strong prognostic factor and should be considered when predicting the risk of metachronous BCCs. When the number of previous BCC diagnosis dates is combined with other readily available characteristics into a prognostic model, patients at high risk of a new BCC can be identified.

Improving prediction models with new markers: a comparison of updating strategies.

BACKGROUND: New markers hold the promise of improving risk prediction for individual patients. We aimed to compare the performance of different strategies to extend a previously developed prediction model with a new marker. METHODS: Our motivating example was the extension of a risk calculator for prostate cancer with a new marker that was available in a relatively small dataset. Performance of the strategies was also investigated in simulations. Development, marker and test sets with different sample sizes originating from the same underlying population were generated. A prediction model was fitted using logistic regression in the development set, extended using the marker set and validated in the test set. Extension strategies considered were re-estimating individual regression coefficients, updating of predictions using conditional likelihood ratios (LR) and imputation of marker values in the development set and subsequently fitting a model in the combined development and marker sets. Sample sizes considered for the development and marker set were 500 and 100, 500 and 500, and 100 and 500 patients. Discriminative ability of the extended models was quantified using the concordance statistic (c-statistic) and calibration was quantified using the calibration slope. RESULTS: All strategies led to extended models with increased discrimination (c-statistic increase from 0.75 to 0.80 in test sets). Strategies estimating a large number of parameters (re-estimation of all coefficients and updating using conditional LR) led to overfitting (calibration slope below 1). Parsimonious methods, limiting the number of coefficients to be re-estimated, or applying shrinkage after model revision, limited the amount of overfitting. Combining the development and marker set using imputation of missing marker values approach led to consistently good performing models in all scenarios. Similar results were observed in the motivating example. CONCLUSION: When the sample with the new marker information is small, parsimonious methods are required to prevent overfitting of a new prediction model. Combining all data with imputation of missing marker values is an attractive option, even if a relatively large marker data set is available.

Impact of adding therapeutic recommendations to risk assessments from a prediction model for postoperative nausea and vomiting.

BACKGROUND: In a large cluster-randomized trial on the impact of a prediction model, presenting the calculated risk of postoperative nausea and vomiting (PONV) on-screen (assistive approach) increased the administration of risk-dependent PONV prophylaxis by anaesthetists. This change in therapeutic decision-making did not improve the patient outcome; that is, the incidence of PONV. The present study aimed to quantify the effects of adding a specific therapeutic recommendation to the predicted risk (directive approach) on PONV prophylaxis decision-making and the incidence of PONV. METHODS: A prospective before-after study was conducted in 1483 elective surgical inpatients. The before-period included care-as-usual and the after-period included the directive risk-based (intervention) strategy. Risk-dependent effects on the administered number of prophylactic antiemetics and incidence of PONV were analysed by mixed-effects regression analysis. RESULTS: During the intervention period anaesthetists administered 0.5 [95% confidence intervals (CIs): 0.4-0.6] more antiemetics for patients identified as being at greater risk of PONV. This directive approach led to a reduction in PONV [odds ratio (OR): 0.60, 95% CI: 0.43-0.83], with an even greater reduction in PONV in high-risk patients (OR: 0.45, 95% CI: 0.28-0.72). CONCLUSIONS: Anaesthetists administered more prophylactic antiemetics when a directive approach was used for risk-tailored intervention compared with care-as-usual. In contrast to the previously studied assistive approach, the increase in PONV prophylaxis now resulted in a lower PONV incidence, particularly in high-risk patients. When one aims for a truly 'PONV-free hospital', a more liberal use of prophylactic antiemetics must be accepted and lower-risk thresholds should be set for the actionable recommendations.

Failure of introduction of food allergens after negative oral food challenge tests in children.

UNLABELLED: One of the purposes to perform an oral food challenge (FC) test is to avoid unnecessary elimination of food allergens. In case of a negative FC test result, the food can be introduced. It is, however, unknown if patients act according to the outcome of the test. This study evaluates the rate of introduction of peanut, hazelnut, cow's milk or hen's egg allergens after a negative FC test. We investigated the introduction rate of children (0-18 years) with a negative FC test visiting the Department of Allergology, Erasmus Medical Centre Rotterdam from 2008 till 2013 and the factors that influence the rate of introduction. Patients were asked to complete a comprehensive questionnaire about their FC test. In total, 157 (38% girls, mean age during challenge 6.9 years) participated in the study. Of these FC tests, 104 (56%) were followed by a successful introduction, 30 (16%) by a partly introduction (traces or processed foods) and 52 (28%) by a failed introduction. Peanut and hazelnut showed a statistically significant lower successful introduction rate. Age, gender, symptoms during FC test, dietary advice and time period to introduction significantly influenced the rate of introduction. One fourth of the children with failure of introducing foods experienced symptoms during the introduction. CONCLUSION: More than one quarter of all children with a negative FC test result did not introduce the food. The FC test in its current form does not achieve its objective for this group of children.

Comparison of two models predicting IVF success; the effect of time trends on model performance.

STUDY QUESTION: How well does the recently developed UK model predicting the success rate of IVF treatment (the 2011 Nelson model) perform in comparison with a UK model developed in the early 1990s (the Templeton model)? SUMMARY ANSWER: Both models showed similar performance, after correction for the increasing success rate over time of IVF. WHAT IS KNOWN ALREADY: For counselling couples undergoing IVF treatment it is of paramount importance to be able to predict success. Several prediction models for the chance of success after IVF treatment have been developed. So far, the Templeton model has been recommended as the best approach after having been validated in several independent patient data sets. The Nelson model, developed in 2011 and characterized by the largest development sample containing the most recently treated couples, may well perform better. STUDY DESIGN, SIZE, DURATION: We tested both models in couples that were included in a national cohort study carried out in the Netherlands between the beginning of January 2002 and the end of December 2004. PARTICIPANTS/MATERIALS, SETTING, METHODS: We analysed the IVF cycles of Dutch couples with primary infertility (n = 5176). The chance of success was calculated using the two UK models that had been developed using the information collected in the Human Fertilisation and Embryology Authority database. Women were treated in 1991-1994 (Templeton) or 2003-2007 (Nelson). The outcome of success for both UK models is the occurrence of a live birth after IVF but the outcome in the Dutch data is an ongoing pregnancy. In order to make the outcomes compatible, we used a factor to convert the chance of live birth to ongoing pregnancy and use the overall terms 'success or no success after IVF'. The discriminative ability and the calibration of both models were assessed, the latter before and after adjustment for time trends in IVF success rates. MAIN RESULTS AND THE ROLE OF CHANCE: The two models showed a similarly limited degree of discriminative ability on the tested data (area under the receiver operating characteristic curve 0.597 for the Templeton model and 0.590 for the Nelson model). The Templeton model underestimated the success rate (observed 21% versus predicted 14%); the Nelson model overestimated the success rate (observed 21% versus predicted 29%). When the models were adjusted for the changing success rates over time, the calibration of both models considerably improved (Templeton observed 21% versus predicted 20%; Nelson observed 21% versus predicted 24%). LIMITATIONS, REASONS FOR CAUTION: We could only test the models in couples with primary infertility because detailed information on secondary infertile couples was lacking in the Dutch data. This shortcoming may have negatively influenced the performance of the Nelson model. WIDER IMPLICATIONS OF THE FINDINGS: The changes in success rates over time should be taken into account when assessing prediction models for estimating the success rate of IVF treatment. In patients with primary infertility, the choice to use the Templeton or Nelson model is arbitrary.

Internal validation of risk models in clustered data: a comparison of bootstrap schemes.

Internal validity of a risk model can be studied efficiently with bootstrapping to assess possible optimism in model performance. Assumptions of the regular bootstrap are violated when the development data are clustered. We compared alternative resampling schemes in clustered data for the estimation of optimism in model performance. A simulation study was conducted to compare regular resampling on only the patient level with resampling on only the cluster level and with resampling sequentially on both the cluster and patient levels (2-step approach). Optimism for the concordance index and calibration slope was estimated. Resampling of only patients or only clusters showed accurate estimates of optimism in model performance. The 2-step approach overestimated the optimism in model performance. If the number of centers or intraclass correlation coefficient was high, resampling of clusters showed more accurate estimates than resampling of patients. The 3 bootstrap schemes also were applied to empirical data that were clustered. The results presented in this paper support the use of resampling on only the clusters for estimation of optimism in model performance when data are clustered.

Prediction of recurrence of hypertensive disorders of pregnancy between 34 and 37 weeks of gestation: a retrospective cohort study.

OBJECTIVE: To assess the recurrence risk of late-preterm hypertensive disease of pregnancy, and to determine whether potential risk factors are predictive. DESIGN: Retrospective cohort study. SETTING: Three secondary and three tertiary care hospitals in the Netherlands. POPULATION: We identified women with a hypertensive disorder in the index pregnancy and delivery at 34-37 weeks of gestation, between January 2000 and December 2002. METHODS: Data were extracted from medical files and women were approached for additional information on subsequent pregnancies. An adverse outcome was defined as the recurrence of a hypertensive disorder in the next subsequent pregnancy. MAIN OUTCOME MEASURES: Absolute risk of recurrence and a prediction model containing demographic and clinical factors predictive for adverse outcome. RESULTS: We identified 425 women who matched the criteria, of whom 351 could be contacted. Of these women, 189 (54%) had had a subsequent pregnancy. Hypertensive disorders recurred in 96 (51%, 95% CI 43-58%) women, of whom 17 (9%, 95% CI 5-14%) delivered again before 37 weeks of gestation. Chronic hypertension and maternal age were the strongest predictors for recurrence. Women undergoing recurrence had a nine-fold chance of developing chronic hypertension (37% versus 6%, OR 8.7, 95% CI 3.3-23). CONCLUSIONS: Women with hypertensive disorders and late-preterm delivery have a 50% chance of recurrence, but only a 9% chance of recurrence resulting in delivery before 37 weeks of gestation. Women with chronic hypertension are prone to develop recurrence, and women with a recurrence more often developed chronic hypertension.

Prediction of low haemoglobin levels in whole blood donors.

BACKGROUND AND OBJECTIVES: Each year, a relevant proportion of whole blood donors is deferred from donation because of low haemoglobin (Hb) levels. Such temporary deferrals are demoralizing, and donors may never return for a donation. Reliable predictions of Hb levels may guide the decision whether donors can be invited for the next donation. In this study, a prediction model was developed for the risk of low Hb levels. MATERIALS AND METHODS: Individual data from 5191 whole blood donors were analysed; 143 donors had a low Hb level. Eleven candidate predictors were considered in logistic regression models to predict low Hb levels. The performance of the prediction model was studied with the receiver operating characteristic (ROC) curve. Internal validity was assessed with a bootstrap procedure. RESULTS: Strong predictors were sex, seasonality, Hb level measured at the previous visit, difference in Hb levels between the previous two visits, time since the previous visit, deferral at the previous visit, and the total number of whole blood donations in the past 2 years. Internal validation showed an area under the ROC curve of 0·87. CONCLUSION: The developed prediction model provides accurate discrimination between donors with low and appropriate Hb levels. The model predictions may be valuable to determine whether donors can be invited for a next donation, or whether some interventions such as postponement of the invitation are warranted. Potentially, this could decrease the number of donor deferrals for low Hb levels.

Progression to microalbuminuria in type 1 diabetes: development and validation of a prediction rule.

AIMS/HYPOTHESIS: Microalbuminuria is common in type 1 diabetes and is associated with an increased risk of renal and cardiovascular disease. We aimed to develop and validate a clinical prediction rule that estimates the absolute risk of microalbuminuria. METHODS: Data from the European Diabetes Prospective Complications Study (n = 1115) were used to develop the prediction rule (development set). Multivariable logistic regression analysis was used to assess the association between potential predictors and progression to microalbuminuria within 7 years. The performance of the prediction rule was assessed with calibration and discrimination (concordance statistic [c-statistic]) measures. The rule was validated in three other diabetes studies (Pittsburgh Epidemiology of Diabetes Complications [EDC] study, Finnish Diabetic Nephropathy [FinnDiane] study and Coronary Artery Calcification in Type 1 Diabetes [CACTI] study). RESULTS: Of patients in the development set, 13% were microalbuminuric after 7 years. Glycosylated haemoglobin, AER, WHR, BMI and ever smoking were found to be the most important predictors. A high-risk group (n = 87 [8%]) was identified with a risk of progression to microalbuminuria of 32%. Predictions showed reasonable discriminative ability, with c-statistic of 0.71. The rule showed good calibration and discrimination in EDC, FinnDiane and CACTI (c-statistic 0.71, 0.79 and 0.79, respectively). CONCLUSIONS/INTERPRETATION: We developed and validated a clinical prediction rule that uses relatively easily obtainable patient characteristics to predict microalbuminuria in patients with type 1 diabetes. This rule can help clinicians to decide on more frequent check-ups for patients at high risk of microalbuminuria in order to prevent long-term chronic complications.

Polytomous logistic regression analysis could be applied more often in diagnostic research.

OBJECTIVE: Physicians commonly consider the presence of all differential diagnoses simultaneously. Polytomous logistic regression modeling allows for simultaneous estimation of the probability of multiple diagnoses. We discuss and (empirically) illustrate the value of this method for diagnostic research. STUDY DESIGN AND SETTING: We used data from a study on the diagnosis of residual retroperitoneal mass histology in patients presenting with nonseminomatous testicular germ cell tumor. The differential diagnoses include benign tissue, mature teratoma, and viable cancer. Probabilities of each diagnosis were estimated with a polytomous logistic regression model and compared with the probabilities estimated from two consecutive dichotomous logistic regression models. RESULTS: We provide interpretations of the odds ratios derived from the polytomous regression model and present a simple score chart to facilitate calculation of predicted probabilities from the polytomous model. For both modeling methods, we show the calibration plots and receiver operating characteristics curve (ROC) areas comparing each diagnostic outcome category with the other two. The ROC areas for benign tissue, mature teratoma, and viable cancer were similar for both modeling methods, 0.83 (95% confidence interval [CI]=0.80-0.85) vs. 0.83 (95% CI=0.80-0.85), 0.78 (95% CI=0.75-0.81) vs. 0.78 (95% CI=0.75-0.81), and 0.66 (95% CI=0.61-0.71) vs. 0.64 (95% CI=0.59-0.69), for polytomous and dichotomous regression models, respectively. CONCLUSION: Polytomous logistic regression is a useful technique to simultaneously model predicted probabilities of multiple diagnostic outcome categories. The performance of a polytomous prediction model can be assessed similarly to a dichotomous logistic regression model, and predictions by a polytomous model can be made with a user-friendly method. Because the simultaneous consideration of the presence of multiple (differential) conditions serves clinical practice better than consideration of the presence of only one target condition, polytomous logistic regression could be applied more often in diagnostic research.

Updating methods improved the performance of a clinical prediction model in new patients.

OBJECTIVE: Ideally, clinical prediction models are generalizable to other patient groups. Unfortunately, they perform regularly worse when validated in new patients and are then often redeveloped. While the original prediction model usually has been developed on a large data set, redevelopment then often occurs on the smaller validation set. Recently, methods to update existing prediction models with the data of new patients have been proposed. We used an existing model that preoperatively predicts the risk of severe postoperative pain (SPP) to compare five updating methods. STUDY DESIGN AND SETTING: The model was tested and updated with a set of 752 new patients (274 [36] with SPP). We studied the discrimination (ability to distinguish between patients with and without SPP) and calibration (agreement between the predicted risks and observed frequencies of SPP) of the five updated models in 283 other patients (100 [35%] with SPP). RESULTS: Simple recalibration methods improved the calibration to a similar extent as revision methods that made more extensive adjustments to the original model. Discrimination could not be improved by any of the methods. CONCLUSION: When the performance is poor in new patients, updating methods can be applied to adjust the model, rather than to develop a new model.

Does ignoring clustering in multicenter data influence the performance of prediction models? A simulation study.

Clinical risk prediction models are increasingly being developed and validated on multicenter datasets. In this article, we present a comprehensive framework for the evaluation of the predictive performance of prediction models at the center level and the population level, considering population-averaged predictions, center-specific predictions, and predictions assuming an average random center effect. We demonstrated in a simulation study that calibration slopes do not only deviate from one because of over- or underfitting of patterns in the development dataset, but also as a result of the choice of the model (standard versus mixed effects logistic regression), the type of predictions (marginal versus conditional versus assuming an average random effect), and the level of model validation (center versus population). In particular, when data is heavily clustered (ICC 20%), center-specific predictions offer the best predictive performance at the population level and the center level. We recommend that models should reflect the data structure, while the level of model validation should reflect the research question.

No Negative Impact of Palliative Sedation on Relatives' Experience of the Dying Phase and Their Wellbeing after the Patient's Death: An Observational Study.

BACKGROUND: Palliative sedation is the widely-used intervention of administering sedating agents to induce a state of unconsciousness to take away a dying patient's perception of otherwise irrelievable symptoms. However, it remains questionable whether this ethically complex intervention is beneficial for patients and whether the associated lack of communication in the last phase of life has a negative impact on relatives' wellbeing. METHODS: An observational questionnaire study was conducted among relatives of a consecutive sample of patients who died a non-sudden death in the Erasmus MC Cancer Institute or in the hospice 'Laurens Cadenza' (both in Rotterdam) between 2010 and 2013. RESULTS: Relatives filled in questionnaires regarding 151 patients who had been sedated and 90 patients who had not been sedated. The median time since all patients had passed away was 21 (IQR 14-32) months. No significant differences were found in relatives´ assessments of the quality of end-of-life care, patients´ quality of life in the last week before death and their quality of dying, between patients who did and did not receive sedation, or in relatives' satisfaction with their own life, their general health and their mental wellbeing after the patient's death. CONCLUSIONS: The use of sedation in these patients appears to have no negative effect on bereaved relatives' evaluation of the patient's dying phase, or on their own wellbeing after the patient's death.

Prognostic value of age for chronic postoperative inguinal pain.

PURPOSE: Chronic postoperative inguinal pain (CPIP) is considered the most common and serious long-term problem after inguinal hernia repair. Young age has been described as a risk factor for developing chronic pain after several surgical procedures. Our aim was to assess if age has prognostic value on CPIP. METHODS: The database of a randomized trial; the LEVEL trial, 669 patients, TEP versus Lichtenstein, was used for analysis. Data on incidence and intensity of preoperative pain, postoperative pain and CPIP at 1 year were collected. The association of age with incidence and intensity of pain was assessed with regression analysis. Further, hernia type and surgical technique were studied in combination with age and CPIP as possible risk factors on CPIP over age alone. RESULTS: Younger patients (18-40 years) presented more often with CPIP than middle-aged patients (40-60 years) and elderly (>60 years); 43 vs. 29 vs. 19 %; overall 27 %. Younger and middle-aged patients had more frequently preoperative pain; 54 vs. 55 vs. 41 % and intensity of pain was higher during the first three postoperative days (VAS on day 1: 5.5 vs. 4.5 vs. 3.9 and on day 3: 3.8 vs. 2.9 vs. 2.6). Indirect-type hernias were seen more often in younger patients (77 vs. 51 vs. 48 %) and were not related to CPIP or with surgical technique. CONCLUSIONS: Almost one out of three patients experiences CPIP. The younger the patient, the higher the risk of CPIP. Hernia type and surgical technique did not influence CPIP.

A simulation study of sample size demonstrated the importance of the number of events per variable to develop prediction models in clustered data.

OBJECTIVES: This study aims to investigate the influence of the amount of clustering [intraclass correlation (ICC) = 0%, 5%, or 20%], the number of events per variable (EPV) or candidate predictor (EPV = 5, 10, 20, or 50), and backward variable selection on the performance of prediction models. STUDY DESIGN AND SETTING: Researchers frequently combine data from several centers to develop clinical prediction models. In our simulation study, we developed models from clustered training data using multilevel logistic regression and validated them in external data. RESULTS: The amount of clustering was not meaningfully associated with the models' predictive performance. The median calibration slope of models built in samples with EPV = 5 and strong clustering (ICC = 20%) was 0.71. With EPV = 5 and ICC = 0%, it was 0.72. A higher EPV related to an increased performance: the calibration slope was 0.85 at EPV = 10 and ICC = 20% and 0.96 at EPV = 50 and ICC = 20%. Variable selection sometimes led to a substantial relative bias in the estimated predictor effects (up to 118% at EPV = 5), but this had little influence on the model's performance in our simulations. CONCLUSION: We recommend at least 10 EPV to fit prediction models in clustered data using logistic regression. Up to 50 EPV may be needed when variable selection is performed.

Internal validation of predictive models: efficiency of some procedures for logistic regression analysis.

The performance of a predictive model is overestimated when simply determined on the sample of subjects that was used to construct the model. Several internal validation methods are available that aim to provide a more accurate estimate of model performance in new subjects. We evaluated several variants of split-sample, cross-validation and bootstrapping methods with a logistic regression model that included eight predictors for 30-day mortality after an acute myocardial infarction. Random samples with a size between n = 572 and n = 9165 were drawn from a large data set (GUSTO-I; n = 40,830; 2851 deaths) to reflect modeling in data sets with between 5 and 80 events per variable. Independent performance was determined on the remaining subjects. Performance measures included discriminative ability, calibration and overall accuracy. We found that split-sample analyses gave overly pessimistic estimates of performance, with large variability. Cross-validation on 10% of the sample had low bias and low variability, but was not suitable for all performance measures. Internal validity could best be estimated with bootstrapping, which provided stable estimates with low bias. We conclude that split-sample validation is inefficient, and recommend bootstrapping for estimation of internal validity of a predictive logistic regression model.

Validation, updating and impact of clinical prediction rules: a review.

OBJECTIVE: To provide an overview of the research steps that need to follow the development of diagnostic or prognostic prediction rules. These steps include validity assessment, updating (if necessary), and impact assessment of clinical prediction rules. STUDY DESIGN AND SETTING: Narrative review covering methodological and empirical prediction studies from primary and secondary care. RESULTS: In general, three types of validation of previously developed prediction rules can be distinguished: temporal, geographical, and domain validations. In case of poor validation, the validation data can be used to update or adjust the previously developed prediction rule to the new circumstances. These update methods differ in extensiveness, with the easiest method a change in model intercept to the outcome occurrence at hand. Prediction rules -- with or without updating -- showing good performance in (various) validation studies may subsequently be subjected to an impact study, to demonstrate whether they change physicians' decisions, improve clinically relevant process parameters, patient outcome, or reduce costs. Finally, whether a prediction rule is implemented successfully in clinical practice depends on several potential barriers to the use of the rule. CONCLUSION: The development of a diagnostic or prognostic prediction rule is just a first step. We reviewed important aspects of the subsequent steps in prediction research.

A new diagnostic rule for deep vein thrombosis: safety and efficiency in clinically relevant subgroups.

BACKGROUND: Recently, a new, simple diagnostic rule was introduced to enable GPs to safely refrain from referring a considerable proportion of the patients suspected of having deep vein thrombosis (DVT). The rule (which includes seven patient history and physical examination items plus the result of a D-dimer test) discriminates 'very low' risk patients (not to be referred) from patients with an increased risk of DVT (to be referred). However, the rule's 'efficiency' (proportion of patients designated by the rule as very low risk) and safety (DVT prevalence among these very low risk patients) may change according to patient characteristics. OBJECTIVE: To test the rule's safety and efficiency in clinically relevant subgroups; i.e. across three age groups, in men and women, and in patients with and without a history of DVT, separately. METHODS: We retrospectively analysed data of 2086 primary care patients suspected of DVT, in whom all rule items and the result of the reference ('gold') standard (compression ultrasonography) were collected. RESULTS: The rule's efficiency decreased with age from 38.1% in the relatively young (<50 years) compared to 9.8% in patients aged > or =70 years. The percentage of DVT among the very low risk patients was <1.5% in all subgroups. The low efficiency in the elderly could be improved without compromising the safety by increasing the D-dimer threshold. CONCLUSION: The rule can safely exclude DVT in primary care patients suspected of DVT, irrespective of age, gender and history of DVT.