Factors associated with inappropriateness of antibiotic prescriptions for acutely ill children presenting to ambulatory care in high-income countries: a systematic review and meta-analysis.

BACKGROUND: Acutely ill children are at risk of unwarranted antibiotic prescribing. Data on the appropriateness of antibiotic prescriptions provide insights into potential tailored interventions to promote antibiotic stewardship. OBJECTIVES: To examine factors associated with the inappropriateness of antibiotic prescriptions for acutely ill children presenting to ambulatory care in high-income countries. METHODS: On 8 September 2022, we systematically searched articles published since 2002 in MEDLINE, Embase, CENTRAL, Web of Science, and grey literature databases. We included studies with acutely ill children presenting to ambulatory care settings in high-income countries reporting on the appropriateness of antibiotic prescriptions. The quality of the studies was evaluated using the Appraisal tool for Cross-Sectional Studies and the Newcastle-Ottawa Scale. Pooled ORs were calculated using random-effects models. Meta-regression, sensitivity and subgroup analysis were also performed. RESULTS: We included 40 articles reporting on 30 different factors and their association with inappropriate antibiotic prescribing. 'Appropriateness' covered a wide range of definitions. The following factors were associated with increased inappropriate antibiotic prescribing: acute otitis media diagnosis [pooled OR (95% CI): 2.02 (0.54-7.48)], GP [pooled OR (95% CI) 1.38 (1.00-1.89)] and rural setting [pooled OR (95% CI) 1.47 (1.08-2.02)]. Older patient age and a respiratory tract infection diagnosis have a tendency to be positively associated with inappropriate antibiotic prescribing, but pooling of studies was not possible. CONCLUSIONS: Prioritizing acute otitis media, GPs, rural areas, older children and respiratory tract infections within antimicrobial stewardship programmes plays a vital role in promoting responsible antibiotic prescribing. The implementation of a standardized definition of appropriateness is essential to evaluate such programmes.

Assessing Performance and Clinical Usefulness in Prediction Models With Survival Outcomes: Practical Guidance for Cox Proportional Hazards Models.

Risk prediction models need thorough validation to assess their performance. Validation of models for survival outcomes poses challenges due to the censoring of observations and the varying time horizon at which predictions can be made. This article describes measures to evaluate predictions and the potential improvement in decision making from survival models based on Cox proportional hazards regression.As a motivating case study, the authors consider the prediction of the composite outcome of recurrence or death (the "event") in patients with breast cancer after surgery. They developed a simple Cox regression model with 3 predictors, as in the Nottingham Prognostic Index, in 2982 women (1275 events over 5 years of follow-up) and externally validated this model in 686 women (285 events over 5 years). Improvement in performance was assessed after the addition of progesterone receptor as a prognostic biomarker.The model predictions can be evaluated across the full range of observed follow-up times or for the event occurring by the end of a fixed time horizon of interest. The authors first discuss recommended statistical measures that evaluate model performance in terms of discrimination, calibration, or overall performance. Further, they evaluate the potential clinical utility of the model to support clinical decision making according to a net benefit measure. They provide SAS and R code to illustrate internal and external validation.The authors recommend the proposed set of performance measures for transparent reporting of the validity of predictions from survival models.

There is no such thing as a validated prediction model.

BACKGROUND: Clinical prediction models should be validated before implementation in clinical practice. But is favorable performance at internal validation or one external validation sufficient to claim that a prediction model works well in the intended clinical context? MAIN BODY: We argue to the contrary because (1) patient populations vary, (2) measurement procedures vary, and (3) populations and measurements change over time. Hence, we have to expect heterogeneity in model performance between locations and settings, and across time. It follows that prediction models are never truly validated. This does not imply that validation is not important. Rather, the current focus on developing new models should shift to a focus on more extensive, well-conducted, and well-reported validation studies of promising models. CONCLUSION: Principled validation strategies are needed to understand and quantify heterogeneity, monitor performance over time, and update prediction models when appropriate. Such strategies will help to ensure that prediction models stay up-to-date and safe to support clinical decision-making.

The Risk of Endometrial Malignancy and Other Endometrial Pathology in Women with Abnormal Uterine Bleeding: An Ultrasound-Based Model Development Study by the IETA Group.

OBJECTIVES: The aim of this study was to develop a model that can discriminate between different etiologies of abnormal uterine bleeding. DESIGN: The International Endometrial Tumor Analysis 1 study is a multicenter observational diagnostic study in 18 bleeding clinics in 9 countries. Consecutive women with abnormal vaginal bleeding presenting for ultrasound examination (n = 2,417) were recruited. The histology was obtained from endometrial sampling, D&C, hysteroscopic resection, hysterectomy, or ultrasound follow-up for >1 year. METHODS: A model was developed using multinomial regression based on age, body mass index, and ultrasound predictors to distinguish between: (1) endometrial atrophy, (2) endometrial polyp or intracavitary myoma, (3) endometrial malignancy or atypical hyperplasia, (4) proliferative/secretory changes, endometritis, or hyperplasia without atypia and validated using leave-center-out cross-validation and bootstrapping. The main outcomes are the model's ability to discriminate between the four outcomes and the calibration of risk estimates. RESULTS: The median age in 2,417 women was 50 (interquartile range 43-57). 414 (17%) women had endometrial atrophy; 996 (41%) had a polyp or myoma; 155 (6%) had an endometrial malignancy or atypical hyperplasia; and 852 (35%) had proliferative/secretory changes, endometritis, or hyperplasia without atypia. The model distinguished well between malignant and benign histology (c-statistic 0.88 95% CI: 0.85-0.91) and between all benign histologies. The probabilities for each of the four outcomes were over- or underestimated depending on the centers. LIMITATIONS: Not all patients had a diagnosis based on histology. The model over- or underestimated the risk for certain outcomes in some centers, indicating local recalibration is advisable. CONCLUSIONS: The proposed model reliably distinguishes between four histological outcomes. This is the first model to discriminate between several outcomes and is the only model applicable when menopausal status is uncertain. The model could be useful for patient management and counseling, and aid in the interpretation of ultrasound findings. Future research is needed to externally validate and locally recalibrate the model.

Posttraumatic stress, anxiety and depression following miscarriage and ectopic pregnancy: a multicenter, prospective, cohort study.

BACKGROUND: Early pregnancy losses are common, but their psychologic sequelae are often overlooked. Previous studies have established links between miscarriage and early symptoms of anxiety and depression. However, the incidence of posttraumatic stress symptoms and the psychologic response specifically to ectopic pregnancies have not been investigated. OBJECTIVE: The purpose of this study was to investigate levels of posttraumatic stress, depression, and anxiety in women in the 9 months after early pregnancy loss, with a focus on miscarriage and ectopic pregnancy. Morbidity at 1 month was compared with a control group in healthy pregnancy. STUDY DESIGN: This was a prospective cohort study. Consecutive women were recruited from the early pregnancy and antenatal clinics at 3 London hospitals and received emailed surveys that contained standardized psychologic assessments that included the Hospital Anxiety and Depression Scale and Posttraumatic stress Diagnostic Scale, at 1, 3, and 9 months after loss. Control subjects were assessed after a dating scan. We assessed the proportion of participants who met the screening criteria for posttraumatic stress and moderate/severe anxiety or depression. We used logistic regression to calculate adjusted odds ratios. RESULTS: Seven hundred thirty-seven of 1098 women (67%) with early pregnancy loss (including 537 miscarriages and 116 ectopic pregnancies) and 171 of 187 control subjects (91%) agreed to participate. Four hundred ninety-two of the women with losses (67%) completed the Hospital Anxiety and Depression Scale after 1 month; 426 women (58%) completed it after 3 months, and 338 women (46%) completed it after 9 months. Eighty-seven control subjects (51%) participated. Criteria for posttraumatic stress were met in 29% of women with early pregnancy loss after 1 month and in 18% after 9 months (odds ratio per month, 0.80; 95% confidence interval, 0.72-0.89). Moderate/severe anxiety was reported in 24% after 1 month and in 17% after 9 months (odds ratio per month, 0.69; 95% confidence interval, 0.50-0.94). Moderate/severe depression was reported in 11% of the women after 1 month and 6% of the women after 9 months (odds ratio per month, 0.87; 95% confidence interval, 0.53-1.44). After miscarriage, proportions after 9 months were 16% for posttraumatic stress, 17% for anxiety, and 5% for depression. Corresponding figures after ectopic pregnancy were 21%, 23%, and 11%, respectively. In contrast, among control women with viable pregnancies, 13% reported moderate-to-severe anxiety (odds ratio loss at 1 month vs controls: 2.14; 95% confidence interval, 1.14-4.36), and 2% reported moderate-to-severe depression (odds ratio loss at 1 month vs control subjects: 3.88; 95% confidence interval, 1.27-19.2). CONCLUSION: Women experience high levels of posttraumatic stress, anxiety, and depression after early pregnancy loss. Distress declines over time but remains at clinically important levels at 9 months.

Developing risk models for multicenter data using standard logistic regression produced suboptimal predictions: A simulation study.

Although multicenter data are common, many prediction model studies ignore this during model development. The objective of this study is to evaluate the predictive performance of regression methods for developing clinical risk prediction models using multicenter data, and provide guidelines for practice. We compared the predictive performance of standard logistic regression, generalized estimating equations, random intercept logistic regression, and fixed effects logistic regression. First, we presented a case study on the diagnosis of ovarian cancer. Subsequently, a simulation study investigated the performance of the different models as a function of the amount of clustering, development sample size, distribution of center-specific intercepts, the presence of a center-predictor interaction, and the presence of a dependency between center effects and predictors. The results showed that when sample sizes were sufficiently large, conditional models yielded calibrated predictions, whereas marginal models yielded miscalibrated predictions. Small sample sizes led to overfitting and unreliable predictions. This miscalibration was worse with more heavily clustered data. Calibration of random intercept logistic regression was better than that of standard logistic regression even when center-specific intercepts were not normally distributed, a center-predictor interaction was present, center effects and predictors were dependent, or when the model was applied in a new center. Therefore, to make reliable predictions in a specific center, we recommend random intercept logistic regression.

Risk of complications in patients with conservatively managed ovarian tumours (IOTA5): a 2-year interim analysis of a multicentre, prospective, cohort study.

BACKGROUND: Ovarian tumours are usually surgically removed because of the presumed risk of complications. Few large prospective studies on long-term follow-up of adnexal masses exist. We aimed to estimate the cumulative incidence of cyst complications and malignancy during the first 2 years of follow-up after adnexal masses have been classified as benign by use of ultrasonography. METHODS: In the international, prospective, cohort International Ovarian Tumor Analysis Phase 5 (IOTA5) study, patients aged 18 years or older with at least one adnexal mass who had been selected for surgery or conservative management after ultrasound assessment were recruited consecutively from 36 cancer and non-cancer centres in 14 countries. Follow-up of patients managed conservatively is ongoing at present. In this 2-year interim analysis, we analysed patients who were selected for conservative management of an adnexal mass judged to be benign on ultrasound on the basis of subjective assessment of ultrasound images. Conservative management included ultrasound and clinical follow-up at intervals of 3 months and 6 months, and then every 12 months thereafter. The main outcomes of this 2-year interim analysis were cumulative incidence of spontaneous resolution of the mass, torsion or cyst rupture, or borderline or invasive malignancy confirmed surgically in patients with a newly diagnosed adnexal mass. IOTA5 is registered with ClinicalTrials.gov, number NCT01698632, and the central Ethics Committee and the Belgian Federal Agency for Medicines and Health Products, number S51375/B32220095331, and is ongoing. FINDINGS: Between Jan 1, 2012, and March 1, 2015, 8519 patients were recruited to IOTA5. 3144 (37%) patients selected for conservative management were eligible for inclusion in our analysis, of whom 221 (7%) had no follow-up data and 336 (11%) were operated on before a planned follow-up scan was done. Of 2587 (82%) patients with follow-up data, 668 (26%) had a mass that was already in follow-up at recruitment, and 1919 (74%) presented with a new mass at recruitment (ie, not already in follow-up in the centre before recruitment). Median follow-up of patients with new masses was 27 months (IQR 14-38). The cumulative incidence of spontaneous resolution within 2 years of follow-up among those with a new mass at recruitment (n=1919) was 20·2% (95% CI 18·4-22·1), and of finding invasive malignancy at surgery was 0·4% (95% CI 0·1-0·6), 0·3% (<0·1-0·5) for a borderline tumour, 0·4% (0·1-0·7) for torsion, and 0·2% (<0·1-0·4) for cyst rupture. INTERPRETATION: Our results suggest that the risk of malignancy and acute complications is low if adnexal masses with benign ultrasound morphology are managed conservatively, which could be of value when counselling patients, and supports conservative management of adnexal masses classified as benign by use of ultrasound. FUNDING: Research Foundation Flanders, KU Leuven, Swedish Research Council.

Calibration: the Achilles heel of predictive analytics.

BACKGROUND: The assessment of calibration performance of risk prediction models based on regression or more flexible machine learning algorithms receives little attention. MAIN TEXT: Herein, we argue that this needs to change immediately because poorly calibrated algorithms can be misleading and potentially harmful for clinical decision-making. We summarize how to avoid poor calibration at algorithm development and how to assess calibration at algorithm validation, emphasizing balance between model complexity and the available sample size. At external validation, calibration curves require sufficiently large samples. Algorithm updating should be considered for appropriate support of clinical practice. CONCLUSION: Efforts are required to avoid poor calibration when developing prediction models, to evaluate calibration when validating models, and to update models when indicated. The ultimate aim is to optimize the utility of predictive analytics for shared decision-making and patient counseling.

Three myths about risk thresholds for prediction models.

BACKGROUND: Clinical prediction models are useful in estimating a patient's risk of having a certain disease or experiencing an event in the future based on their current characteristics. Defining an appropriate risk threshold to recommend intervention is a key challenge in bringing a risk prediction model to clinical application; such risk thresholds are often defined in an ad hoc way. This is problematic because tacitly assumed costs of false positive and false negative classifications may not be clinically sensible. For example, when choosing the risk threshold that maximizes the proportion of patients correctly classified, false positives and false negatives are assumed equally costly. Furthermore, small to moderate sample sizes may lead to unstable optimal thresholds, which requires a particularly cautious interpretation of results. MAIN TEXT: We discuss how three common myths about risk thresholds often lead to inappropriate risk stratification of patients. First, we point out the contexts of counseling and shared decision-making in which a continuous risk estimate is more useful than risk stratification. Second, we argue that threshold selection should reflect the consequences of the decisions made following risk stratification. Third, we emphasize that there is usually no universally optimal threshold but rather that a plausible risk threshold depends on the clinical context. Consequently, we recommend to present results for multiple risk thresholds when developing or validating a prediction model. CONCLUSION: Bearing in mind these three considerations can avoid inappropriate allocation (and non-allocation) of interventions. Using discriminating and well-calibrated models will generate better clinical outcomes if context-dependent thresholds are used.

Predicting the risk of malignancy in adnexal masses based on the Simple Rules from the International Ovarian Tumor Analysis group.

BACKGROUND: Accurate methods to preoperatively characterize adnexal tumors are pivotal for optimal patient management. A recent metaanalysis concluded that the International Ovarian Tumor Analysis algorithms such as the Simple Rules are the best approaches to preoperatively classify adnexal masses as benign or malignant. OBJECTIVE: We sought to develop and validate a model to predict the risk of malignancy in adnexal masses using the ultrasound features in the Simple Rules. STUDY DESIGN: This was an international cross-sectional cohort study involving 22 oncology centers, referral centers for ultrasonography, and general hospitals. We included consecutive patients with an adnexal tumor who underwent a standardized transvaginal ultrasound examination and were selected for surgery. Data on 5020 patients were recorded in 3 phases from 2002 through 2012. The 5 Simple Rules features indicative of a benign tumor (B-features) and the 5 features indicative of malignancy (M-features) are based on the presence of ascites, tumor morphology, and degree of vascularity at ultrasonography. Gold standard was the histopathologic diagnosis of the adnexal mass (pathologist blinded to ultrasound findings). Logistic regression analysis was used to estimate the risk of malignancy based on the 10 ultrasound features and type of center. The diagnostic performance was evaluated by area under the receiver operating characteristic curve, sensitivity, specificity, positive likelihood ratio (LR+), negative likelihood ratio (LR-), positive predictive value (PPV), negative predictive value (NPV), and calibration curves. RESULTS: Data on 4848 patients were analyzed. The malignancy rate was 43% (1402/3263) in oncology centers and 17% (263/1585) in other centers. The area under the receiver operating characteristic curve on validation data was very similar in oncology centers (0.917; 95% confidence interval, 0.901-0.931) and other centers (0.916; 95% confidence interval, 0.873-0.945). Risk estimates showed good calibration. In all, 23% of patients in the validation data set had a very low estimated risk (<1%) and 48% had a high estimated risk (≥30%). For the 1% risk cutoff, sensitivity was 99.7%, specificity 33.7%, LR+ 1.5, LR- 0.010, PPV 44.8%, and NPV 98.9%. For the 30% risk cutoff, sensitivity was 89.0%, specificity 84.7%, LR+ 5.8, LR- 0.13, PPV 75.4%, and NPV 93.9%. CONCLUSION: Quantification of the risk of malignancy based on the Simple Rules has good diagnostic performance both in oncology centers and other centers. A simple classification based on these risk estimates may form the basis of a clinical management system. Patients with a high risk may benefit from surgery by a gynecological oncologist, while patients with a lower risk may be managed locally.

ADNEX risk prediction model for diagnosis of ovarian cancer: systematic review and meta-analysis of external validation studies.

Objectives: To conduct a systematic review of studies externally validating the ADNEX (Assessment of Different Neoplasias in the adnexa) model for diagnosis of ovarian cancer and to present a meta-analysis of its performance. Design: Systematic review and meta-analysis of external validation studies. Data sources: Medline, Embase, Web of Science, Scopus, and Europe PMC, from 15 October 2014 to 15 May 2023. Eligibility criteria for selecting studies: All external validation studies of the performance of ADNEX, with any study design and any study population of patients with an adnexal mass. Two independent reviewers extracted the data. Disagreements were resolved by discussion. Reporting quality of the studies was scored with the TRIPOD (Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis) reporting guideline, and methodological conduct and risk of bias with PROBAST (Prediction model Risk Of Bias Assessment Tool). Random effects meta-analysis of the area under the receiver operating characteristic curve (AUC), sensitivity and specificity at the 10% risk of malignancy threshold, and net benefit and relative utility at the 10% risk of malignancy threshold were performed. Results: 47 studies (17 007 tumours) were included, with a median study sample size of 261 (range 24-4905). On average, 61% of TRIPOD items were reported. Handling of missing data, justification of sample size, and model calibration were rarely described. 91% of validations were at high risk of bias, mainly because of the unexplained exclusion of incomplete cases, small sample size, or no assessment of calibration. The summary AUC to distinguish benign from malignant tumours in patients who underwent surgery was 0.93 (95% confidence interval 0.92 to 0.94, 95% prediction interval 0.85 to 0.98) for ADNEX with the serum biomarker, cancer antigen 125 (CA125), as a predictor (9202 tumours, 43 centres, 18 countries, and 21 studies) and 0.93 (95% confidence interval 0.91 to 0.94, 95% prediction interval 0.85 to 0.98) for ADNEX without CA125 (6309 tumours, 31 centres, 13 countries, and 12 studies). The estimated probability that the model has use clinically in a new centre was 95% (with CA125) and 91% (without CA125). When restricting analysis to studies with a low risk of bias, summary AUC values were 0.93 (with CA125) and 0.91 (without CA125), and estimated probabilities that the model has use clinically were 89% (with CA125) and 87% (without CA125). Conclusions: The results of the meta-analysis indicated that ADNEX performed well in distinguishing between benign and malignant tumours in populations from different countries and settings, regardless of whether the serum biomarker, CA125, was used as a predictor. A key limitation was that calibration was rarely assessed. Systematic review registration: PROSPERO CRD42022373182.

Development and validation of a risk prediction model for hospital admission in COVID-19 patients presenting to primary care.

BACKGROUND: There is a paucity of prognostic models for COVID-19 that are usable for in-office patient assessment in general practice (GP). OBJECTIVES: To develop and validate a risk prediction model for hospital admission with readily available predictors. METHODS: A retrospective cohort study linking GP records from 8 COVID-19 centres and 55 general practices in the Netherlands to hospital admission records. The development cohort spanned March to June 2020, the validation cohort March to June 2021. The primary outcome was hospital admission within 14 days. We used geographic leave-region-out cross-validation in the development cohort and temporal validation in the validation cohort. RESULTS: In the development cohort, 4,806 adult patients with COVID-19 consulted their GP (median age 56, 56% female); in the validation cohort 830 patients did (median age 56, 52% female). In the development and validation cohort respectively, 292 (6.1%) and 126 (15.2%) were admitted to the hospital within 14 days, respectively. A logistic regression model based on sex, smoking, symptoms, vital signs and comorbidities predicted hospital admission with a c-index of 0.84 (95% CI 0.83 to 0.86) at geographic cross-validation and 0.79 (95% CI 0.74 to 0.83) at temporal validation, and was reasonably well calibrated (intercept -0.08, 95% CI -0.98 to 0.52, slope 0.89, 95% CI 0.71 to 1.07 at geographic cross-validation and intercept 0.02, 95% CI -0.21 to 0.24, slope 0.82, 95% CI 0.64 to 1.00 at temporal validation). CONCLUSION: We derived a risk model using readily available variables at GP assessment to predict hospital admission for COVID-19. It performed accurately across regions and waves. Further validation on cohorts with acquired immunity and newer SARS-CoV-2 variants is recommended.

A general practice prediction model based on signs and symptoms of COVID-19 patients reliably predicted hospitalisation.The model performed well in second-wave data with other dominant variants and changed testing and vaccination policies.In an emerging pandemic, GP data can be leveraged to develop prognostic models for decision support and to predict hospitalisation rates.

Table 0; documenting the steps to go from clinical database to research dataset.

OBJECTIVES: Data-driven decision support tools have been increasingly recognized to transform health care. However, such tools are often developed on predefined research datasets without adequate knowledge of the origin of this data and how it was selected. How a dataset is extracted from a clinical database can profoundly impact the validity, interpretability and interoperability of the dataset, and downstream analyses, yet is rarely reported. Therefore, we present a case study illustrating how a definitive patient list was extracted from a clinical source database and how this can be reported. STUDY DESIGN AND SETTING: A single-center observational study was performed at an academic hospital in the Netherlands to illustrate the impact of selecting a definitive patient list for research from a clinical source database, and the importance of documenting this process. All admissions from the critical care database admitted between January 1, 2013, and January 1, 2023, were used. RESULTS: An interdisciplinary team collaborated to identify and address potential sources of data insufficiency and uncertainty. We demonstrate a stepwise data preparation process, reducing the clinical source database of 54,218 admissions to a definitive patient list of 21,553 admissions. Transparent documentation of the data preparation process improves the quality of the definitive patient list before analysis of the corresponding patient data. This study generated seven important recommendations for preparing observational health-care data for research purposes. CONCLUSION: Documenting data preparation is essential for understanding a research dataset originating from a clinical source database before analyzing health-care data. The findings contribute to establishing data standards and offer insights into the complexities of preparing health-care data for scientific investigation. Meticulous data preparation and documentation thereof will improve research validity and advance critical care.

Screening for data clustering in multicenter studies: the residual intraclass correlation.

BACKGROUND: In multicenter studies, center-specific variations in measurements may arise for various reasons, such as low interrater reliability, differences in equipment, deviations from the protocol, sociocultural characteristics, and differences in patient populations due to e.g. local referral patterns. The aim of this research is to derive measures for the degree of clustering. We present a method to detect heavily clustered variables and to identify physicians with outlying measurements. METHODS: We use regression models with fixed effects to account for patient case-mix and a random cluster intercept to study clustering by physicians. We propose to use the residual intraclass correlation (RICC), the proportion of residual variance that is situated at the cluster level, to detect variables that are influenced by clustering. An RICC of 0 indicates that the variance in the measurements is not due to variation between clusters. We further suggest, where appropriate, to evaluate RICC in combination with R2, the proportion of variance that is explained by the fixed effects. Variables with a high R2 may have benefits that outweigh the disadvantages of clustering in terms of statistical analysis. We apply the proposed methods to a dataset collected for the development of models for ovarian tumor diagnosis. We study the variability in 18 tumor characteristics collected through ultrasound examination, 4 patient characteristics, and the serum marker CA-125 measured by 40 physicians on 2407 patients. RESULTS: The RICC showed large variation between variables: from 2.2% for age to 25.1% for the amount of fluid in the pouch of Douglas. Seven variables had an RICC above 15%, indicating that a considerable part of the variance is due to systematic differences at the physician level, rather than random differences at the patient level. Accounting for differences in ultrasound machine quality reduced the RICC for a number of blood flow measurements. CONCLUSIONS: We recommend that the degree of data clustering is addressed during the monitoring and analysis of multicenter studies. The RICC is a useful tool that expresses the degree of clustering as a percentage. Specific applications are data quality monitoring and variable screening prior to the development of a prediction model.

Value-of-Information Analysis for External Validation of Risk Prediction Models.

BACKGROUND: A previously developed risk prediction model needs to be validated before being used in a new population. The finite size of the validation sample entails that there is uncertainty around model performance. We apply value-of-information (VoI) methodology to quantify the consequence of uncertainty in terms of net benefit (NB). METHODS: We define the expected value of perfect information (EVPI) for model validation as the expected loss in NB due to not confidently knowing which of the alternative decisions confers the highest NB. We propose bootstrap-based and asymptotic methods for EVPI computations and conduct simulation studies to compare their performance. In a case study, we use the non-US subsets of a clinical trial as the development sample for predicting mortality after myocardial infarction and calculate the validation EVPI for the US subsample. RESULTS: The computation methods generated similar EVPI values in simulation studies. EVPI generally declined with larger samples. In the case study, at the prespecified threshold of 0.02, the best decision with current information would be to use the model, with an incremental NB of 0.0020 over treating all. At this threshold, the EVPI was 0.0005 (relative EVPI = 25%). When scaled to the annual number of heart attacks in the US, the expected NB loss due to uncertainty was equal to 400 true positives or 19,600 false positives, indicating the value of further model validation. CONCLUSION: VoI methods can be applied to the NB calculated during external validation of clinical prediction models. While uncertainty does not directly affect the clinical implications of NB findings, validation EVPI provides an objective perspective to the need for further validation and can be reported alongside NB in external validation studies. HIGHLIGHTS: External validation is a critical step when transporting a risk prediction model to a new setting, but the finite size of the validation sample creates uncertainty about the performance of the model.In decision theory, such uncertainty is associated with loss of net benefit because it can prevent one from identifying whether the use of the model is beneficial over alternative strategies.We define the expected value of perfect information for external validation as the expected loss in net benefit by not confidently knowing if the use of the model is net beneficial.The adoption of a model for a new population should be based on its expected net benefit; independently, value-of-information methods can be used to decide whether further validation studies are warranted.

Decision curve analysis: confidence intervals and hypothesis testing for net benefit.

BACKGROUND: A number of recent papers have proposed methods to calculate confidence intervals and p values for net benefit used in decision curve analysis. These papers are sparse on the rationale for doing so. We aim to assess the relation between sampling variability, inference, and decision-analytic concepts. METHODS AND RESULTS: We review the underlying theory of decision analysis. When we are forced into a decision, we should choose the option with the highest expected utility, irrespective of p values or uncertainty. This is in some distinction to traditional hypothesis testing, where a decision such as whether to reject a given hypothesis can be postponed. Application of inference for net benefit would generally be harmful. In particular, insisting that differences in net benefit be statistically significant would dramatically change the criteria by which we consider a prediction model to be of value. We argue instead that uncertainty related to sampling variation for net benefit should be thought of in terms of the value of further research. Decision analysis tells us which decision to make for now, but we may also want to know how much confidence we should have in that decision. If we are insufficiently confident that we are right, further research is warranted. CONCLUSION: Null hypothesis testing or simple consideration of confidence intervals are of questionable value for decision curve analysis, and methods such as value of information analysis or approaches to assess the probability of benefit should be considered instead.

What proportion of clinical prediction models make it to clinical practice? Protocol for a two-track follow-up study of prediction model development publications.

INTRODUCTION: It is known that only a limited proportion of developed clinical prediction models (CPMs) are implemented and/or used in clinical practice. This may result in a large amount of research waste, even when considering that some CPMs may demonstrate poor performance. Cross-sectional estimates of the numbers of CPMs that have been developed, validated, evaluated for impact or utilized in practice, have been made in specific medical fields, but studies across multiple fields and studies following up the fate of CPMs are lacking. METHODS AND ANALYSIS: We have conducted a systematic search for prediction model studies published between January 1995 and December 2020 using the Pubmed and Embase databases, applying a validated search strategy. Taking random samples for every calendar year, abstracts and articles were screened until a target of 100 CPM development studies were identified. Next, we will perform a forward citation search of the resulting CPM development article cohort to identify articles on external validation, impact assessment or implementation of those CPMs. We will also invite the authors of the development studies to complete an online survey to track implementation and clinical utilization of the CPMs.We will conduct a descriptive synthesis of the included studies, using data from the forward citation search and online survey to quantify the proportion of developed models that are validated, assessed for their impact, implemented and/or used in patient care. We will conduct time-to-event analysis using Kaplan-Meier plots. ETHICS AND DISSEMINATION: No patient data are involved in the research. Most information will be extracted from published articles. We request written informed consent from the survey respondents. Results will be disseminated through publication in a peer-reviewed journal and presented at international conferences. OSF REGISTRATION: (https://osf.io/nj8s9).