The Vienna Prediction Model for identifying patients at low risk of recurrent venous thromboembolism: a prospective cohort study.

BACKGROUND AND AIMS: Patients with unprovoked venous thromboembolism (VTE) have a high recurrence risk, and guidelines suggest extended-phase anticoagulation. Many patients never experience recurrence but are exposed to bleeding. The aim of this study was to assess the performance of the Vienna Prediction Model (VPM) and to evaluate if the VPM accurately identifies these patients. METHODS: In patients with unprovoked VTE, the VPM was performed 3 weeks after anticoagulation withdrawal. Those with a predicted 1-year recurrence risk of ≤5.5% were prospectively followed. Study endpoint was recurrent VTE over 2 years. RESULTS: A total of 818 patients received anticoagulation for a median of 3.9 months. 520 patients (65%) had a predicted annual recurrence risk of ≤5.5%. During a median time of 23.9 months, 52 patients had non-fatal recurrence. The recurrence risk was 5.2% [95% confidence interval (CI) 3.2-7.2] at 1 year and 11.2% (95% CI 8.3-14) at 2 years. Model calibration was adequate after 1 year. The VPM underestimated the recurrence risk of patients with a 2-year recurrence rate of >5%. In a post-hoc analysis, the VPM's baseline hazard was recalibrated. Bootstrap validation confirmed an ideal ratio of observed and expected recurrence events. The recurrence risk was highest in men with proximal deep-vein thrombosis or pulmonary embolism and lower in women regardless of the site of incident VTE. CONCLUSIONS: In this prospective evaluation of the performance of the VPM, the 1-year rate of recurrence in patients with unprovoked VTE was 5.2%. Recalibration improved identification of patients at low recurrence risk and stratification into distinct low-risk categories.

Flexible parametrization of graph-theoretical features from individual-specific networks for prediction.

Statistical techniques are needed to analyze data structures with complex dependencies such that clinically useful information can be extracted. Individual-specific networks, which capture dependencies in complex biological systems, are often summarized by graph-theoretical features. These features, which lend themselves to outcome modeling, can be subject to high variability due to arbitrary decisions in network inference and noise. Correlation-based adjacency matrices often need to be sparsified before meaningful graph-theoretical features can be extracted, requiring the data analysts to determine an optimal threshold. To address this issue, we propose to incorporate a flexible weighting function over the full range of possible thresholds to capture the variability of graph-theoretical features over the threshold domain. The potential of this approach, which extends concepts from functional data analysis to a graph-theoretical setting, is explored in a plasmode simulation study using real functional magnetic resonance imaging (fMRI) data from the Autism Brain Imaging Data Exchange (ABIDE) Preprocessed initiative. The simulations show that our modeling approach yields accurate estimates of the functional form of the weight function, improves inference efficiency, and achieves a comparable or reduced root mean square prediction error compared to competitor modeling approaches. This assertion holds true in settings where both complex functional forms underlie the outcome-generating process and a universal threshold value is employed. We demonstrate the practical utility of our approach by using resting-state fMRI data to predict biological age in children. Our study establishes the flexible modeling approach as a statistically principled, serious competitor to ad-hoc methods with superior performance.

Phases of methodological research in biostatistics-Building the evidence base for new methods.

Although new biostatistical methods are published at a very high rate, many of these developments are not trustworthy enough to be adopted by the scientific community. We propose a framework to think about how a piece of methodological work contributes to the evidence base for a method. Similar to the well-known phases of clinical research in drug development, we propose to define four phases of methodological research. These four phases cover (I) proposing a new methodological idea while providing, for example, logical reasoning or proofs, (II) providing empirical evidence, first in a narrow target setting, then (III) in an extended range of settings and for various outcomes, accompanied by appropriate application examples, and (IV) investigations that establish a method as sufficiently well-understood to know when it is preferred over others and when it is not; that is, its pitfalls. We suggest basic definitions of the four phases to provoke thought and discussion rather than devising an unambiguous classification of studies into phases. Too many methodological developments finish before phase III/IV, but we give two examples with references. Our concept rebalances the emphasis to studies in phases III and IV, that is, carefully planned method comparison studies and studies that explore the empirical properties of existing methods in a wider range of problems.

Predicting pediatric kidney disease progression-are 3 variables all you need?

Accurate estimation of chronic kidney disease (CKD) progression risk is vital for clinical decision-making. Existing risk equations lack validation in pediatric CKD populations. Ng et al. developed new risk equations using the CKD in Children and European Study Consortium for Chronic Kidney Disorders Affecting Pediatric Patients cohorts. The elementary model, incorporating estimated glomerular filtration rate, urine protein-creatinine ratio, and diagnosis, exhibited excellent discrimination and calibration at external validation. External validation of enriched models is pending. The equations have the potential to aid pediatric CKD centers in patient counseling and care planning.

Sex differences in the survival benefit of kidney transplantation: a retrospective cohort study using target trial emulation.

BACKGROUND: Kidney transplantation is the preferred treatment for eligible patients with kidney failure who need renal replacement therapy. However, it remains unclear whether the anticipated survival benefit from kidney transplantation is different for women and men. METHODS: We included all dialysis patients recorded in the Austrian Dialysis and Transplant Registry who were waitlisted for their first kidney transplant between 2000 and 2018. In order to estimate the causal effect of kidney transplantation on 10-year restricted mean survival time, we mimicked a series of controlled clinical trials and applied inverse probability of treatment and censoring weighted sequential Cox models. RESULTS: This study included 4408 patients (33% female) with a mean age of 52 years. Glomerulonephritis was the most common primary renal disease both in women (27%) and men (28%). Kidney transplantation led to a gain of 2.22 years (95% CI 1.88 to 2.49) compared with dialysis over a 10-year follow-up. The effect was smaller in women (1.95 years, 95% CI 1.38 to 2.41) than in men (2.35 years, 95% CI 1.92 to 2.70) due to a better survival on dialysis. Across ages the survival benefit of transplantation over a follow-up of 10 years was smaller in younger women and men and increased with age, showing a peak for both women and men aged about 60 years. CONCLUSIONS: There were few differences in survival benefit by transplantation between females and males. Females had better survival than males on the waitlist receiving dialysis and similar survival to males after transplantation.

Photonic quantum state transfer between a cold atomic gas and a crystal.

Interfacing fundamentally different quantum systems is key to building future hybrid quantum networks. Such heterogeneous networks offer capabilities superior to those of their homogeneous counterparts, as they merge the individual advantages of disparate quantum nodes in a single network architecture. However, few investigations of optical hybrid interconnections have been carried out, owing to fundamental and technological challenges such as wavelength and bandwidth matching of the interfacing photons. Here we report optical quantum interconnection of two disparate matter quantum systems with photon storage capabilities. We show that a quantum state can be transferred faithfully between a cold atomic ensemble and a rare-earth-doped crystal by means of a single photon at 1,552 nanometre telecommunication wavelength, using cascaded quantum frequency conversion. We demonstrate that quantum correlations between a photon and a single collective spin excitation in the cold atomic ensemble can be transferred to the solid-state system. We also show that single-photon time-bin qubits generated in the cold atomic ensemble can be converted, stored and retrieved from the crystal with a conditional qubit fidelity of more than 85 per cent. Our results open up the prospect of optically connecting quantum nodes with different capabilities and represent an important step towards the realization of large-scale hybrid quantum networks.

Critical appraisal of artificial intelligence-based prediction models for cardiovascular disease.

The medical field has seen a rapid increase in the development of artificial intelligence (AI)-based prediction models. With the introduction of such AI-based prediction model tools and software in cardiovascular patient care, the cardiovascular researcher and healthcare professional are challenged to understand the opportunities as well as the limitations of the AI-based predictions. In this article, we present 12 critical questions for cardiovascular health professionals to ask when confronted with an AI-based prediction model. We aim to support medical professionals to distinguish the AI-based prediction models that can add value to patient care from the AI that does not.

Modeling pulse wave velocity trajectories-challenges, opportunities, and pitfalls.

In this commentary, we discuss the analysis of trajectories of pulse wave velocity in a longitudinal cohort study of children with chronic kidney disease (the Cardiovascular Comorbidity in Children with Chronic Kidney Disease - Transplantation study). We revisit the analysis made by the study authors and unravel some additional limitations. We also reevaluate the implicit assumptions that were made in the chosen analysis and suggest extensions of the basic linear mixed model to obtain more differentiated answers to research questions in nephrology.

Topical imiquimod compared with conization to treat cervical high-grade squamous intraepithelial lesions: Multicenter, randomized controlled trial.

OBJECTIVE: In a previous phase II trial, we showed that topical imiquimod (IMQ) therapy is an efficacious treatment for high-grade squamous intraepithelial lesion (HSIL). Aim of the present study was to investigate the non-inferiority of a 16-week topical, self-applied IMQ therapy compared to large loop excision of the transformation zone (LLETZ) in patients diagnosed with HSIL. METHODS: Phase III randomized, controlled, multicenter, open trial performed by Austrian Gynecologic Oncology group. Patients with histologically proven cervical intraepithelial neoplasia (CIN)2 (30 years and older) or CIN3 (18 years and older) and satisfactory colposcopy were randomized to topical IMQ treatment or LLETZ. Successful treatment was defined as negative HPV high-risk test result 6 months after start of the treatment. Secondary endpoints were histological outcome and HPV clearance rates. RESULTS: Within 3 years 93 patients were randomized, received the allocated treatment and were available for ITT analysis. In the IMQ group negative HPV test at 6 months after treatment start was observed in 22/51 (43.1%) of patients compared to 27/42 (64.3%) in the LLETZ group on ITT analysis (rate difference 21.2%-points, 95% two-sided CI: 0.8 to 39.1). In the IMQ group histologic regression 6 months after treatment was observed in 32/51 (63%) of patients and complete histologic remission was observed in 19/51 (37%) of patients. Complete surgical resection was observed in 84% after LLETZ. CONCLUSION: In women with HSIL, IMQ treatment results in lower HPV clearance rates when compared to LLETZ. LLETZ remains the standard for women with HSIL when treatment is required. CLINICAL TRIAL REGISTRATION: ClinicalTrials.gov Identifier: NCT01283763, EudraCT number: 2012-004518-32.

An investigation of penalization and data augmentation to improve convergence of generalized estimating equations for clustered binary outcomes.

BACKGROUND: In binary logistic regression data are 'separable' if there exists a linear combination of explanatory variables which perfectly predicts the observed outcome, leading to non-existence of some of the maximum likelihood coefficient estimates. A popular solution to obtain finite estimates even with separable data is Firth's logistic regression (FL), which was originally proposed to reduce the bias in coefficient estimates. The question of convergence becomes more involved when analyzing clustered data as frequently encountered in clinical research, e.g. data collected in several study centers or when individuals contribute multiple observations, using marginal logistic regression models fitted by generalized estimating equations (GEE). From our experience we suspect that separable data are a sufficient, but not a necessary condition for non-convergence of GEE. Thus, we expect that generalizations of approaches that can handle separable uncorrelated data may reduce but not fully remove the non-convergence issues of GEE. METHODS: We investigate one recently proposed and two new extensions of FL to GEE. With 'penalized GEE' the GEE are treated as score equations, i.e. as derivatives of a log-likelihood set to zero, which are then modified as in FL. We introduce two approaches motivated by the equivalence of FL and maximum likelihood estimation with iteratively augmented data. Specifically, we consider fully iterated and single-step versions of this 'augmented GEE' approach. We compare the three approaches with respect to convergence behavior, practical applicability and performance using simulated data and a real data example. RESULTS: Our simulations indicate that all three extensions of FL to GEE substantially improve convergence compared to ordinary GEE, while showing a similar or even better performance in terms of accuracy of coefficient estimates and predictions. Penalized GEE often slightly outperforms the augmented GEE approaches, but this comes at the cost of a higher burden of implementation. CONCLUSIONS: When fitting marginal logistic regression models using GEE on sparse data we recommend to apply penalized GEE if one has access to a suitable software implementation and single-step augmented GEE otherwise.

Evaluating methods for Lasso selective inference in biomedical research: a comparative simulation study.

BACKGROUND: Variable selection for regression models plays a key role in the analysis of biomedical data. However, inference after selection is not covered by classical statistical frequentist theory, which assumes a fixed set of covariates in the model. This leads to over-optimistic selection and replicability issues. METHODS: We compared proposals for selective inference targeting the submodel parameters of the Lasso and its extension, the adaptive Lasso: sample splitting, selective inference conditional on the Lasso selection (SI), and universally valid post-selection inference (PoSI). We studied the properties of the proposed selective confidence intervals available via R software packages using a neutral simulation study inspired by real data commonly seen in biomedical studies. Furthermore, we present an exemplary application of these methods to a publicly available dataset to discuss their practical usability. RESULTS: Frequentist properties of selective confidence intervals by the SI method were generally acceptable, but the claimed selective coverage levels were not attained in all scenarios, in particular with the adaptive Lasso. The actual coverage of the extremely conservative PoSI method exceeded the nominal levels, and this method also required the greatest computational effort. Sample splitting achieved acceptable actual selective coverage levels, but the method is inefficient and leads to less accurate point estimates. The choice of inference method had a large impact on the resulting interval estimates, thereby necessitating that the user is acutely aware of the goal of inference in order to interpret and communicate the results. CONCLUSIONS: Despite violating nominal coverage levels in some scenarios, selective inference conditional on the Lasso selection is our recommended approach for most cases. If simplicity is strongly favoured over efficiency, then sample splitting is an alternative. If only few predictors undergo variable selection (i.e. up to 5) or the avoidance of false positive claims of significance is a concern, then the conservative approach of PoSI may be useful. For the adaptive Lasso, SI should be avoided and only PoSI and sample splitting are recommended. In summary, we find selective inference useful to assess the uncertainties in the importance of individual selected predictors for future applications.

Individual-specific networks for prediction modelling - A scoping review of methods.

BACKGROUND: Recent advances in biotechnology enable the acquisition of high-dimensional data on individuals, posing challenges for prediction models which traditionally use covariates such as clinical patient characteristics. Alternative forms of covariate representations for the features derived from these modern data modalities should be considered that can utilize their intrinsic interconnection. The connectivity information between these features can be represented as an individual-specific network defined by a set of nodes and edges, the strength of which can vary from individual to individual. Global or local graph-theoretical features describing the network may constitute potential prognostic biomarkers instead of or in addition to traditional covariates and may replace the often unsuccessful search for individual biomarkers in a high-dimensional predictor space. METHODS: We conducted a scoping review to identify, collate and critically appraise the state-of-art in the use of individual-specific networks for prediction modelling in medicine and applied health research, published during 2000-2020 in the electronic databases PubMed, Scopus and Embase. RESULTS: Our scoping review revealed the main application areas namely neurology and pathopsychology, followed by cancer research, cardiology and pathology (N = 148). Network construction was mainly based on Pearson correlation coefficients of repeated measurements, but also alternative approaches (e.g. partial correlation, visibility graphs) were found. For covariates measured only once per individual, network construction was mostly based on quantifying an individual's contribution to the overall group-level structure. Despite the multitude of identified methodological approaches for individual-specific network inference, the number of studies that were intended to enable the prediction of clinical outcomes for future individuals was quite limited, and most of the models served as proof of concept that network characteristics can in principle be useful for prediction. CONCLUSION: The current body of research clearly demonstrates the value of individual-specific network analysis for prediction modelling, but it has not yet been considered as a general tool outside the current areas of application. More methodological research is still needed on well-founded strategies for network inference, especially on adequate network sparsification and outcome-guided graph-theoretical feature extraction and selection, and on how networks can be exploited efficiently for prediction modelling.

Does the relative importance of the OxCAP-MH's capability items differ according to mental ill-health experience?

BACKGROUND: Some capability dimensions may be more important than others in determining someone's well-being, and these preferences might be dependent on ill-health experience. This study aimed to explore the relative preference weights of the 16 items of the German language version of the OxCAP-MH (Oxford Capability questionnaire-Mental Health) capability instrument and their differences across cohorts with alternative levels of mental ill-health experience. METHODS: A Best-Worst-Scaling (BWS) survey was conducted in Austria among 1) psychiatric patients (direct mental ill-health experience), 2) (mental) healthcare experts (indirect mental ill-health experience), and 3) primary care patients with no mental ill-health experience. Relative importance scores for each item of the German OxCAP-MH instrument were calculated using Hierarchical Bayes estimation. Rank analysis and multivariable linear regression analysis with robust standard errors were used to explore the relative importance of the OxCAP-MH items across the three cohorts. RESULTS: The study included 158 participants with complete cases and acceptable fit statistic. The relative importance scores for the full cohort ranged from 0.76 to 15.72. Findings of the BWS experiment indicated that the items Self-determination and Limitation in daily activities were regarded as the most important for all three cohorts. Freedom of expression was rated significantly less important by psychiatric patients than by the other two cohorts, while Having suitable accommodation appeared significantly less important by the expert cohort. There were no further significant differences in the relative preference weights of OxCAP-MH items between the cohorts or according to gender. CONCLUSIONS: Our study indicates significant between-item but limited mental ill-health related heterogeneity in the relative preference weights of the different capability items within the OxCAP-MH. The findings support the future development of preference-based value sets elicited from the general population for comparative economic evaluation purposes.

A comparison of full model specification and backward elimination of potential confounders when estimating marginal and conditional causal effects on binary outcomes from observational data.

A common view in epidemiology is that automated confounder selection methods, such as backward elimination, should be avoided as they can lead to biased effect estimates and underestimation of their variance. Nevertheless, backward elimination remains regularly applied. We investigated if and under which conditions causal effect estimation in observational studies can improve by using backward elimination on a prespecified set of potential confounders. An expression was derived that quantifies how variable omission relates to bias and variance of effect estimators. Additionally, 3960 scenarios were defined and investigated by simulations comparing bias and mean squared error (MSE) of the conditional log odds ratio, log(cOR), and the marginal log risk ratio, log(mRR), between full models including all prespecified covariates and backward elimination of these covariates. Applying backward elimination resulted in a mean bias of 0.03 for log(cOR) and 0.02 for log(mRR), compared to 0.56 and 0.52 for log(cOR) and log(mRR), respectively, for a model without any covariate adjustment, and no bias for the full model. In less than 3% of the scenarios considered, the MSE of the log(cOR) or log(mRR) was slightly lower (max 3%) when backward elimination was used compared to the full model. When an initial set of potential confounders can be specified based on background knowledge, there is minimal added value of backward elimination. We advise not to use it and otherwise to provide ample arguments supporting its use.

Using Background Knowledge from Preceding Studies for Building a Random Forest Prediction Model: A Plasmode Simulation Study.

There is an increasing interest in machine learning (ML) algorithms for predicting patient outcomes, as these methods are designed to automatically discover complex data patterns. For example, the random forest (RF) algorithm is designed to identify relevant predictor variables out of a large set of candidates. In addition, researchers may also use external information for variable selection to improve model interpretability and variable selection accuracy, thereby prediction quality. However, it is unclear to which extent, if at all, RF and ML methods may benefit from external information. In this paper, we examine the usefulness of external information from prior variable selection studies that used traditional statistical modeling approaches such as the Lasso, or suboptimal methods such as univariate selection. We conducted a plasmode simulation study based on subsampling a data set from a pharmacoepidemiologic study with nearly 200,000 individuals, two binary outcomes and 1152 candidate predictor (mainly sparse binary) variables. When the scope of candidate predictors was reduced based on external knowledge RF models achieved better calibration, that is, better agreement of predictions and observed outcome rates. However, prediction quality measured by cross-entropy, AUROC or the Brier score did not improve. We recommend appraising the methodological quality of studies that serve as an external information source for future prediction model development.

Interval cancer after colonoscopy in the Austrian National Screening Programme: influence of physician and patient factors.

OBJECTIVE: Postscreening colorectal cancer (PSCRC) after screening colonoscopy is associated with endoscopists' performance and characteristics of resected lesions. Prior studies have shown that adenoma detection rate (ADR) is a decisive factor for PSCRC, but correlations with other parameters need further analysis and ADR may change over time. DESIGN: Cohort study including individuals undergoing screening colonoscopy between 1/2008 and 12/2019 performed by physicians participating in a quality assurance programme in Austria. Data were linked with hospitalisation data for the diagnosis of PSCRC (defined as CRC diagnosis >6 months after colonoscopy). ADR was defined dynamically in relation to the time point of subsequent colonoscopies; high-risk groups of patients were those with an adenoma ≥10 mm, or with high-grade dysplasia, or villous or tubulovillous histology, or a serrated lesion ≥10 mm or with dysplasia, or colonoscopies with ≥3 lesions. Main outcome was PSCRC for each risk group (negative colonoscopy, hyperplastic polyps, low-risk and high-risk group of patients) after colonoscopy by endoscopists with an ADR <20% compared with endoscopists with an ADR ≥20%. RESULTS: 352 685 individuals were included in the study (51.0% women, median age 60 years) of which 10.5% were classified as high-risk group. During a median follow-up of 55.4 months, 241 (0.06%) PSCRC were identified; of 387 participating physicians, 19.6% had at least one PSCRC (8.4% two or more). While higher endoscopist ADR decreased PSCRC incidence (HR per 1% increase 0.97, 95% CI 0.95 to 0.98), affiliation to the high-risk group of patients was also associated with higher PSCRC incidence (HR 3.27, 95% CI 2.36 to 4.00). Similar correlations were seen with regards to high-risk, and advanced adenomas. The risk for PSCRC was significantly higher after colonoscopy by an endoscopist with an ADR <20% as compared with an endoscopist with an ADR ≥20% in patients after negative colonoscopy (HR 2.01, 95% CI 1.35 to 3.0, p<0.001) and for the high-risk group of patients (HR 2.51, 95% CI 1.49 to 4.22, p<0.001). CONCLUSION: A dynamic calculation of the ADR takes into account changes over time but confirms the correlation of ADR and interval cancer. Both lesion characteristics and endoscopists ADR may play a similar role for the risk of PSCRC. This should be considered in deciding about appropriate surveillance intervals in the future.

Selection of variables for multivariable models: Opportunities and limitations in quantifying model stability by resampling.

Statistical models are often fitted to obtain a concise description of the association of an outcome variable with some covariates. Even if background knowledge is available to guide preselection of covariates, stepwise variable selection is commonly applied to remove irrelevant ones. This practice may introduce additional variability and selection is rarely certain. However, these issues are often ignored and model stability is not questioned. Several resampling-based measures were proposed to describe model stability, including variable inclusion frequencies (VIFs), model selection frequencies, relative conditional bias (RCB), and root mean squared difference ratio (RMSDR). The latter two were recently proposed to assess bias and variance inflation induced by variable selection. Here, we study the consistency and accuracy of resampling estimates of these measures and the optimal choice of the resampling technique. In particular, we compare subsampling and bootstrapping for assessing stability of linear, logistic, and Cox models obtained by backward elimination in a simulation study. Moreover, we exemplify the estimation and interpretation of all suggested measures in a study on cardiovascular risk. The VIF and the model selection frequency are only consistently estimated in the subsampling approach. By contrast, the bootstrap is advantageous in terms of bias and precision for estimating the RCB as well as the RMSDR. Though, unbiased estimation of the latter quantity requires independence of covariates, which is rarely encountered in practice. Our study stresses the importance of addressing model stability after variable selection and shows how to cope with it.

To tune or not to tune, a case study of ridge logistic regression in small or sparse datasets.

BACKGROUND: For finite samples with binary outcomes penalized logistic regression such as ridge logistic regression has the potential of achieving smaller mean squared errors (MSE) of coefficients and predictions than maximum likelihood estimation. There is evidence, however, that ridge logistic regression can result in highly variable calibration slopes in small or sparse data situations. METHODS: In this paper, we elaborate this issue further by performing a comprehensive simulation study, investigating the performance of ridge logistic regression in terms of coefficients and predictions and comparing it to Firth's correction that has been shown to perform well in low-dimensional settings. In addition to tuned ridge regression where the penalty strength is estimated from the data by minimizing some measure of the out-of-sample prediction error or information criterion, we also considered ridge regression with pre-specified degree of shrinkage. We included 'oracle' models in the simulation study in which the complexity parameter was chosen based on the true event probabilities (prediction oracle) or regression coefficients (explanation oracle) to demonstrate the capability of ridge regression if truth was known. RESULTS: Performance of ridge regression strongly depends on the choice of complexity parameter. As shown in our simulation and illustrated by a data example, values optimized in small or sparse datasets are negatively correlated with optimal values and suffer from substantial variability which translates into large MSE of coefficients and large variability of calibration slopes. In contrast, in our simulations pre-specifying the degree of shrinkage prior to fitting led to accurate coefficients and predictions even in non-ideal settings such as encountered in the context of rare outcomes or sparse predictors. CONCLUSIONS: Applying tuned ridge regression in small or sparse datasets is problematic as it results in unstable coefficients and predictions. In contrast, determining the degree of shrinkage according to some meaningful prior assumptions about true effects has the potential to reduce bias and stabilize the estimates.

Statistical model building: Background "knowledge" based on inappropriate preselection causes misspecification.

BACKGROUND: Statistical model building requires selection of variables for a model depending on the model's aim. In descriptive and explanatory models, a common recommendation often met in the literature is to include all variables in the model which are assumed or known to be associated with the outcome independent of their identification with data driven selection procedures. An open question is, how reliable this assumed "background knowledge" truly is. In fact, "known" predictors might be findings from preceding studies which may also have employed inappropriate model building strategies. METHODS: We conducted a simulation study assessing the influence of treating variables as "known predictors" in model building when in fact this knowledge resulting from preceding studies might be insufficient. Within randomly generated preceding study data sets, model building with variable selection was conducted. A variable was subsequently considered as a "known" predictor if a predefined number of preceding studies identified it as relevant. RESULTS: Even if several preceding studies identified a variable as a "true" predictor, this classification is often false positive. Moreover, variables not identified might still be truly predictive. This especially holds true if the preceding studies employed inappropriate selection methods such as univariable selection. CONCLUSIONS: The source of "background knowledge" should be evaluated with care. Knowledge generated on preceding studies can cause misspecification.

The roles of predictors in cardiovascular risk models - a question of modeling culture?

BACKGROUND: While machine learning (ML) algorithms may predict cardiovascular outcomes more accurately than statistical models, their result is usually not representable by a transparent formula. Hence, it is often unclear how specific values of predictors lead to the predictions. We aimed to demonstrate with graphical tools how predictor-risk relations in cardiovascular risk prediction models fitted by ML algorithms and by statistical approaches may differ, and how sample size affects the stability of the estimated relations. METHODS: We reanalyzed data from a large registry of 1.5 million participants in a national health screening program. Three data analysts developed analytical strategies to predict cardiovascular events within 1 year from health screening. This was done for the full data set and with gradually reduced sample sizes, and each data analyst followed their favorite modeling approach. Predictor-risk relations were visualized by partial dependence and individual conditional expectation plots. RESULTS: When comparing the modeling algorithms, we found some similarities between these visualizations but also occasional divergence. The smaller the sample size, the more the predictor-risk relation depended on the modeling algorithm used, and also sampling variability played an increased role. Predictive performance was similar if the models were derived on the full data set, whereas smaller sample sizes favored simpler models. CONCLUSION: Predictor-risk relations from ML models may differ from those obtained by statistical models, even with large sample sizes. Hence, predictors may assume different roles in risk prediction models. As long as sample size is sufficient, predictive accuracy is not largely affected by the choice of algorithm.