RESUMO
Statistical regression models are used for predicting outcomes based on the values of some predictor variables or for describing the association of an outcome with predictors. With a data set at hand, a regression model can be easily fit with standard software packages. This bears the risk that data analysts may rush to perform sophisticated analyses without sufficient knowledge of basic properties, associations in and errors of their data, leading to wrong interpretation and presentation of the modeling results that lacks clarity. Ignorance about special features of the data such as redundancies or particular distributions may even invalidate the chosen analysis strategy. Initial data analysis (IDA) is prerequisite to regression analyses as it provides knowledge about the data needed to confirm the appropriateness of or to refine a chosen model building strategy, to interpret the modeling results correctly, and to guide the presentation of modeling results. In order to facilitate reproducibility, IDA needs to be preplanned, an IDA plan should be included in the general statistical analysis plan of a research project, and results should be well documented. Biased statistical inference of the final regression model can be minimized if IDA abstains from evaluating associations of outcome and predictors, a key principle of IDA. We give advice on which aspects to consider in an IDA plan for data screening in the context of regression modeling to supplement the statistical analysis plan. We illustrate this IDA plan for data screening in an example of a typical diagnostic modeling project and give recommendations for data visualizations.
Assuntos
Modelos Estatísticos , Humanos , Análise de Regressão , Interpretação Estatística de Dados , Análise Multivariada , Reprodutibilidade dos Testes , Software , Análise de DadosRESUMO
Researchers often perform data-driven variable selection when modeling the associations between an outcome and multiple independent variables in regression analysis. Variable selection may improve the interpretability, parsimony and/or predictive accuracy of a model. Yet variable selection can also have negative consequences, such as false exclusion of important variables or inclusion of noise variables, biased estimation of regression coefficients, underestimated standard errors and invalid confidence intervals, as well as model instability. While the potential advantages and disadvantages of variable selection have been discussed in the literature for decades, few large-scale simulation studies have neutrally compared data-driven variable selection methods with respect to their consequences for the resulting models. We present the protocol for a simulation study that will evaluate different variable selection methods: forward selection, stepwise forward selection, backward elimination, augmented backward elimination, univariable selection, univariable selection followed by backward elimination, and penalized likelihood approaches (Lasso, relaxed Lasso, adaptive Lasso). These methods will be compared with respect to false inclusion and/or exclusion of variables, consequences on bias and variance of the estimated regression coefficients, the validity of the confidence intervals for the coefficients, the accuracy of the estimated variable importance ranking, and the predictive performance of the selected models. We consider both linear and logistic regression in a low-dimensional setting (20 independent variables with 10 true predictors and 10 noise variables). The simulation will be based on real-world data from the National Health and Nutrition Examination Survey (NHANES). Publishing this study protocol ahead of performing the simulation increases transparency and allows integrating the perspective of other experts into the study design.
Assuntos
Simulação por Computador , Humanos , Análise de Regressão , Análise MultivariadaRESUMO
Background: Venous thromboembolism (VTE) is associated with various long-term complications. Objectives: We aimed to investigate the association of clinical characteristics at VTE diagnosis with functional limitations 3 and 12 months afterward. Methods: We conducted a prospective cohort study of VTE patients, excluding patients with cancer, pregnancy, and postpartum period. Functional limitations were assessed with the post-VTE functional status (PVFS) scale (range, 0-4) within 21 days of diagnosis, after 3 and 12 months (prospectively), and 1 month before diagnosis (retrospectively). Twelve-month follow-up was only performed in patients on anticoagulation. We fitted 2 proportional odds logistic regression models for the 3- and 12-month follow-ups and computed odds ratios (ORs) with 95% bootstrap percentile confidence intervals (CIs). Results: We included 307 patients (42% female, median age 55.6 years) with a median (IQR) PVFS scale grade of 2 (2-3) at study inclusion and 0 (0-0) before diagnosis. After 3 months, PVFS scale grade in 269 patients was 1 (0-2). Female sex (OR, 2.15; 95% CI, 1.26-4.14), body mass index (OR per 1 kg/m2 increase, 1.05; 95% CI, 1.00-1.10), functional limitations at baseline, and older age were associated with functional limitations. After 12 months, PVFS scale grade in 124 patients was 1 (0-2). Female sex (OR, 4.47; 95% CI, 2.11-16.00), history of cardiovascular/pulmonary disease (OR, 2.36; 95% CI, 1.01-6.89), and functional limitations at baseline were associated with functional limitations. Conclusion: Functional limitations in VTE patients improved 3 and 12 months after diagnosis but did not return to pre-VTE values. We identified clinical characteristics that could help identify patients at risk of persisting functional limitations after VTE.
RESUMO
Statistical techniques are needed to analyze data structures with complex dependencies such that clinically useful information can be extracted. Individual-specific networks, which capture dependencies in complex biological systems, are often summarized by graph-theoretical features. These features, which lend themselves to outcome modeling, can be subject to high variability due to arbitrary decisions in network inference and noise. Correlation-based adjacency matrices often need to be sparsified before meaningful graph-theoretical features can be extracted, requiring the data analysts to determine an optimal threshold. To address this issue, we propose to incorporate a flexible weighting function over the full range of possible thresholds to capture the variability of graph-theoretical features over the threshold domain. The potential of this approach, which extends concepts from functional data analysis to a graph-theoretical setting, is explored in a plasmode simulation study using real functional magnetic resonance imaging (fMRI) data from the Autism Brain Imaging Data Exchange (ABIDE) Preprocessed initiative. The simulations show that our modeling approach yields accurate estimates of the functional form of the weight function, improves inference efficiency, and achieves a comparable or reduced root mean square prediction error compared to competitor modeling approaches. This assertion holds true in settings where both complex functional forms underlie the outcome-generating process and a universal threshold value is employed. We demonstrate the practical utility of our approach by using resting-state fMRI data to predict biological age in children. Our study establishes the flexible modeling approach as a statistically principled, serious competitor to ad-hoc methods with superior performance.
Assuntos
Simulação por Computador , Imageamento por Ressonância Magnética , Humanos , Criança , Encéfalo/diagnóstico por imagem , Modelos Estatísticos , Transtorno AutísticoRESUMO
Quality indicators serve as a tool to measure and improve evidence-based health care. Often the transition from inpatient to outpatient care is not sufficiently included. The focus of this work was to develop and to evaluate methods to define relevant cross-sector quality indicators based on Austrian claims data, using myocardial infarction as tracer.
Assuntos
Infarto do Miocárdio , Indicadores de Qualidade em Assistência à Saúde , Humanos , Assistência Ambulatorial , Pacientes Internados , Infarto do Miocárdio/terapiaRESUMO
Although new biostatistical methods are published at a very high rate, many of these developments are not trustworthy enough to be adopted by the scientific community. We propose a framework to think about how a piece of methodological work contributes to the evidence base for a method. Similar to the well-known phases of clinical research in drug development, we propose to define four phases of methodological research. These four phases cover (I) proposing a new methodological idea while providing, for example, logical reasoning or proofs, (II) providing empirical evidence, first in a narrow target setting, then (III) in an extended range of settings and for various outcomes, accompanied by appropriate application examples, and (IV) investigations that establish a method as sufficiently well-understood to know when it is preferred over others and when it is not; that is, its pitfalls. We suggest basic definitions of the four phases to provoke thought and discussion rather than devising an unambiguous classification of studies into phases. Too many methodological developments finish before phase III/IV, but we give two examples with references. Our concept rebalances the emphasis to studies in phases III and IV, that is, carefully planned method comparison studies and studies that explore the empirical properties of existing methods in a wider range of problems.
Assuntos
Bioestatística , Projetos de PesquisaRESUMO
BACKGROUND AND AIMS: Patients with unprovoked venous thromboembolism (VTE) have a high recurrence risk, and guidelines suggest extended-phase anticoagulation. Many patients never experience recurrence but are exposed to bleeding. The aim of this study was to assess the performance of the Vienna Prediction Model (VPM) and to evaluate if the VPM accurately identifies these patients. METHODS: In patients with unprovoked VTE, the VPM was performed 3 weeks after anticoagulation withdrawal. Those with a predicted 1-year recurrence risk of ≤5.5% were prospectively followed. Study endpoint was recurrent VTE over 2 years. RESULTS: A total of 818 patients received anticoagulation for a median of 3.9 months. 520 patients (65%) had a predicted annual recurrence risk of ≤5.5%. During a median time of 23.9 months, 52 patients had non-fatal recurrence. The recurrence risk was 5.2% [95% confidence interval (CI) 3.2-7.2] at 1 year and 11.2% (95% CI 8.3-14) at 2 years. Model calibration was adequate after 1 year. The VPM underestimated the recurrence risk of patients with a 2-year recurrence rate of >5%. In a post-hoc analysis, the VPM's baseline hazard was recalibrated. Bootstrap validation confirmed an ideal ratio of observed and expected recurrence events. The recurrence risk was highest in men with proximal deep-vein thrombosis or pulmonary embolism and lower in women regardless of the site of incident VTE. CONCLUSIONS: In this prospective evaluation of the performance of the VPM, the 1-year rate of recurrence in patients with unprovoked VTE was 5.2%. Recalibration improved identification of patients at low recurrence risk and stratification into distinct low-risk categories.
Assuntos
Embolia Pulmonar , Tromboembolia Venosa , Masculino , Humanos , Feminino , Tromboembolia Venosa/epidemiologia , Estudos Prospectivos , Anticoagulantes/uso terapêutico , Recidiva , Fatores de RiscoRESUMO
Accurate estimation of chronic kidney disease (CKD) progression risk is vital for clinical decision-making. Existing risk equations lack validation in pediatric CKD populations. Ng et al. developed new risk equations using the CKD in Children and European Study Consortium for Chronic Kidney Disorders Affecting Pediatric Patients cohorts. The elementary model, incorporating estimated glomerular filtration rate, urine protein-creatinine ratio, and diagnosis, exhibited excellent discrimination and calibration at external validation. External validation of enriched models is pending. The equations have the potential to aid pediatric CKD centers in patient counseling and care planning.
Assuntos
Insuficiência Renal Crônica , Humanos , Criança , Taxa de Filtração Glomerular , Testes de Função Renal , Insuficiência Renal Crônica/diagnóstico , Creatinina , Progressão da Doença , RimRESUMO
Objective: This retrospective study evaluates the performance of UK National Institute for Health and Care Excellence (NICE) Guidelines on management of ruptured abdominal aortic aneurysms in a "real world setting" by emulating a hypothetical target trial with data from two European Aortic Centers. Methods: Clinical data was retrospectively collected for all patients who had undergone ruptured endovascular aneurysm repair (rEVAR) and ruptured open surgical repair (rOSR). Survival analysis was performed comparing NICE compliance to usual care strategy. NICE compliers were defined as: female patients undergoing rEVAR; male patients >70 years old undergoing rEVAR; and male patients ≤70 years old undergoing rOSR. Hemodynamic instability was considered additionally. Results: This multicenter study included 298 patients treated for rAAA. The majority of patients were treated with rOSR (186 rOSR vs. 112 rEVAR). Overall, 184 deaths (68 [37%] with rEVAR and 116 [63%] with rOSR) were observed during the study period. Overall survival under usual care was 69.2% at 30 days, 56.5% at one year, and 42.4% at 5 years. NICE compliance gave survival outcomes of 73.1% at 30 days, 60.2% at 1 year and 42.9% at 5 years. The risk ratios at these time points, comparing NICE-compliance to usual care, were 0.88, 0.92 and 0.99, respectively. Conclusions: We support NICE recommendations to manage men below the age of 71 years and hemodynamic stability with rOSR. There was a slight survival advantage for NICE compliers overall, in men >70 years and women of all ages.
RESUMO
BACKGROUND: Kidney transplantation is the preferred treatment for eligible patients with kidney failure who need renal replacement therapy. However, it remains unclear whether the anticipated survival benefit from kidney transplantation is different for women and men. METHODS: We included all dialysis patients recorded in the Austrian Dialysis and Transplant Registry who were waitlisted for their first kidney transplant between 2000 and 2018. In order to estimate the causal effect of kidney transplantation on 10-year restricted mean survival time, we mimicked a series of controlled clinical trials and applied inverse probability of treatment and censoring weighted sequential Cox models. RESULTS: This study included 4408 patients (33% female) with a mean age of 52 years. Glomerulonephritis was the most common primary renal disease both in women (27%) and men (28%). Kidney transplantation led to a gain of 2.22 years (95% CI 1.88 to 2.49) compared with dialysis over a 10-year follow-up. The effect was smaller in women (1.95 years, 95% CI 1.38 to 2.41) than in men (2.35 years, 95% CI 1.92 to 2.70) due to a better survival on dialysis. Across ages the survival benefit of transplantation over a follow-up of 10 years was smaller in younger women and men and increased with age, showing a peak for both women and men aged about 60 years. CONCLUSIONS: There were few differences in survival benefit by transplantation between females and males. Females had better survival than males on the waitlist receiving dialysis and similar survival to males after transplantation.
Assuntos
Falência Renal Crônica , Transplante de Rim , Humanos , Masculino , Feminino , Pessoa de Meia-Idade , Diálise Renal , Falência Renal Crônica/cirurgia , Estudos Retrospectivos , Caracteres SexuaisRESUMO
TP53 is the most commonly mutated gene in cancer and has been shown to form amyloid-like aggregates, similar to key proteins in neurodegenerative diseases. Nonetheless, the clinical implications of p53 aggregation remain unclear. Here, we investigated the presence and clinical relevance of p53 aggregates in serous ovarian cancer (OC). Using the p53-Seprion-ELISA, p53 aggregates were detected in 46 out of 81 patients, with a detection rate of 84.3% in patients with missense mutations. High p53 aggregation was associated with prolonged progression-free survival. We found associations of overall survival with p53 aggregates, but they did not reach statistical significance. Interestingly, p53 aggregation was significantly associated with elevated levels of p53 autoantibodies and increased apoptosis, suggesting that high levels of p53 aggregates may trigger an immune response and/or exert a cytotoxic effect. To conclude, for the first time, we demonstrated that p53 aggregates are an independent prognostic marker in serous OC. P53-targeted therapies based on the amount of these aggregates may improve the patient's prognosis.
Assuntos
Cistadenocarcinoma Seroso , Neoplasias Ovarianas , Humanos , Feminino , Proteína Supressora de Tumor p53/genética , Proteína Supressora de Tumor p53/metabolismo , Neoplasias Ovarianas/metabolismo , Prognóstico , Carcinoma Epitelial do Ovário , Cistadenocarcinoma Seroso/genética , Biomarcadores , MutaçãoRESUMO
BACKGROUND: The performance of models for binary outcomes can be described by measures such as the concordance statistic (c-statistic, area under the curve), the discrimination slope, or the Brier score. At internal validation, data resampling techniques, e.g., cross-validation, are frequently employed to correct for optimism in these model performance criteria. Especially with small samples or rare events, leave-one-out cross-validation is a popular choice. METHODS: Using simulations and a real data example, we compared the effect of different resampling techniques on the estimation of c-statistics, discrimination slopes, and Brier scores for three estimators of logistic regression models, including the maximum likelihood and two maximum penalized likelihood estimators. RESULTS: Our simulation study confirms earlier studies reporting that leave-one-out cross-validated c-statistics can be strongly biased towards zero. In addition, our study reveals that this bias is even more pronounced for model estimators shrinking estimated probabilities towards the observed event fraction, such as ridge regression. Leave-one-out cross-validation also provided pessimistic estimates of the discrimination slope but nearly unbiased estimates of the Brier score. CONCLUSIONS: We recommend to use leave-pair-out cross-validation, fivefold cross-validation with repetitions, the enhanced or the .632+ bootstrap to estimate c-statistics, and leave-pair-out or fivefold cross-validation to estimate discrimination slopes.
RESUMO
Importance: Type 2 diabetes increases the risk of progressive diabetic kidney disease, but reliable prediction tools that can be used in clinical practice and aid in patients' understanding of disease progression are currently lacking. Objective: To develop and externally validate a model to predict future trajectories in estimated glomerular filtration rate (eGFR) in adults with type 2 diabetes and chronic kidney disease using data from 3 European multinational cohorts. Design, Setting, and Participants: This prognostic study used baseline and follow-up information collected between February 2010 and December 2019 from 3 prospective multinational cohort studies: PROVALID (Prospective Cohort Study in Patients with Type 2 Diabetes Mellitus for Validation of Biomarkers), GCKD (German Chronic Kidney Disease), and DIACORE (Diabetes Cohorte). A total of 4637 adult participants (aged 18-75 years) with type 2 diabetes and mildly to moderately impaired kidney function (baseline eGFR of ≥30 mL/min/1.73 m2) were included. Data were analyzed between June 30, 2021, and January 31, 2023. Main Outcomes and Measures: Thirteen variables readily available from routine clinical care visits (age, sex, body mass index; smoking status; hemoglobin A1c [mmol/mol and percentage]; hemoglobin, and serum cholesterol levels; mean arterial pressure, urinary albumin-creatinine ratio, and intake of glucose-lowering, blood-pressure lowering, or lipid-lowering medication) were selected as predictors. Repeated eGFR measurements at baseline and follow-up visits were used as the outcome. A linear mixed-effects model for repeated eGFR measurements at study entry up to the last recorded follow-up visit (up to 5 years after baseline) was fit and externally validated. Results: Among 4637 adults with type 2 diabetes and chronic kidney disease (mean [SD] age at baseline, 63.5 [9.1] years; 2680 men [57.8%]; all of White race), 3323 participants from the PROVALID and GCKD studies (mean [SD] age at baseline, 63.2 [9.3] years; 1864 men [56.1%]) were included in the model development cohort, and 1314 participants from the DIACORE study (mean [SD] age at baseline, 64.5 [8.3] years; 816 men [62.1%]) were included in the external validation cohort, with a mean (SD) follow-up of 5.0 (0.6) years. Updating the random coefficient estimates with baseline eGFR values yielded improved predictive performance, which was particularly evident in the visual inspection of the calibration curve (calibration slope at 5 years: 1.09; 95% CI, 1.04-1.15). The prediction model had good discrimination in the validation cohort, with the lowest C statistic at 5 years after baseline (0.79; 95% CI, 0.77-0.80). The model also had predictive accuracy, with an R2 ranging from 0.70 (95% CI, 0.63-0.76) at year 1 to 0.58 (95% CI, 0.53-0.63) at year 5. Conclusions and Relevance: In this prognostic study, a reliable prediction model was developed and externally validated; the robust model was well calibrated and capable of predicting kidney function decline up to 5 years after baseline. The results and prediction model are publicly available in an accompanying web-based application, which may open the way for improved prediction of individual eGFR trajectories and disease progression.
Assuntos
Diabetes Mellitus Tipo 2 , Insuficiência Renal Crônica , Masculino , Adulto , Humanos , Diabetes Mellitus Tipo 2/complicações , Taxa de Filtração Glomerular , Estudos Prospectivos , Progressão da DoençaRESUMO
Randomization is an effective design option to prevent bias from confounding in the evaluation of the causal effect of interventions on outcomes. However, in some cases, randomization is not possible, making subsequent adjustment for confounders essential to obtain valid results. Several methods exist to adjust for confounding, with multivariable modeling being among the most widely used. The main challenge is to determine which variables should be included in the causal model and to specify appropriate functional relations for continuous variables in the model. While the statistical literature gives a variety of recommendations on how to build multivariable regression models in practice, this guidance is often unknown to applied researchers. We set out to investigate the current practice of explanatory regression modeling to control confounding in the field of cardiac rehabilitation, for which mainly non-randomized observational studies are available. In particular, we conducted a systematic methods review to identify and compare statistical methodology with respect to statistical model building in the context of the existing recent systematic review CROS-II, which evaluated the prognostic effect of cardiac rehabilitation. CROS-II identified 28 observational studies, which were published between 2004 and 2018. Our methods review revealed that 24 (86%) of the included studies used methods to adjust for confounding. Of these, 11 (46%) mentioned how the variables were selected and two studies (8%) considered functional forms for continuous variables. The use of background knowledge for variable selection was barely reported and data-driven variable selection methods were applied frequently. We conclude that in the majority of studies, the methods used to develop models to investigate the effect of cardiac rehabilitation on outcomes do not meet common criteria for appropriate statistical model building and that reporting often lacks precision.
Assuntos
Reabilitação Cardíaca , Humanos , Modelos Teóricos , Modelos EstatísticosRESUMO
BACKGROUND: The coronavirus disease 2019 (COVID-19) pandemic demands reliable prognostic models for estimating the risk of long COVID. We developed and validated a prediction model to estimate the probability of known common long COVID symptoms at least 60 days after acute COVID-19. METHODS: The prognostic model was built based on data from a multicentre prospective Swiss cohort study. Included were adult patients diagnosed with COVID-19 between February and December 2020 and treated as outpatients, at ward or intensive/intermediate care unit. Perceived long-term health impairments, including reduced exercise tolerance/reduced resilience, shortness of breath and/or tiredness (REST), were assessed after a follow-up time between 60 and 425 days. The data set was split into a derivation and a geographical validation cohort. Predictors were selected out of twelve candidate predictors based on three methods, namely the augmented backward elimination (ABE) method, the adaptive best-subset selection (ABESS) method and model-based recursive partitioning (MBRP) approach. Model performance was assessed with the scaled Brier score, concordance c statistic and calibration plot. The final prognostic model was determined based on best model performance. RESULTS: In total, 2799 patients were included in the analysis, of which 1588 patients were in the derivation cohort and 1211 patients in the validation cohort. The REST prevalence was similar between the cohorts with 21.6% (n = 343) in the derivation cohort and 22.1% (n = 268) in the validation cohort. The same predictors were selected with the ABE and ABESS approach. The final prognostic model was based on the ABE and ABESS selected predictors. The corresponding scaled Brier score in the validation cohort was 18.74%, model discrimination was 0.78 (95% CI: 0.75 to 0.81), calibration slope was 0.92 (95% CI: 0.78 to 1.06) and calibration intercept was -0.06 (95% CI: -0.22 to 0.09). CONCLUSION: The proposed model was validated to identify COVID-19-infected patients at high risk for REST symptoms. Before implementing the prognostic model in daily clinical practice, the conduct of an impact study is recommended.
RESUMO
Importance: Kidney transplant is considered beneficial in terms of survival compared with continued dialysis for patients with kidney failure. However, randomized clinical trials are infeasible, and available evidence from cohort studies is at high risk of bias. Objective: To compare restricted mean survival times (RMSTs) between patients who underwent transplant and patients continuing dialysis across transplant candidate ages and depending on waiting time, applying target trial emulation methods. Design, Setting, and Participants: In this retrospective cohort study, patients aged 18 years or older appearing on the wait list for their first single-organ deceased donor kidney transplant between January 1, 2000, and December 31, 2018, in Austria were evaluated. Available data were obtained from the Austrian Dialysis and Transplant Registry and Eurotransplant and included repeated updates on wait-listing status and relevant covariates. Data were analyzed between August 1, 2019, and December 23, 2021. Exposures: A target trial was emulated in which patients were randomized to either receive the transplant immediately (treatment group) or to continue dialysis and never receive a transplant (control group) at each time an organ became available. Main Outcomes and Measures: The primary outcome was time from transplant allocation to death. Effect sizes in terms of RMSTs were obtained using a sequential Cox approach. Results: Among the 4445 included patients (2974 men [66.9%]; mean [SD] age, 52.2 [13.2] years), transplant was associated with increased survival time across all considered ages compared with continuing dialysis and remaining on the wait list within a 10-year follow-up. The estimated RMST differences were 0.57 years (95% CI, -0.14 to 1.84 years) at age 20 years, 3.01 years (95% CI, 2.50 to 3.54 years) at age 60 years, and 2.48 years (95% CI, 1.88 to 3.04 years) at age 70 years. The survival benefit for patients who underwent transplant across ages was independent of waiting time. Conclusions and Relevance: The findings of this study suggest that kidney transplant prolongs the survival time of persons with kidney failure across all candidate ages and waiting times.
Assuntos
Falência Renal Crônica , Transplante de Rim , Insuficiência Renal , Adulto , Idoso , Humanos , Falência Renal Crônica/cirurgia , Transplante de Rim/métodos , Masculino , Pessoa de Meia-Idade , Diálise Renal , Estudos Retrospectivos , Adulto JovemRESUMO
BACKGROUND: Variable selection for regression models plays a key role in the analysis of biomedical data. However, inference after selection is not covered by classical statistical frequentist theory, which assumes a fixed set of covariates in the model. This leads to over-optimistic selection and replicability issues. METHODS: We compared proposals for selective inference targeting the submodel parameters of the Lasso and its extension, the adaptive Lasso: sample splitting, selective inference conditional on the Lasso selection (SI), and universally valid post-selection inference (PoSI). We studied the properties of the proposed selective confidence intervals available via R software packages using a neutral simulation study inspired by real data commonly seen in biomedical studies. Furthermore, we present an exemplary application of these methods to a publicly available dataset to discuss their practical usability. RESULTS: Frequentist properties of selective confidence intervals by the SI method were generally acceptable, but the claimed selective coverage levels were not attained in all scenarios, in particular with the adaptive Lasso. The actual coverage of the extremely conservative PoSI method exceeded the nominal levels, and this method also required the greatest computational effort. Sample splitting achieved acceptable actual selective coverage levels, but the method is inefficient and leads to less accurate point estimates. The choice of inference method had a large impact on the resulting interval estimates, thereby necessitating that the user is acutely aware of the goal of inference in order to interpret and communicate the results. CONCLUSIONS: Despite violating nominal coverage levels in some scenarios, selective inference conditional on the Lasso selection is our recommended approach for most cases. If simplicity is strongly favoured over efficiency, then sample splitting is an alternative. If only few predictors undergo variable selection (i.e. up to 5) or the avoidance of false positive claims of significance is a concern, then the conservative approach of PoSI may be useful. For the adaptive Lasso, SI should be avoided and only PoSI and sample splitting are recommended. In summary, we find selective inference useful to assess the uncertainties in the importance of individual selected predictors for future applications.
Assuntos
Pesquisa Biomédica , Simulação por Computador , HumanosRESUMO
There is an increasing interest in machine learning (ML) algorithms for predicting patient outcomes, as these methods are designed to automatically discover complex data patterns. For example, the random forest (RF) algorithm is designed to identify relevant predictor variables out of a large set of candidates. In addition, researchers may also use external information for variable selection to improve model interpretability and variable selection accuracy, thereby prediction quality. However, it is unclear to which extent, if at all, RF and ML methods may benefit from external information. In this paper, we examine the usefulness of external information from prior variable selection studies that used traditional statistical modeling approaches such as the Lasso, or suboptimal methods such as univariate selection. We conducted a plasmode simulation study based on subsampling a data set from a pharmacoepidemiologic study with nearly 200,000 individuals, two binary outcomes and 1152 candidate predictor (mainly sparse binary) variables. When the scope of candidate predictors was reduced based on external knowledge RF models achieved better calibration, that is, better agreement of predictions and observed outcome rates. However, prediction quality measured by cross-entropy, AUROC or the Brier score did not improve. We recommend appraising the methodological quality of studies that serve as an external information source for future prediction model development.