Data-Driven Simulations to Assess the Impact of Study Imperfections in Time-to-Event Analyses.

Quantitative bias analysis (QBA) permits assessment of the expected impact of various imperfections of the available data on the results and conclusions of a particular real-world study. This article extends QBA methodology to multivariable time-to-event analyses with right-censored endpoints, possibly including time-varying exposures or covariates. The proposed approach employs data-driven simulations, which preserve important features of the data at hand while offering flexibility in controlling the parameters and assumptions that may affect the results. First, the steps required to perform data-driven simulations are described, and then two examples of real-world time-to-event analyses illustrate their implementation and the insights they may offer. The first example focuses on the omission of an important time-invariant predictor of the outcome in a prognostic study of cancer mortality, and permits separating the expected impact of confounding bias from non-collapsibility. The second example assesses how imprecise timing of an interval-censored event - ascertained only at sparse times of clinic visits - affects its estimated association with a time-varying drug exposure. The simulation results also provide a basis for comparing the performance of two alternative strategies for imputing the unknown event times in this setting. The R scripts that permit the reproduction of our examples are provided.

Raising awareness of uncertain choices in empirical data analysis: A teaching concept toward replicable research practices.

Throughout their education and when reading the scientific literature, students may get the impression that there is a unique and correct analysis strategy for every data analysis task and that this analysis strategy will always yield a significant and noteworthy result. This expectation conflicts with a growing realization that there is a multiplicity of possible analysis strategies in empirical research, which will lead to overoptimism and nonreplicable research findings if it is combined with result-dependent selective reporting. Here, we argue that students are often ill-equipped for real-world data analysis tasks and unprepared for the dangers of selectively reporting the most promising results. We present a seminar course intended for advanced undergraduates and beginning graduate students of data analysis fields such as statistics, data science, or bioinformatics that aims to increase the awareness of uncertain choices in the analysis of empirical data and present ways to deal with these choices through theoretical modules and practical hands-on sessions.

A comparison of hyperparameter tuning procedures for clinical prediction models: A simulation study.

Tuning hyperparameters, such as the regularization parameter in Ridge or Lasso regression, is often aimed at improving the predictive performance of risk prediction models. In this study, various hyperparameter tuning procedures for clinical prediction models were systematically compared and evaluated in low-dimensional data. The focus was on out-of-sample predictive performance (discrimination, calibration, and overall prediction error) of risk prediction models developed using Ridge, Lasso, Elastic Net, or Random Forest. The influence of sample size, number of predictors and events fraction on performance of the hyperparameter tuning procedures was studied using extensive simulations. The results indicate important differences between tuning procedures in calibration performance, while generally showing similar discriminative performance. The one-standard-error rule for tuning applied to cross-validation (1SE CV) often resulted in severe miscalibration. Standard non-repeated and repeated cross-validation (both 5-fold and 10-fold) performed similarly well and outperformed the other tuning procedures. Bootstrap showed a slight tendency to more severe miscalibration than standard cross-validation-based tuning procedures. Differences between tuning procedures were larger for smaller sample sizes, lower events fractions and fewer predictors. These results imply that the choice of tuning procedure can have a profound influence on the predictive performance of prediction models. The results support the application of standard 5-fold or 10-fold cross-validation that minimizes out-of-sample prediction error. Despite an increased computational burden, we found no clear benefit of repeated over non-repeated cross-validation for hyperparameter tuning. We warn against the potentially detrimental effects on model calibration of the popular 1SE CV rule for tuning prediction models in low-dimensional settings.

Phases of methodological research in biostatistics-Building the evidence base for new methods.

Although new biostatistical methods are published at a very high rate, many of these developments are not trustworthy enough to be adopted by the scientific community. We propose a framework to think about how a piece of methodological work contributes to the evidence base for a method. Similar to the well-known phases of clinical research in drug development, we propose to define four phases of methodological research. These four phases cover (I) proposing a new methodological idea while providing, for example, logical reasoning or proofs, (II) providing empirical evidence, first in a narrow target setting, then (III) in an extended range of settings and for various outcomes, accompanied by appropriate application examples, and (IV) investigations that establish a method as sufficiently well-understood to know when it is preferred over others and when it is not; that is, its pitfalls. We suggest basic definitions of the four phases to provoke thought and discussion rather than devising an unambiguous classification of studies into phases. Too many methodological developments finish before phase III/IV, but we give two examples with references. Our concept rebalances the emphasis to studies in phases III and IV, that is, carefully planned method comparison studies and studies that explore the empirical properties of existing methods in a wider range of problems.

Explaining the optimistic performance evaluation of newly proposed methods: A cross-design validation experiment.

The constant development of new data analysis methods in many fields of research is accompanied by an increasing awareness that these new methods often perform better in their introductory paper than in subsequent comparison studies conducted by other researchers. We attempt to explain this discrepancy by conducting a systematic experiment that we call "cross-design validation of methods". In the experiment, we select two methods designed for the same data analysis task, reproduce the results shown in each paper, and then reevaluate each method based on the study design (i.e., datasets, competing methods, and evaluation criteria) that was used to show the abilities of the other method. We conduct the experiment for two data analysis tasks, namely cancer subtyping using multiomic data and differential gene expression analysis. Three of the four methods included in the experiment indeed perform worse when they are evaluated on the new study design, which is mainly caused by the different datasets. Apart from illustrating the many degrees of freedom existing in the assessment of a method and their effect on its performance, our experiment suggests that the performance discrepancies between original and subsequent papers may not only be caused by the nonneutrality of the authors proposing the new method but also by differences regarding the level of expertise and field of application. Authors of new methods should thus focus not only on a transparent and extensive evaluation but also on comprehensive method documentation that enables the correct use of their methods in subsequent studies.

Stereotactic radiosurgery versus whole-brain radiotherapy in patients with 4-10 brain metastases: A nonrandomized controlled trial.

BACKGROUND AND PURPOSE: There is no randomized evidence comparing whole-brain radiotherapy (WBRT) and stereotactic radiosurgery (SRS) in the treatment of multiple brain metastases. This prospective nonrandomized controlled single arm trial attempts to reduce the gap until prospective randomized controlled trial results are available. MATERIAL AND METHODS: We included patients with 4-10 brain metastases and ECOG performance status ≤ 2 from all histologies except small-cell lung cancer, germ cell tumors, and lymphoma. The retrospective WBRT-cohort was selected 2:1 from consecutive patients treated within 2012-2017. Propensity-score matching was performed to adjust for confounding factors such as sex, age, primary tumor histology, dsGPA score, and systemic therapy. SRS was performed using a LINAC-based single-isocenter technique employing prescription doses from 15-20Gyx1 at the 80% isodose line. The historical control consisted of equivalent WBRT dose regimens of either 3Gyx10 or 2.5Gyx14. RESULTS: Patients were recruited from 2017-2020, end of follow-up was July 1st, 2021. 40 patients were recruited to the SRS-cohort and 70 patients were eligible as controls in the WBRT-cohort. Median OS, and iPFS were 10.4 months (95%-CI 9.3-NA) and 7.1 months (95%-CI 3.9-14.2) for the SRS-cohort, and 6.5 months (95%-CI 4.9-10.4), and 5.9 months (95%-CI 4.1-8.8) for the WBRT-cohort, respectively. Differences were non-significant for OS (HR: 0.65; 95%-CI 0.40-1.05; P =.074) and iPFS (P =.28). No grade III toxicities were observed in the SRS-cohort. CONCLUSION: This trial did not meet its primary endpoint as the OS-improvement of SRS compared to WBRT was non-significant and thus superiority could not be proven. Prospective randomized trials in the era of immunotherapy and targeted therapies are warranted.

Statistical analysis of high-dimensional biomedical data: a gentle introduction to analytical goals, common approaches and challenges.

BACKGROUND: In high-dimensional data (HDD) settings, the number of variables associated with each observation is very large. Prominent examples of HDD in biomedical research include omics data with a large number of variables such as many measurements across the genome, proteome, or metabolome, as well as electronic health records data that have large numbers of variables recorded for each patient. The statistical analysis of such data requires knowledge and experience, sometimes of complex methods adapted to the respective research questions. METHODS: Advances in statistical methodology and machine learning methods offer new opportunities for innovative analyses of HDD, but at the same time require a deeper understanding of some fundamental statistical concepts. Topic group TG9 "High-dimensional data" of the STRATOS (STRengthening Analytical Thinking for Observational Studies) initiative provides guidance for the analysis of observational studies, addressing particular statistical challenges and opportunities for the analysis of studies involving HDD. In this overview, we discuss key aspects of HDD analysis to provide a gentle introduction for non-statisticians and for classically trained statisticians with little experience specific to HDD. RESULTS: The paper is organized with respect to subtopics that are most relevant for the analysis of HDD, in particular initial data analysis, exploratory data analysis, multiple testing, and prediction. For each subtopic, main analytical goals in HDD settings are outlined. For each of these goals, basic explanations for some commonly used analysis methods are provided. Situations are identified where traditional statistical methods cannot, or should not, be used in the HDD setting, or where adequate analytic tools are still lacking. Many key references are provided. CONCLUSIONS: This review aims to provide a solid statistical foundation for researchers, including statisticians and non-statisticians, who are new to research with HDD or simply want to better evaluate and understand the results of HDD analyses.

Over-optimism in unsupervised microbiome analysis: Insights from network learning and clustering.

In recent years, unsupervised analysis of microbiome data, such as microbial network analysis and clustering, has increased in popularity. Many new statistical and computational methods have been proposed for these tasks. This multiplicity of analysis strategies poses a challenge for researchers, who are often unsure which method(s) to use and might be tempted to try different methods on their dataset to look for the "best" ones. However, if only the best results are selectively reported, this may cause over-optimism: the "best" method is overly fitted to the specific dataset, and the results might be non-replicable on validation data. Such effects will ultimately hinder research progress. Yet so far, these topics have been given little attention in the context of unsupervised microbiome analysis. In our illustrative study, we aim to quantify over-optimism effects in this context. We model the approach of a hypothetical microbiome researcher who undertakes four unsupervised research tasks: clustering of bacterial genera, hub detection in microbial networks, differential microbial network analysis, and clustering of samples. While these tasks are unsupervised, the researcher might still have certain expectations as to what constitutes interesting results. We translate these expectations into concrete evaluation criteria that the hypothetical researcher might want to optimize. We then randomly split an exemplary dataset from the American Gut Project into discovery and validation sets multiple times. For each research task, multiple method combinations (e.g., methods for data normalization, network generation, and/or clustering) are tried on the discovery data, and the combination that yields the best result according to the evaluation criterion is chosen. While the hypothetical researcher might only report this result, we also apply the "best" method combination to the validation dataset. The results are then compared between discovery and validation data. In all four research tasks, there are notable over-optimism effects; the results on the validation data set are worse compared to the discovery data, averaged over multiple random splits into discovery/validation data. Our study thus highlights the importance of validation and replication in microbiome analysis to obtain reliable results and demonstrates that the issue of over-optimism goes beyond the context of statistical testing and fishing for significance.

Relationship between age at initiation of cysteamine treatment, adherence with therapy, and glomerular kidney function in infantile nephropathic cystinosis.

Infantile nephropathic cystinosis, due to impaired transport of cystine out of lysosomes, occurs with an incidence of 1 in 100-200,000 live births. It is characterized by renal Fanconi syndrome in the first year of life and glomerular dysfunction progression to end-stage kidney disease by approximately 10 years of age. Treatment with oral cysteamine therapy helps preserve glomerular function, but affected individuals eventually require kidney replacement therapy. This is because glomerular damage had already occurred by the time a child is diagnosed with cystinosis, typically in the second year of life. We performed a retrospective multicenter study to investigate the impact of initiating cysteamine treatment within the first 2 months of life in some infants and comparing two different levels of adherence in patients diagnosed at the typical age. We collected 3983 data points from 55 patients born between 1997 and 2020; 52 patients with 1592 data points could be further evaluated. These data were first analyzed by dividing the patient cohort into three groups: (i) standard treatment start with good adherence, (ii) standard treatment start with less good adherence, and (iii) early treatment start. At every age, mean estimated glomerular filtration rate (eGFR) was higher in early-treated patients than in later-treated patients. Second, a generalized additive mixed model (GAMM) was applied showing that patients with initiation of treatment before 2 months of age are expected to have a 34 ml/min/1.73 m2 higher eGFR than patients with later treatment start while controlling for adherence and patients' age. These data strongly suggest that oral cysteamine treatment initiated within 2 months of birth preserves kidney function in infantile nephropathic cystinosis and provide evidence of the utility of newborn screening for this disease.

Effective methods to enhance medical students' cardioversion and transcutaneous cardiac pacing skills retention - a prospective controlled study.

BACKGROUND: Guideline-based therapy of cardiac arrhythmias is important for many physicians from the beginning of their training. Practical training of the required skills to treat cardiac arrhythmias is useful for acquiring these skills but does not seem sufficient for skill retention. The aim of this study was to compare different retention methods for skills required to treat cardiac arrhythmias with respect to the performance of these skills in an assessment. METHODS: Seventy-one final-year medical students participated in a newly designed workshop to train synchronized cardioversion (SC) and transcutaneous cardiac pacing (TCP) skills in 2020. All participants completed an objective structured clinical examination (OSCE 1) one week after the training. Afterwards, the participants were stratified and randomized into three groups. Nine weeks later, one group received a standard operating procedure (SOP) for the skills, one group participated in a second workshop (SW), and one group received no further intervention (control). Ten weeks after the first training, all groups participated in OSCE 2. RESULTS: The average score of all students in OSCE 1 was 15.6 ± 0.8 points with no significant differences between the three groups. Students in the control group reached a significantly (p < 0.001) lower score in OSCE 2 (-2.0 points, CI: [-2.9;-1.1]) than in OSCE 1. Students in the SOP-group achieved on average the same result in OSCE 2 as in OSCE 1 (0 points, CI: [-0.63;+0.63]). Students who completed a second skills training (SW-group) scored not significantly higher in OSCE 2 compared to OSCE 1 (+0.4 points, CI: [-0.29;+1.12]). The OSCE 2 scores in groups SOP and SW were neither significantly different nor statistically equivalent. CONCLUSIONS: Partial loss of SC and TCP skills acquired in a workshop can be prevented after 10 weeks by reading an SOP as well as by a second workshop one week before the second assessment. Refreshing practical skills with an SOP could provide an effective and inexpensive method for skills retention compared to repeating a training. Further studies need to show whether this effect also exists for other skills and how frequently an SOP should be re-read for appropriate long-term retention of complex skills.

Critical appraisal of artificial intelligence-based prediction models for cardiovascular disease.

The medical field has seen a rapid increase in the development of artificial intelligence (AI)-based prediction models. With the introduction of such AI-based prediction model tools and software in cardiovascular patient care, the cardiovascular researcher and healthcare professional are challenged to understand the opportunities as well as the limitations of the AI-based predictions. In this article, we present 12 critical questions for cardiovascular health professionals to ask when confronted with an AI-based prediction model. We aim to support medical professionals to distinguish the AI-based prediction models that can add value to patient care from the AI that does not.

COMPANION: development of a patient-centred complexity and casemix classification for adult palliative care patients based on needs and resource use - a protocol for a cross-sectional multi-centre study.

BACKGROUND: A casemix classification based on patients' needs can serve to better describe the patient group in palliative care and thus help to develop adequate future care structures and enable national benchmarking and quality control. However, in Germany, there is no such an evidence-based system to differentiate the complexity of patients' needs in palliative care. Therefore, the study aims to develop a patient-oriented, nationally applicable complexity and casemix classification for adult palliative care patients in Germany. METHODS: COMPANION is a mixed-methods study with data derived from three subprojects. Subproject 1: Prospective, cross-sectional multi-centre study collecting data on patients' needs which reflect the complexity of the respective patient situation, as well as data on resources that are required to meet these needs in specialist palliative care units, palliative care advisory teams, and specialist palliative home care. Subproject 2: Qualitative study including the development of a literature-based preliminary list of characteristics, expert interviews, and a focus group to develop a taxonomy for specialist palliative care models. Subproject 3: Multi-centre costing study based on resource data from subproject 1 and data of study centres. Data and results from the three subprojects will inform each other and form the basis for the development of the casemix classification. Ultimately, the casemix classification will be developed by applying Classification and Regression Tree (CART) analyses using patient and complexity data from subproject 1 and patient-related cost data from subproject 3. DISCUSSION: This is the first multi-centre costing study that integrates the structure and process characteristics of different palliative care settings in Germany with individual patient care. The mixed methods design and variety of included data allow for the development of a casemix classification that reflect on the complexity of the research subject. The consecutive inclusion of all patients cared for in participating study centres within the time of data collection allows for a comprehensive description of palliative care patients and their needs. A limiting factor is that data will be collected at least partly during the COVID-19 pandemic and potential impact of the pandemic on health care and the research topic cannot be excluded. TRIAL REGISTRATION: German Register for Clinical Studies trial registration number: DRKS00020517 .

The multiplicity of analysis strategies jeopardizes replicability: lessons learned across disciplines.

For a given research question, there are usually a large variety of possible analysis strategies acceptable according to the scientific standards of the field, and there are concerns that this multiplicity of analysis strategies plays an important role in the non-replicability of research findings. Here, we define a general framework on common sources of uncertainty arising in computational analyses that lead to this multiplicity, and apply this framework within an overview of approaches proposed across disciplines to address the issue. Armed with this framework, and a set of recommendations derived therefrom, researchers will be able to recognize strategies applicable to their field and use them to generate findings more likely to be replicated in future studies, ultimately improving the credibility of the scientific process.

On the optimistic performance evaluation of newly introduced bioinformatic methods.

Most research articles presenting new data analysis methods claim that "the new method performs better than existing methods," but the veracity of such statements is questionable. Our manuscript discusses and illustrates consequences of the optimistic bias occurring during the evaluation of novel data analysis methods, that is, all biases resulting from, for example, selection of datasets or competing methods, better ability to fix bugs in a preferred method, and selective reporting of method variants. We quantitatively investigate this bias using an example from epigenetic analysis: normalization methods for data generated by the Illumina HumanMethylation450K BeadChip microarray.

Large-scale benchmark study of survival prediction methods using multi-omics data.

Multi-omics data, that is, datasets containing different types of high-dimensional molecular variables, are increasingly often generated for the investigation of various diseases. Nevertheless, questions remain regarding the usefulness of multi-omics data for the prediction of disease outcomes such as survival time. It is also unclear which methods are most appropriate to derive such prediction models. We aim to give some answers to these questions through a large-scale benchmark study using real data. Different prediction methods from machine learning and statistics were applied on 18 multi-omics cancer datasets (35 to 1000 observations, up to 100 000 variables) from the database 'The Cancer Genome Atlas' (TCGA). The considered outcome was the (censored) survival time. Eleven methods based on boosting, penalized regression and random forest were compared, comprising both methods that do and that do not take the group structure of the omics variables into account. The Kaplan-Meier estimate and a Cox model using only clinical variables were used as reference methods. The methods were compared using several repetitions of 5-fold cross-validation. Uno's C-index and the integrated Brier score served as performance metrics. The results indicate that methods taking into account the multi-omics structure have a slightly better prediction performance. Taking this structure into account can protect the predictive information in low-dimensional groups-especially clinical variables-from not being exploited during prediction. Moreover, only the block forest method outperformed the Cox model on average, and only slightly. This indicates, as a by-product of our study, that in the considered TCGA studies the utility of multi-omics data for prediction purposes was limited. Contact:moritz.herrmann@stat.uni-muenchen.de, +49 89 2180 3198 Supplementary information: Supplementary data are available at Briefings in Bioinformatics online. All analyses are reproducible using R code freely available on Github.

NetCoMi: network construction and comparison for microbiome data in R.

MOTIVATION: Estimating microbial association networks from high-throughput sequencing data is a common exploratory data analysis approach aiming at understanding the complex interplay of microbial communities in their natural habitat. Statistical network estimation workflows comprise several analysis steps, including methods for zero handling, data normalization and computing microbial associations. Since microbial interactions are likely to change between conditions, e.g. between healthy individuals and patients, identifying network differences between groups is often an integral secondary analysis step. Thus far, however, no unifying computational tool is available that facilitates the whole analysis workflow of constructing, analysing and comparing microbial association networks from high-throughput sequencing data. RESULTS: Here, we introduce NetCoMi (Network Construction and comparison for Microbiome data), an R package that integrates existing methods for each analysis step in a single reproducible computational workflow. The package offers functionality for constructing and analysing single microbial association networks as well as quantifying network differences. This enables insights into whether single taxa, groups of taxa or the overall network structure change between groups. NetCoMi also contains functionality for constructing differential networks, thus allowing to assess whether single pairs of taxa are differentially associated between two groups. Furthermore, NetCoMi facilitates the construction and analysis of dissimilarity networks of microbiome samples, enabling a high-level graphical summary of the heterogeneity of an entire microbiome sample collection. We illustrate NetCoMi's wide applicability using data sets from the GABRIELA study to compare microbial associations in settled dust from children's rooms between samples from two study centers (Ulm and Munich). AVAILABILITY: R scripts used for producing the examples shown in this manuscript are provided as supplementary data. The NetCoMi package, together with a tutorial, is available at https://github.com/stefpeschel/NetCoMi. CONTACT: Tel:+49 89 3187 43258; stefanie.peschel@mail.de. SUPPLEMENTARY INFORMATION: Supplementary data are available at Briefings in Bioinformatics online.