Lack of evidence for predictive utility from resting state fMRI data for individual exposure-based cognitive behavioral therapy outcomes: A machine learning study in two large multi-site samples in anxiety disorders.

Data-based predictions of individual Cognitive Behavioral Therapy (CBT) treatment response are a fundamental step towards precision medicine. Past studies demonstrated only moderate prediction accuracy (i.e. ability to discriminate between responders and non-responders of a given treatment) when using clinical routine data such as demographic and questionnaire data, while neuroimaging data achieved superior prediction accuracy. However, these studies may be considerably biased due to very limited sample sizes and bias-prone methodology. Adequately powered and cross-validated samples are a prerequisite to evaluate predictive performance and to identify the most promising predictors. We therefore analyzed resting state functional magnet resonance imaging (rs-fMRI) data from two large clinical trials to test whether functional neuroimaging data continues to provide good prediction accuracy in much larger samples. Data came from two distinct German multicenter studies on exposure-based CBT for anxiety disorders, the Protect-AD and SpiderVR studies. We separately and independently preprocessed baseline rs-fMRI data from n = 220 patients (Protect-AD) and n = 190 patients (SpiderVR) and extracted a variety of features, including ROI-to-ROI and edge-functional connectivity, sliding-windows, and graph measures. Including these features in sophisticated machine learning pipelines, we found that predictions of individual outcomes never significantly differed from chance level, even when conducting a range of exploratory post-hoc analyses. Moreover, resting state data never provided prediction accuracy beyond the sociodemographic and clinical data. The analyses were independent of each other in terms of selecting methods to process resting state data for prediction input as well as in the used parameters of the machine learning pipelines, corroborating the external validity of the results. These similar findings in two independent studies, analyzed separately, urge caution regarding the interpretation of promising prediction results based on neuroimaging data from small samples and emphasizes that some of the prediction accuracies from previous studies may result from overestimation due to homogeneous data and weak cross-validation schemes. The promise of resting-state neuroimaging data to play an important role in the prediction of CBT treatment outcomes in patients with anxiety disorders remains yet to be delivered.

Genetic, individual, and familial risk correlates of brain network controllability in major depressive disorder.

Many therapeutic interventions in psychiatry can be viewed as attempts to influence the brain's large-scale, dynamic network state transitions. Building on connectome-based graph analysis and control theory, Network Control Theory is emerging as a powerful tool to quantify network controllability-i.e., the influence of one brain region over others regarding dynamic network state transitions. If and how network controllability is related to mental health remains elusive. Here, from Diffusion Tensor Imaging data, we inferred structural connectivity and inferred calculated network controllability parameters to investigate their association with genetic and familial risk in patients diagnosed with major depressive disorder (MDD, n = 692) and healthy controls (n = 820). First, we establish that controllability measures differ between healthy controls and MDD patients while not varying with current symptom severity or remission status. Second, we show that controllability in MDD patients is associated with polygenic scores for MDD and psychiatric cross-disorder risk. Finally, we provide evidence that controllability varies with familial risk of MDD and bipolar disorder as well as with body mass index. In summary, we show that network controllability is related to genetic, individual, and familial risk in MDD patients. We discuss how these insights into individual variation of network controllability may inform mechanistic models of treatment response prediction and personalized intervention-design in mental health.

Reduced fractional anisotropy in bipolar disorder v. major depressive disorder independent of current symptoms.

BACKGROUND: Patients with bipolar disorder (BD) show reduced fractional anisotropy (FA) compared to patients with major depressive disorder (MDD). Little is known about whether these differences are mood state-independent or influenced by acute symptom severity. Therefore, the aim of this study was (1) to replicate abnormalities in white matter microstructure in BD v. MDD and (2) to investigate whether these vary across depressed, euthymic, and manic mood. METHODS: In this cross-sectional diffusion tensor imaging study, n = 136 patients with BD were compared to age- and sex-matched MDD patients and healthy controls (HC) (n = 136 each). Differences in FA were investigated using tract-based spatial statistics. Using interaction models, the influence of acute symptom severity and mood state on the differences between patient groups were tested. RESULTS: Analyses revealed a main effect of diagnosis on FA across all three groups (ptfce-FWE = 0.003). BD patients showed reduced FA compared to both MDD (ptfce-FWE = 0.005) and HC (ptfce-FWE < 0.001) in large bilateral clusters. These consisted of several white matter tracts previously described in the literature, including commissural, association, and projection tracts. There were no significant interaction effects between diagnosis and symptom severity or mood state (all ptfce-FWE > 0.704). CONCLUSIONS: Results indicated that the difference between BD and MDD was independent of depressive and manic symptom severity and mood state. Disruptions in white matter microstructure in BD might be a trait effect of the disorder. The potential of FA values to be used as a biomarker to differentiate BD from MDD should be further addressed in future studies using longitudinal designs.

Recommendations for machine learning benchmarks in neuroimaging.

The field of neuroimaging has embraced methods from machine learning in a variety of ways. Although an increasing number of initiatives have published open-access neuroimaging datasets, specifically designed benchmarks are rare in the field. In this article, we first describe how benchmarks in computer science and biomedical imaging have fostered methodological progress in machine learning. Second, we identify the special characteristics of neuroimaging data and outline what researchers have to ensure when establishing a neuroimaging benchmark, how datasets should be composed and how adequate evaluation criteria can be chosen. Based on lessons learned from machine learning benchmarks, we argue for an extended evaluation procedure that, next to applying suitable performance metrics, focuses on scientifically relevant aspects such as explainability, robustness, uncertainty, computational efficiency and code quality. Lastly, we envision a collaborative neuroimaging benchmarking platform that combines the discussed aspects in a collaborative and agile framework, allowing researchers across disciplines to work together on the key predictive problems of the field of neuroimaging and psychiatry.

FreeSurfer-based segmentation of hippocampal subfields: A review of methods and applications, with a novel quality control procedure for ENIGMA studies and other collaborative efforts.

Structural hippocampal abnormalities are common in many neurological and psychiatric disorders, and variation in hippocampal measures is related to cognitive performance and other complex phenotypes such as stress sensitivity. Hippocampal subregions are increasingly studied, as automated algorithms have become available for mapping and volume quantification. In the context of the Enhancing Neuro Imaging Genetics through Meta Analysis Consortium, several Disease Working Groups are using the FreeSurfer software to analyze hippocampal subregion (subfield) volumes in patients with neurological and psychiatric conditions along with data from matched controls. In this overview, we explain the algorithm's principles, summarize measurement reliability studies, and demonstrate two additional aspects (subfield autocorrelation and volume/reliability correlation) with illustrative data. We then explain the rationale for a standardized hippocampal subfield segmentation quality control (QC) procedure for improved pipeline harmonization. To guide researchers to make optimal use of the algorithm, we discuss how global size and age effects can be modeled, how QC steps can be incorporated and how subfields may be aggregated into composite volumes. This discussion is based on a synopsis of 162 published neuroimaging studies (01/2013-12/2019) that applied the FreeSurfer hippocampal subfield segmentation in a broad range of domains including cognition and healthy aging, brain development and neurodegeneration, affective disorders, psychosis, stress regulation, neurotoxicity, epilepsy, inflammatory disease, childhood adversity and posttraumatic stress disorder, and candidate and whole genome (epi-)genetics. Finally, we highlight points where FreeSurfer-based hippocampal subfield studies may be optimized.

Brain aging in major depressive disorder: results from the ENIGMA major depressive disorder working group.

Major depressive disorder (MDD) is associated with an increased risk of brain atrophy, aging-related diseases, and mortality. We examined potential advanced brain aging in adult MDD patients, and whether this process is associated with clinical characteristics in a large multicenter international dataset. We performed a mega-analysis by pooling brain measures derived from T1-weighted MRI scans from 19 samples worldwide. Healthy brain aging was estimated by predicting chronological age (18-75 years) from 7 subcortical volumes, 34 cortical thickness and 34 surface area, lateral ventricles and total intracranial volume measures separately in 952 male and 1236 female controls from the ENIGMA MDD working group. The learned model coefficients were applied to 927 male controls and 986 depressed males, and 1199 female controls and 1689 depressed females to obtain independent unbiased brain-based age predictions. The difference between predicted "brain age" and chronological age was calculated to indicate brain-predicted age difference (brain-PAD). On average, MDD patients showed a higher brain-PAD of +1.08 (SE 0.22) years (Cohen's d = 0.14, 95% CI: 0.08-0.20) compared with controls. However, this difference did not seem to be driven by specific clinical characteristics (recurrent status, remission status, antidepressant medication use, age of onset, or symptom severity). This highly powered collaborative effort showed subtle patterns of age-related structural brain abnormalities in MDD. Substantial within-group variance and overlap between groups were observed. Longitudinal studies of MDD and somatic health outcomes are needed to further assess the clinical value of these brain-PAD estimates.

Brain structural abnormalities in obesity: relation to age, genetic risk, and common psychiatric disorders : Evidence through univariate and multivariate mega-analysis including 6420 participants from the ENIGMA MDD working group.

Emerging evidence suggests that obesity impacts brain physiology at multiple levels. Here we aimed to clarify the relationship between obesity and brain structure using structural MRI (n = 6420) and genetic data (n = 3907) from the ENIGMA Major Depressive Disorder (MDD) working group. Obesity (BMI > 30) was significantly associated with cortical and subcortical abnormalities in both mass-univariate and multivariate pattern recognition analyses independent of MDD diagnosis. The most pronounced effects were found for associations between obesity and lower temporo-frontal cortical thickness (maximum Cohen´s d (left fusiform gyrus) = -0.33). The observed regional distribution and effect size of cortical thickness reductions in obesity revealed considerable similarities with corresponding patterns of lower cortical thickness in previously published studies of neuropsychiatric disorders. A higher polygenic risk score for obesity significantly correlated with lower occipital surface area. In addition, a significant age-by-obesity interaction on cortical thickness emerged driven by lower thickness in older participants. Our findings suggest a neurobiological interaction between obesity and brain structure under physiological and pathological brain conditions.

Cortical surface area alterations shaped by genetic load for neuroticism.

Neuroticism has been shown to act as an important risk factor for major depressive disorder (MDD). Genetic and neuroimaging research has independently revealed biological correlates of neurotic personality including cortical alterations in brain regions of high relevance for affective disorders. Here we investigated the influence of a polygenic score for neuroticism (PGS) on cortical brain structure in a joint discovery sample of n = 746 healthy controls (HC) and n = 268 MDD patients. Findings were validated in an independent replication sample (n = 341 HC and n = 263 MDD). Subgroup analyses stratified for case-control status and analyses of associations between neurotic phenotype and cortical measures were carried out. PGS for neuroticism was significantly associated with a decreased cortical surface area of the inferior parietal cortex, the precuneus, the rostral cingulate cortex and the inferior frontal gyrus in the discovery sample. Similar associations between PGS and surface area of the inferior parietal cortex and the precuneus were demonstrated in the replication sample. Subgroup analyses revealed negative associations in the latter regions between PGS and surface area in both HC and MDD subjects. Neurotic phenotype was negatively correlated with surface area in similar cortical regions including the inferior parietal cortex and the precuneus. No significant associations between PGS and cortical thickness were detected. The morphometric overlap of associations between both PGS and neurotic phenotype in similar cortical regions closely related to internally focused cognition points to the potential relevance of genetically shaped cortical alterations in the development of neuroticism.

Influence of electroconvulsive therapy on white matter structure in a diffusion tensor imaging study.

BACKGROUND: Electroconvulsive therapy (ECT) is a fast-acting intervention for major depressive disorder. Previous studies indicated neurotrophic effects following ECT that might contribute to changes in white matter brain structure. We investigated the influence of ECT in a non-randomized prospective study focusing on white matter changes over time. METHODS: Twenty-nine severely depressed patients receiving ECT in addition to inpatient treatment, 69 severely depressed patients with inpatient treatment (NON-ECT) and 52 healthy controls (HC) took part in a non-randomized prospective study. Participants were scanned twice, approximately 6 weeks apart, using diffusion tensor imaging, applying tract-based spatial statistics. Additional correlational analyses were conducted in the ECT subsample to investigate the effects of seizure duration and therapeutic response. RESULTS: Mean diffusivity (MD) increased after ECT in the right hemisphere, which was an ECT-group-specific effect. Seizure duration was associated with decreased fractional anisotropy (FA) following ECT. Longitudinal changes in ECT were not associated with therapy response. However, within the ECT group only, baseline FA was positively and MD negatively associated with post-ECT symptomatology. CONCLUSION: Our data suggest that ECT changes white matter integrity, possibly reflecting increased permeability of the blood-brain barrier, resulting in disturbed communication of fibers. Further, baseline diffusion metrics were associated with therapy response. Coherent fiber structure could be a prerequisite for a generalized seizure and inhibitory brain signaling necessary to successfully inhibit increased seizure activity.

deepbet: Fast brain extraction of T1-weighted MRI using Convolutional Neural Networks.

BACKGROUND: Brain extraction in magnetic resonance imaging (MRI) data is an important segmentation step in many neuroimaging preprocessing pipelines. Image segmentation is one of the research fields in which deep learning had the biggest impact in recent years. Consequently, traditional brain extraction methods are now being replaced by deep learning-based methods. METHOD: Here, we used a unique dataset compilation comprising 7837 T1-weighted (T1w) MR images from 191 different OpenNeuro datasets in combination with advanced deep learning methods to build a fast, high-precision brain extraction tool called deepbet. RESULTS: deepbet sets a novel state-of-the-art performance during cross-dataset validation with a median Dice score (DSC) of 99.0 on unseen datasets, outperforming the current best performing deep learning (DSC=97.9) and classic (DSC=96.5) methods. While current methods are more sensitive to outliers, deepbet achieves a Dice score of >97.4 across all 7837 images from 191 different datasets. This robustness was additionally tested in 5 external datasets, which included challenging clinical MR images. During visual exploration of each method's output which resulted in the lowest Dice score, major errors could be found for all of the tested tools except deepbet. Finally, deepbet uses a compute efficient variant of the UNet architecture, which accelerates brain extraction by a factor of ≈10 compared to current methods, enabling the processing of one image in ≈2 s on low level hardware. CONCLUSIONS: In conclusion, deepbet demonstrates superior performance and reliability in brain extraction across a wide range of T1w MR images of adults, outperforming existing top tools. Its high minimal Dice score and minimal objective errors, even in challenging conditions, validate deepbet as a highly dependable tool for accurate brain extraction. deepbet can be conveniently installed via "pip install deepbet" and is publicly accessible at https://github.com/wwu-mmll/deepbet.

GateNet: A novel neural network architecture for automated flow cytometry gating.

BACKGROUND AND OBJECTIVE: Flow cytometry is a widely used technique for identifying cell populations in patient-derived fluids, such as peripheral blood (PB) or cerebrospinal fluid (CSF). Despite its ubiquity in research and clinical practice, the process of gating, i.e., manually identifying cell types, is labor-intensive and error-prone. The objective of this study is to address this challenge by introducing GateNet, a neural network architecture designed for fully end-to-end automated gating without the need for correcting batch effects. METHODS: For this study a unique dataset is used which comprises over 8,000,000 events from N = 127 PB and CSF samples which were manually labeled independently by four experts. Applying cross-validation, the classification performance of GateNet is compared to the human experts performance. Additionally, GateNet is applied to a publicly available dataset to evaluate generalization. The classification performance is measured using the F1 score. RESULTS: GateNet achieves F1 scores ranging from 0.910 to 0.997 demonstrating human-level performance on samples unseen during training. In the publicly available dataset, GateNet confirms its generalization capabilities with an F1 score of 0.936. Importantly, we also show that GateNet only requires ≈10 samples to reach human-level performance. Finally, gating with GateNet only takes 15 microseconds per event utilizing graphics processing units (GPU). CONCLUSIONS: GateNet enables fully end-to-end automated gating in flow cytometry, overcoming the labor-intensive and error-prone nature of manual adjustments. The neural network achieves human-level performance on unseen samples and generalizes well to diverse datasets. Notably, its data efficiency, requiring only ∼10 samples to reach human-level performance, positions GateNet as a widely applicable tool across various domains of flow cytometry.

A Systematic Evaluation of Machine Learning-Based Biomarkers for Major Depressive Disorder.

Importance: Biological psychiatry aims to understand mental disorders in terms of altered neurobiological pathways. However, for one of the most prevalent and disabling mental disorders, major depressive disorder (MDD), no informative biomarkers have been identified. Objective: To evaluate whether machine learning (ML) can identify a multivariate biomarker for MDD. Design, Setting, and Participants: This study used data from the Marburg-Münster Affective Disorders Cohort Study, a case-control clinical neuroimaging study. Patients with acute or lifetime MDD and healthy controls aged 18 to 65 years were recruited from primary care and the general population in Münster and Marburg, Germany, from September 11, 2014, to September 26, 2018. The Münster Neuroimaging Cohort (MNC) was used as an independent partial replication sample. Data were analyzed from April 2022 to June 2023. Exposure: Patients with MDD and healthy controls. Main Outcome and Measure: Diagnostic classification accuracy was quantified on an individual level using an extensive ML-based multivariate approach across a comprehensive range of neuroimaging modalities, including structural and functional magnetic resonance imaging and diffusion tensor imaging as well as a polygenic risk score for depression. Results: Of 1801 included participants, 1162 (64.5%) were female, and the mean (SD) age was 36.1 (13.1) years. There were a total of 856 patients with MDD (47.5%) and 945 healthy controls (52.5%). The MNC replication sample included 1198 individuals (362 with MDD [30.1%] and 836 healthy controls [69.9%]). Training and testing a total of 4 million ML models, mean (SD) accuracies for diagnostic classification ranged between 48.1% (3.6%) and 62.0% (4.8%). Integrating neuroimaging modalities and stratifying individuals based on age, sex, treatment, or remission status does not enhance model performance. Findings were replicated within study sites and also observed in structural magnetic resonance imaging within MNC. Under simulated conditions of perfect reliability, performance did not significantly improve. Analyzing model errors suggests that symptom severity could be a potential focus for identifying MDD subgroups. Conclusion and Relevance: Despite the improved predictive capability of multivariate compared with univariate neuroimaging markers, no informative individual-level MDD biomarker-even under extensive ML optimization in a large sample of diagnosed patients-could be identified.

Evidence from comprehensive independent validation studies for smooth pursuit dysfunction as a sensorimotor biomarker for psychosis.

Smooth pursuit eye movements are considered a well-established and quantifiable biomarker of sensorimotor function in psychosis research. Identifying psychotic syndromes on an individual level based on neurobiological markers is limited by heterogeneity and requires comprehensive external validation to avoid overestimation of prediction models. Here, we studied quantifiable sensorimotor measures derived from smooth pursuit eye movements in a large sample of psychosis probands (N = 674) and healthy controls (N = 305) using multivariate pattern analysis. Balanced accuracies of 64% for the prediction of psychosis status are in line with recent results from other large heterogenous psychiatric samples. They are confirmed by external validation in independent large samples including probands with (1) psychosis (N = 727) versus healthy controls (N = 292), (2) psychotic (N = 49) and non-psychotic bipolar disorder (N = 36), and (3) non-psychotic affective disorders (N = 119) and psychosis (N = 51) yielding accuracies of 65%, 66% and 58%, respectively, albeit slightly different psychosis syndromes. Our findings make a significant contribution to the identification of biologically defined profiles of heterogeneous psychosis syndromes on an individual level underlining the impact of sensorimotor dysfunction in psychosis.

Towards a network control theory of electroconvulsive therapy response.

Electroconvulsive Therapy (ECT) is arguably the most effective intervention for treatment-resistant depression. While large interindividual variability exists, a theory capable of explaining individual response to ECT remains elusive. To address this, we posit a quantitative, mechanistic framework of ECT response based on Network Control Theory (NCT). Then, we empirically test our approach and employ it to predict ECT treatment response. To this end, we derive a formal association between Postictal Suppression Index (PSI)-an ECT seizure quality index-and whole-brain modal and average controllability, NCT metrics based on white-matter brain network architecture, respectively. Exploiting the known association of ECT response and PSI, we then hypothesized an association between our controllability metrics and ECT response mediated by PSI. We formally tested this conjecture in N = 50 depressive patients undergoing ECT. We show that whole-brain controllability metrics based on pre-ECT structural connectome data predict ECT response in accordance with our hypotheses. In addition, we show the expected mediation effects via PSI. Importantly, our theoretically motivated metrics are at least on par with extensive machine learning models based on pre-ECT connectome data. In summary, we derived and tested a control-theoretic framework capable of predicting ECT response based on individual brain network architecture. It makes testable, quantitative predictions regarding individual therapeutic response, which are corroborated by strong empirical evidence. Our work might constitute a starting point for a comprehensive, quantitative theory of personalized ECT interventions rooted in control theory.

An uncertainty-aware, shareable, and transparent neural network architecture for brain-age modeling.

The deviation between chronological age and age predicted from neuroimaging data has been identified as a sensitive risk marker of cross-disorder brain changes, growing into a cornerstone of biological age research. However, machine learning models underlying the field do not consider uncertainty, thereby confounding results with training data density and variability. Also, existing models are commonly based on homogeneous training sets, often not independently validated, and cannot be shared because of data protection issues. Here, we introduce an uncertainty-aware, shareable, and transparent Monte Carlo dropout composite quantile regression (MCCQR) Neural Network trained on N = 10,691 datasets from the German National Cohort. The MCCQR model provides robust, distribution-free uncertainty quantification in high-dimensional neuroimaging data, achieving lower error rates compared with existing models. In two examples, we demonstrate that it prevents spurious associations and increases power to detect deviant brain aging. We make the pretrained model and code publicly available.

Quantifying Deviations of Brain Structure and Function in Major Depressive Disorder Across Neuroimaging Modalities.

Importance: Identifying neurobiological differences between patients with major depressive disorder (MDD) and healthy individuals has been a mainstay of clinical neuroscience for decades. However, recent meta-analyses have raised concerns regarding the replicability and clinical relevance of brain alterations in depression. Objective: To quantify the upper bounds of univariate effect sizes, estimated predictive utility, and distributional dissimilarity of healthy individuals and those with depression across structural magnetic resonance imaging (MRI), diffusion-tensor imaging, and functional task-based as well as resting-state MRI, and to compare results with an MDD polygenic risk score (PRS) and environmental variables. Design, Setting, and Participants: This was a cross-sectional, case-control clinical neuroimaging study. Data were part of the Marburg-Münster Affective Disorders Cohort Study. Patients with depression and healthy controls were recruited from primary care and the general population in Münster and Marburg, Germany. Study recruitment was performed from September 11, 2014, to September 26, 2018. The sample comprised patients with acute and chronic MDD as well as healthy controls in the age range of 18 to 65 years. Data were analyzed from October 29, 2020, to April 7, 2022. Main Outcomes and Measures: Primary analyses included univariate partial effect size (η2), classification accuracy, and distributional overlapping coefficient for healthy individuals and those with depression across neuroimaging modalities, controlling for age, sex, and additional modality-specific confounding variables. Secondary analyses included patient subgroups for acute or chronic depressive status. Results: A total of 1809 individuals (861 patients [47.6%] and 948 controls [52.4%]) were included in the analysis (mean [SD] age, 35.6 [13.2] years; 1165 female patients [64.4%]). The upper bound of the effect sizes of the single univariate measures displaying the largest group difference ranged from partial η2 of 0.004 to 0.017, and distributions overlapped between 87% and 95%, with classification accuracies ranging between 54% and 56% across neuroimaging modalities. This pattern remained virtually unchanged when considering either only patients with acute or chronic depression. Differences were comparable with those found for PRS but substantially smaller than for environmental variables. Conclusions and Relevance: Results of this case-control study suggest that even for maximum univariate biological differences, deviations between patients with MDD and healthy controls were remarkably small, single-participant prediction was not possible, and similarity between study groups dominated. Biological psychiatry should facilitate meaningful outcome measures or predictive approaches to increase the potential for a personalization of the clinical practice.

Systematic misestimation of machine learning performance in neuroimaging studies of depression.

We currently observe a disconcerting phenomenon in machine learning studies in psychiatry: While we would expect larger samples to yield better results due to the availability of more data, larger machine learning studies consistently show much weaker performance than the numerous small-scale studies. Here, we systematically investigated this effect focusing on one of the most heavily studied questions in the field, namely the classification of patients suffering from Major Depressive Disorder (MDD) and healthy controls based on neuroimaging data. Drawing upon structural MRI data from a balanced sample of N = 1868 MDD patients and healthy controls from our recent international Predictive Analytics Competition (PAC), we first trained and tested a classification model on the full dataset which yielded an accuracy of 61%. Next, we mimicked the process by which researchers would draw samples of various sizes (N = 4 to N = 150) from the population and showed a strong risk of misestimation. Specifically, for small sample sizes (N = 20), we observe accuracies of up to 95%. For medium sample sizes (N = 100) accuracies up to 75% were found. Importantly, further investigation showed that sufficiently large test sets effectively protect against performance misestimation whereas larger datasets per se do not. While these results question the validity of a substantial part of the current literature, we outline the relatively low-cost remedy of larger test sets, which is readily available in most cases.