Association between resting-state connectivity patterns in the defensive system network and treatment response in spider phobia-a replication approach.

Although highly effective on average, exposure-based treatments do not work equally well for all patients with anxiety disorders. The identification of pre-treatment response-predicting patient characteristics may enable patient stratification. Preliminary research highlights the relevance of inhibitory fronto-limbic networks as such. We aimed to identify pre-treatment neural signatures differing between exposure treatment responders and non-responders in spider phobia and to validate results through rigorous replication. Data of a bi-centric intervention study comprised clinical phenotyping and pre-treatment resting-state functional connectivity (rsFC) data of n = 79 patients with spider phobia (discovery sample) and n = 69 patients (replication sample). RsFC data analyses were accomplished using the Matlab-based CONN-toolbox with harmonized analyses protocols at both sites. Treatment response was defined by a reduction of >30% symptom severity from pre- to post-treatment (Spider Phobia Questionnaire Score, primary outcome). Secondary outcome was defined by a reduction of >50% in a Behavioral Avoidance Test (BAT). Mean within-session fear reduction functioned as a process measure for exposure. Compared to non-responders and pre-treatment, results in the discovery sample seemed to indicate that responders exhibited stronger negative connectivity between frontal and limbic structures and were characterized by heightened connectivity between the amygdala and ventral visual pathway regions. Patients exhibiting high within-session fear reduction showed stronger excitatory connectivity within the prefrontal cortex than patients with low within-session fear reduction. Whereas these results could be replicated by another team using the same data (cross-team replication), cross-site replication of the discovery sample findings in the independent replication sample was unsuccessful. Results seem to support negative fronto-limbic connectivity as promising ingredient to enhance response rates in specific phobia but lack sufficient replication. Further research is needed to obtain a valid basis for clinical decision-making and the development of individually tailored treatment options. Notably, future studies should regularly include replication approaches in their protocols.

A Systematic Evaluation of Machine Learning-Based Biomarkers for Major Depressive Disorder.

Importance: Biological psychiatry aims to understand mental disorders in terms of altered neurobiological pathways. However, for one of the most prevalent and disabling mental disorders, major depressive disorder (MDD), no informative biomarkers have been identified. Objective: To evaluate whether machine learning (ML) can identify a multivariate biomarker for MDD. Design, Setting, and Participants: This study used data from the Marburg-Münster Affective Disorders Cohort Study, a case-control clinical neuroimaging study. Patients with acute or lifetime MDD and healthy controls aged 18 to 65 years were recruited from primary care and the general population in Münster and Marburg, Germany, from September 11, 2014, to September 26, 2018. The Münster Neuroimaging Cohort (MNC) was used as an independent partial replication sample. Data were analyzed from April 2022 to June 2023. Exposure: Patients with MDD and healthy controls. Main Outcome and Measure: Diagnostic classification accuracy was quantified on an individual level using an extensive ML-based multivariate approach across a comprehensive range of neuroimaging modalities, including structural and functional magnetic resonance imaging and diffusion tensor imaging as well as a polygenic risk score for depression. Results: Of 1801 included participants, 1162 (64.5%) were female, and the mean (SD) age was 36.1 (13.1) years. There were a total of 856 patients with MDD (47.5%) and 945 healthy controls (52.5%). The MNC replication sample included 1198 individuals (362 with MDD [30.1%] and 836 healthy controls [69.9%]). Training and testing a total of 4 million ML models, mean (SD) accuracies for diagnostic classification ranged between 48.1% (3.6%) and 62.0% (4.8%). Integrating neuroimaging modalities and stratifying individuals based on age, sex, treatment, or remission status does not enhance model performance. Findings were replicated within study sites and also observed in structural magnetic resonance imaging within MNC. Under simulated conditions of perfect reliability, performance did not significantly improve. Analyzing model errors suggests that symptom severity could be a potential focus for identifying MDD subgroups. Conclusion and Relevance: Despite the improved predictive capability of multivariate compared with univariate neuroimaging markers, no informative individual-level MDD biomarker-even under extensive ML optimization in a large sample of diagnosed patients-could be identified.

Childhood trauma moderates schizotypy-related brain morphology: analyses of 1182 healthy individuals from the ENIGMA schizotypy working group.

BACKGROUND: Schizotypy represents an index of psychosis-proneness in the general population, often associated with childhood trauma exposure. Both schizotypy and childhood trauma are linked to structural brain alterations, and it is possible that trauma exposure moderates the extent of brain morphological differences associated with schizotypy. METHODS: We addressed this question using data from a total of 1182 healthy adults (age range: 18-65 years old, 647 females/535 males), pooled from nine sites worldwide, contributing to the Enhancing NeuroImaging Genetics through Meta-Analysis (ENIGMA) Schizotypy working group. All participants completed both the Schizotypal Personality Questionnaire Brief version (SPQ-B), and the Childhood Trauma Questionnaire (CTQ), and underwent a 3D T1-weighted brain MRI scan from which regional indices of subcortical gray matter volume and cortical thickness were determined. RESULTS: A series of multiple linear regressions revealed that differences in cortical thickness in four regions-of-interest were significantly associated with interactions between schizotypy and trauma; subsequent moderation analyses indicated that increasing levels of schizotypy were associated with thicker left caudal anterior cingulate gyrus, right middle temporal gyrus and insula, and thinner left caudal middle frontal gyrus, in people exposed to higher (but not low or average) levels of childhood trauma. This was found in the context of morphological changes directly associated with increasing levels of schizotypy or increasing levels of childhood trauma exposure. CONCLUSIONS: These results suggest that alterations in brain regions critical for higher cognitive and integrative processes that are associated with schizotypy may be enhanced in individuals exposed to high levels of trauma.

Interrelated effects of age and parenthood on whole-brain controllability: protective effects of parenthood in mothers.

Background: Controllability is a measure of the brain's ability to orchestrate neural activity which can be quantified in terms of properties of the brain's network connectivity. Evidence from the literature suggests that aging can exert a general effect on whole-brain controllability. Mounting evidence, on the other hand, suggests that parenthood and motherhood in particular lead to long-lasting changes in brain architecture that effectively slow down brain aging. We hypothesize that parenthood might preserve brain controllability properties from aging. Methods: In a sample of 814 healthy individuals (aged 33.9 ± 12.7 years, 522 females), we estimate whole-brain controllability and compare the aging effects in subjects with vs. those without children. We use diffusion tensor imaging (DTI) to estimate the brain structural connectome. The level of brain control is then calculated from the connectomic properties of the brain structure. Specifically, we measure the network control over many low-energy state transitions (average controllability) and the network control over difficult-to-reach states (modal controllability). Results and conclusion: In nulliparous females, whole-brain average controllability increases, and modal controllability decreases with age, a trend that we do not observe in parous females. Statistical comparison of the controllability metrics shows that modal controllability is higher and average controllability is lower in parous females compared to nulliparous females. In men, we observed the same trend, but the difference between nulliparous and parous males do not reach statistical significance. Our results provide strong evidence that parenthood contradicts aging effects on brain controllability and the effect is stronger in mothers.

Shared and distinct structural brain networks related to childhood maltreatment and social support: connectome-based predictive modeling.

Childhood maltreatment (CM) has been associated with changes in structural brain connectivity even in the absence of mental illness. Social support, an important protective factor in the presence of childhood maltreatment, has been positively linked to white matter integrity. However, the shared effects of current social support and CM and their association with structural connectivity remain to be investigated. They might shed new light on the neurobiological basis of the protective mechanism of social support. Using connectome-based predictive modeling (CPM), we analyzed structural connectomes of N = 904 healthy adults derived from diffusion-weighted imaging. CPM predicts phenotypes from structural connectivity through a cross-validation scheme. Distinct and shared networks of white matter tracts predicting childhood trauma questionnaire scores and the social support questionnaire were identified. Additional analyses were applied to assess the stability of the results. CM and social support were predicted significantly from structural connectome data (all rs ≥ 0.119, all ps ≤ 0.016). Edges predicting CM and social support were inversely correlated, i.e., positively correlated with CM and negatively with social support, and vice versa, with a focus on frontal and temporal regions including the insula and superior temporal lobe. CPM reveals the predictive value of the structural connectome for CM and current social support. Both constructs are inversely associated with connectivity strength in several brain tracts. While this underlines the interconnectedness of these experiences, it suggests social support acts as a protective factor following adverse childhood experiences, compensating for brain network alterations. Future longitudinal studies should focus on putative moderating mechanisms buffering these adverse experiences.

Altered brain dynamic in major depressive disorder: state and trait features.

Temporal neural synchrony disruption can be linked to a variety of symptoms of major depressive disorder (MDD), including mood rigidity and the inability to break the cycle of negative emotion or attention biases. This might imply that altered dynamic neural synchrony may play a role in the persistence and exacerbation of MDD symptoms. Our study aimed to investigate the changes in whole-brain dynamic patterns of the brain functional connectivity and activity related to depression using the hidden Markov model (HMM) on resting-state functional magnetic resonance imaging (rs-fMRI) data. We compared the patterns of brain functional dynamics in a large sample of 314 patients with MDD (65.9% female; age (mean ± standard deviation): 35.9 ± 13.4) and 498 healthy controls (59.4% female; age: 34.0 ± 12.8). The HMM model was used to explain variations in rs-fMRI functional connectivity and averaged functional activity across the whole-brain by using a set of six unique recurring states. This study compared the proportion of time spent in each state and the average duration of visits to each state to assess stability between different groups. Compared to healthy controls, patients with MDD showed significantly higher proportional time spent and temporal stability in a state characterized by weak functional connectivity within and between all brain networks and relatively strong averaged functional activity of regions located in the somatosensory motor (SMN), salience (SN), and dorsal attention (DAN) networks. Both proportional time spent and temporal stability of this brain state was significantly associated with depression severity. Healthy controls, in contrast to the MDD group, showed proportional time spent and temporal stability in a state with relatively strong functional connectivity within and between all brain networks but weak averaged functional activity across the whole brain. These findings suggest that disrupted brain functional synchrony across time is present in MDD and associated with current depression severity.

Towards a network control theory of electroconvulsive therapy response.

Electroconvulsive Therapy (ECT) is arguably the most effective intervention for treatment-resistant depression. While large interindividual variability exists, a theory capable of explaining individual response to ECT remains elusive. To address this, we posit a quantitative, mechanistic framework of ECT response based on Network Control Theory (NCT). Then, we empirically test our approach and employ it to predict ECT treatment response. To this end, we derive a formal association between Postictal Suppression Index (PSI)-an ECT seizure quality index-and whole-brain modal and average controllability, NCT metrics based on white-matter brain network architecture, respectively. Exploiting the known association of ECT response and PSI, we then hypothesized an association between our controllability metrics and ECT response mediated by PSI. We formally tested this conjecture in N = 50 depressive patients undergoing ECT. We show that whole-brain controllability metrics based on pre-ECT structural connectome data predict ECT response in accordance with our hypotheses. In addition, we show the expected mediation effects via PSI. Importantly, our theoretically motivated metrics are at least on par with extensive machine learning models based on pre-ECT connectome data. In summary, we derived and tested a control-theoretic framework capable of predicting ECT response based on individual brain network architecture. It makes testable, quantitative predictions regarding individual therapeutic response, which are corroborated by strong empirical evidence. Our work might constitute a starting point for a comprehensive, quantitative theory of personalized ECT interventions rooted in control theory.

Genetic, individual, and familial risk correlates of brain network controllability in major depressive disorder.

Many therapeutic interventions in psychiatry can be viewed as attempts to influence the brain's large-scale, dynamic network state transitions. Building on connectome-based graph analysis and control theory, Network Control Theory is emerging as a powerful tool to quantify network controllability-i.e., the influence of one brain region over others regarding dynamic network state transitions. If and how network controllability is related to mental health remains elusive. Here, from Diffusion Tensor Imaging data, we inferred structural connectivity and inferred calculated network controllability parameters to investigate their association with genetic and familial risk in patients diagnosed with major depressive disorder (MDD, n = 692) and healthy controls (n = 820). First, we establish that controllability measures differ between healthy controls and MDD patients while not varying with current symptom severity or remission status. Second, we show that controllability in MDD patients is associated with polygenic scores for MDD and psychiatric cross-disorder risk. Finally, we provide evidence that controllability varies with familial risk of MDD and bipolar disorder as well as with body mass index. In summary, we show that network controllability is related to genetic, individual, and familial risk in MDD patients. We discuss how these insights into individual variation of network controllability may inform mechanistic models of treatment response prediction and personalized intervention-design in mental health.

Canonical Correlation Analysis and Partial Least Squares for Identifying Brain-Behavior Associations: A Tutorial and a Comparative Study.

Canonical correlation analysis (CCA) and partial least squares (PLS) are powerful multivariate methods for capturing associations across 2 modalities of data (e.g., brain and behavior). However, when the sample size is similar to or smaller than the number of variables in the data, standard CCA and PLS models may overfit, i.e., find spurious associations that generalize poorly to new data. Dimensionality reduction and regularized extensions of CCA and PLS have been proposed to address this problem, yet most studies using these approaches have some limitations. This work gives a theoretical and practical introduction into the most common CCA/PLS models and their regularized variants. We examine the limitations of standard CCA and PLS when the sample size is similar to or smaller than the number of variables. We discuss how dimensionality reduction and regularization techniques address this problem and explain their main advantages and disadvantages. We highlight crucial aspects of the CCA/PLS analysis framework, including optimizing the hyperparameters of the model and testing the identified associations for statistical significance. We apply the described CCA/PLS models to simulated data and real data from the Human Connectome Project and Alzheimer's Disease Neuroimaging Initiative (both of n > 500). We use both low- and high-dimensionality versions of these data (i.e., ratios between sample size and variables in the range of ∼1-10 and ∼0.1-0.01, respectively) to demonstrate the impact of data dimensionality on the models. Finally, we summarize the key lessons of the tutorial.

Brain structural correlates of recurrence following the first episode in patients with major depressive disorder.

Former prospective studies showed that the occurrence of relapse in Major Depressive Disorder (MDD) is associated with volume loss in the insula, hippocampus and dorsolateral prefrontal cortex (DLPFC). However, these studies were confounded by the patient's lifetime disease history, as the number of previous episodes predict future recurrence. In order to analyze neural correlates of recurrence irrespective of prior disease course, this study prospectively examined changes in brain structure in patients with first-episode depression (FED) over 2 years. N = 63 FED patients and n = 63 healthy controls (HC) underwent structural magnetic resonance imaging at baseline and after 2 years. According to their disease course during the follow-up interval, patients were grouped into n = 21 FED patients with recurrence (FEDrec) during follow-up and n = 42 FED patients with stable remission (FEDrem). Gray matter volume changes were analysed using group by time interaction analyses of covariance for the DLPFC, hippocampus and insula. Significant group by time interactions in the DLPFC and insula emerged. Pairwise comparisons showed that FEDrec had greater volume decline in the DLPFC and insula from baseline to follow-up compared with FEDrem and HC. No group by time interactions in the hippocampus were found. Cross-sectional analyses at baseline and follow-up revealed no differences between groups. This longitudinal study provides evidence for neural alterations in the DLPFC and insula related to a detrimental course in MDD. These effects of recurrence are already detectable at initial stages of MDD and seem to occur without any prior disease history, emphasizing the importance of early interventions preventing depressive recurrence.

Quantifying Deviations of Brain Structure and Function in Major Depressive Disorder Across Neuroimaging Modalities.

Importance: Identifying neurobiological differences between patients with major depressive disorder (MDD) and healthy individuals has been a mainstay of clinical neuroscience for decades. However, recent meta-analyses have raised concerns regarding the replicability and clinical relevance of brain alterations in depression. Objective: To quantify the upper bounds of univariate effect sizes, estimated predictive utility, and distributional dissimilarity of healthy individuals and those with depression across structural magnetic resonance imaging (MRI), diffusion-tensor imaging, and functional task-based as well as resting-state MRI, and to compare results with an MDD polygenic risk score (PRS) and environmental variables. Design, Setting, and Participants: This was a cross-sectional, case-control clinical neuroimaging study. Data were part of the Marburg-Münster Affective Disorders Cohort Study. Patients with depression and healthy controls were recruited from primary care and the general population in Münster and Marburg, Germany. Study recruitment was performed from September 11, 2014, to September 26, 2018. The sample comprised patients with acute and chronic MDD as well as healthy controls in the age range of 18 to 65 years. Data were analyzed from October 29, 2020, to April 7, 2022. Main Outcomes and Measures: Primary analyses included univariate partial effect size (η2), classification accuracy, and distributional overlapping coefficient for healthy individuals and those with depression across neuroimaging modalities, controlling for age, sex, and additional modality-specific confounding variables. Secondary analyses included patient subgroups for acute or chronic depressive status. Results: A total of 1809 individuals (861 patients [47.6%] and 948 controls [52.4%]) were included in the analysis (mean [SD] age, 35.6 [13.2] years; 1165 female patients [64.4%]). The upper bound of the effect sizes of the single univariate measures displaying the largest group difference ranged from partial η2 of 0.004 to 0.017, and distributions overlapped between 87% and 95%, with classification accuracies ranging between 54% and 56% across neuroimaging modalities. This pattern remained virtually unchanged when considering either only patients with acute or chronic depression. Differences were comparable with those found for PRS but substantially smaller than for environmental variables. Conclusions and Relevance: Results of this case-control study suggest that even for maximum univariate biological differences, deviations between patients with MDD and healthy controls were remarkably small, single-participant prediction was not possible, and similarity between study groups dominated. Biological psychiatry should facilitate meaningful outcome measures or predictive approaches to increase the potential for a personalization of the clinical practice.

Recommendations for machine learning benchmarks in neuroimaging.

The field of neuroimaging has embraced methods from machine learning in a variety of ways. Although an increasing number of initiatives have published open-access neuroimaging datasets, specifically designed benchmarks are rare in the field. In this article, we first describe how benchmarks in computer science and biomedical imaging have fostered methodological progress in machine learning. Second, we identify the special characteristics of neuroimaging data and outline what researchers have to ensure when establishing a neuroimaging benchmark, how datasets should be composed and how adequate evaluation criteria can be chosen. Based on lessons learned from machine learning benchmarks, we argue for an extended evaluation procedure that, next to applying suitable performance metrics, focuses on scientifically relevant aspects such as explainability, robustness, uncertainty, computational efficiency and code quality. Lastly, we envision a collaborative neuroimaging benchmarking platform that combines the discussed aspects in a collaborative and agile framework, allowing researchers across disciplines to work together on the key predictive problems of the field of neuroimaging and psychiatry.

An uncertainty-aware, shareable, and transparent neural network architecture for brain-age modeling.

The deviation between chronological age and age predicted from neuroimaging data has been identified as a sensitive risk marker of cross-disorder brain changes, growing into a cornerstone of biological age research. However, machine learning models underlying the field do not consider uncertainty, thereby confounding results with training data density and variability. Also, existing models are commonly based on homogeneous training sets, often not independently validated, and cannot be shared because of data protection issues. Here, we introduce an uncertainty-aware, shareable, and transparent Monte Carlo dropout composite quantile regression (MCCQR) Neural Network trained on N = 10,691 datasets from the German National Cohort. The MCCQR model provides robust, distribution-free uncertainty quantification in high-dimensional neuroimaging data, achieving lower error rates compared with existing models. In two examples, we demonstrate that it prevents spurious associations and increases power to detect deviant brain aging. We make the pretrained model and code publicly available.

White matter fiber microstructure is associated with prior hospitalizations rather than acute symptomatology in major depressive disorder.

BACKGROUND: Eighty percent of all patients suffering from major depressive disorder (MDD) relapse at least once in their lifetime. Thus, understanding the neurobiological underpinnings of the course of MDD is of utmost importance. A detrimental course of illness in MDD was most consistently associated with superior longitudinal fasciculus (SLF) fiber integrity. As similar associations were, however, found between SLF fiber integrity and acute symptomatology, this study attempts to disentangle associations attributed to current depression from long-term course of illness. METHODS: A total of 531 patients suffering from acute (N = 250) or remitted (N = 281) MDD from the FOR2107-cohort were analyzed in this cross-sectional study using tract-based spatial statistics for diffusion tensor imaging. First, the effects of disease state (acute v. remitted), current symptom severity (BDI-score) and course of illness (number of hospitalizations) on fractional anisotropy (FA), mean diffusivity (MD), radial diffusivity (RD), and axial diffusivity were analyzed separately. Second, disease state and BDI-scores were analyzed in conjunction with the number of hospitalizations to disentangle their effects. RESULTS: Disease state (pFWE < 0.042) and number of hospitalizations (pFWE< 0.032) were associated with decreased FA and increased MD and RD in the bilateral SLF. A trend was found for the BDI-score (pFWE > 0.067). When analyzed simultaneously only the effect of course of illness remained significant (pFWE < 0.040) mapping to the right SLF. CONCLUSIONS: Decreased FA and increased MD and RD values in the SLF are associated with more hospitalizations when controlling for current psychopathology. SLF fiber integrity could reflect cumulative illness burden at a neurobiological level and should be targeted in future longitudinal analyses.

Association of brain white matter microstructure with cognitive performance in major depressive disorder and healthy controls: a diffusion-tensor imaging study.

Cognitive deficits are central attendant symptoms of major depressive disorder (MDD) with a crucial impact in patients' everyday life. Thus, it is of particular clinical importance to understand their pathophysiology. The aim of this study was to investigate a possible relationship between brain structure and cognitive performance in MDD patients in a well-characterized sample. N = 1007 participants (NMDD = 482, healthy controls (HC): NHC = 525) were selected from the FOR2107 cohort for this diffusion-tensor imaging study employing tract-based spatial statistics. We conducted a principal component analysis (PCA) to reduce neuropsychological test results, and to discover underlying factors of cognitive performance in MDD patients. We tested the association between fractional anisotropy (FA) and diagnosis (MDD vs. HC) and cognitive performance factors. The PCA yielded a single general cognitive performance factor that differed significantly between MDD patients and HC (P < 0.001). We found a significant main effect of the general cognitive performance factor in FA (Ptfce-FWE = 0.002) in a large bilateral cluster consisting of widespread frontotemporal-association fibers. In MDD patients this effect was independent of medication intake, the presence of comorbid diagnoses, the number of previous hospitalizations, and depressive symptomatology. This study provides robust evidence that white matter disturbances and cognitive performance seem to be associated. This association was independent of diagnosis, though MDD patients show more pronounced deficits and lower FA values in the global white matter fiber structure. This suggests a more general, rather than the depression-specific neurological basis for cognitive deficits.

Identification of transdiagnostic psychiatric disorder subtypes using unsupervised learning.

Psychiatric disorders show heterogeneous symptoms and trajectories, with current nosology not accurately reflecting their molecular etiology and the variability and symptomatic overlap within and between diagnostic classes. This heterogeneity impedes timely and targeted treatment. Our study aimed to identify psychiatric patient clusters that share clinical and genetic features and may profit from similar therapies. We used high-dimensional data clustering on deep clinical data to identify transdiagnostic groups in a discovery sample (N = 1250) of healthy controls and patients diagnosed with depression, bipolar disorder, schizophrenia, schizoaffective disorder, and other psychiatric disorders. We observed five diagnostically mixed clusters and ordered them based on severity. The least impaired cluster 0, containing most healthy controls, showed general well-being. Clusters 1-3 differed predominantly regarding levels of maltreatment, depression, daily functioning, and parental bonding. Cluster 4 contained most patients diagnosed with psychotic disorders and exhibited the highest severity in many dimensions, including medication load. Depressed patients were present in all clusters, indicating that we captured different disease stages or subtypes. We replicated all but the smallest cluster 1 in an independent sample (N = 622). Next, we analyzed genetic differences between clusters using polygenic scores (PGS) and the psychiatric family history. These genetic variables differed mainly between clusters 0 and 4 (prediction area under the receiver operating characteristic curve (AUC) = 81%; significant PGS: cross-disorder psychiatric risk, schizophrenia, and educational attainment). Our results confirm that psychiatric disorders consist of heterogeneous subtypes sharing molecular factors and symptoms. The identification of transdiagnostic clusters advances our understanding of the heterogeneity of psychiatric disorders and may support the development of personalized treatments.

Systematic misestimation of machine learning performance in neuroimaging studies of depression.

We currently observe a disconcerting phenomenon in machine learning studies in psychiatry: While we would expect larger samples to yield better results due to the availability of more data, larger machine learning studies consistently show much weaker performance than the numerous small-scale studies. Here, we systematically investigated this effect focusing on one of the most heavily studied questions in the field, namely the classification of patients suffering from Major Depressive Disorder (MDD) and healthy controls based on neuroimaging data. Drawing upon structural MRI data from a balanced sample of N = 1868 MDD patients and healthy controls from our recent international Predictive Analytics Competition (PAC), we first trained and tested a classification model on the full dataset which yielded an accuracy of 61%. Next, we mimicked the process by which researchers would draw samples of various sizes (N = 4 to N = 150) from the population and showed a strong risk of misestimation. Specifically, for small sample sizes (N = 20), we observe accuracies of up to 95%. For medium sample sizes (N = 100) accuracies up to 75% were found. Importantly, further investigation showed that sufficiently large test sets effectively protect against performance misestimation whereas larger datasets per se do not. While these results question the validity of a substantial part of the current literature, we outline the relatively low-cost remedy of larger test sets, which is readily available in most cases.