Over-optimism in unsupervised microbiome analysis: Insights from network learning and clustering.

In recent years, unsupervised analysis of microbiome data, such as microbial network analysis and clustering, has increased in popularity. Many new statistical and computational methods have been proposed for these tasks. This multiplicity of analysis strategies poses a challenge for researchers, who are often unsure which method(s) to use and might be tempted to try different methods on their dataset to look for the "best" ones. However, if only the best results are selectively reported, this may cause over-optimism: the "best" method is overly fitted to the specific dataset, and the results might be non-replicable on validation data. Such effects will ultimately hinder research progress. Yet so far, these topics have been given little attention in the context of unsupervised microbiome analysis. In our illustrative study, we aim to quantify over-optimism effects in this context. We model the approach of a hypothetical microbiome researcher who undertakes four unsupervised research tasks: clustering of bacterial genera, hub detection in microbial networks, differential microbial network analysis, and clustering of samples. While these tasks are unsupervised, the researcher might still have certain expectations as to what constitutes interesting results. We translate these expectations into concrete evaluation criteria that the hypothetical researcher might want to optimize. We then randomly split an exemplary dataset from the American Gut Project into discovery and validation sets multiple times. For each research task, multiple method combinations (e.g., methods for data normalization, network generation, and/or clustering) are tried on the discovery data, and the combination that yields the best result according to the evaluation criterion is chosen. While the hypothetical researcher might only report this result, we also apply the "best" method combination to the validation dataset. The results are then compared between discovery and validation data. In all four research tasks, there are notable over-optimism effects; the results on the validation data set are worse compared to the discovery data, averaged over multiple random splits into discovery/validation data. Our study thus highlights the importance of validation and replication in microbiome analysis to obtain reliable results and demonstrates that the issue of over-optimism goes beyond the context of statistical testing and fishing for significance.

Development of prevalence and incidence of non-tuberculous mycobacteria in German laboratories from 2016 to 2020.

Numbers of non-tuberculous mycobacteria (NTM) pulmonary diseases (PD) have been repeatedly reported as increasing over the last decades, particularly in Europe. Sound epidemiological data are however missing for most European regions. This study calculated prevalence and incidence of NTM recovered from patients' lungs in Germany, the largest Central European country, over a five-year period. It furthermore determined regional particularities of NTM species and results from susceptibility testing. 22 German NTM laboratories provided their mycobacteriological diagnostic data of 11,430 NTM isolates recovered from 5998 pulmonary patients representing 30% of all notified NTM-PD cases of Germany from 2016 to 2020. NTM incidence and prevalence were calculated for every study year. The presented epidemiological indicators are particularly reliant as TB surveillance data were used as a reference and TB notification reaches almost 100% in Germany. Laboratory incidence and prevalence of NTM recovered from respiratory samples ranged from 4.5-4.9 and from 5.3-5.8/100,000 for the population of Germany, respectively, and did not change over the five-year study period. Prevalence and incidence were stable also when stratifying for facultative pathogenic NTM, M. avium/intracellulare complex (MAIC), and M. abscessus/chelonae complex (MABSC). The proportion of NTM with drug susceptibility testing (DST) increased from 27.3% (2016) to 43.8% (2020). The unchanging laboratory NTM prevalence/incidence in Germany represents a "ceiling" of possible NTM-PD notification when diagnostic strategies do not change in the coming years. A notable increase in NTM-DST may indicate better notification of NTM-PD and/or awareness of new clinical guidelines but still remains below clinical needs.