Critical Assessment of Metagenome Interpretation: the second round of challenges.

Evaluating metagenomic software is key for optimizing metagenome interpretation and focus of the Initiative for the Critical Assessment of Metagenome Interpretation (CAMI). The CAMI II challenge engaged the community to assess methods on realistic and complex datasets with long- and short-read sequences, created computationally from around 1,700 new and known genomes, as well as 600 new plasmids and viruses. Here we analyze 5,002 results by 76 program versions. Substantial improvements were seen in assembly, some due to long-read data. Related strains still were challenging for assembly and genome recovery through binning, as was assembly quality for the latter. Profilers markedly matured, with taxon profilers and binners excelling at higher bacterial ranks, but underperforming for viruses and Archaea. Clinical pathogen detection results revealed a need to improve reproducibility. Runtime and memory usage analyses identified efficient programs, including top performers with other metrics. The results identify challenges and guide researchers in selecting methods for analyses.

Critical Assessment of Metagenome Interpretation-a benchmark of metagenomics software.

Methods for assembly, taxonomic profiling and binning are key to interpreting metagenome data, but a lack of consensus about benchmarking complicates performance assessment. The Critical Assessment of Metagenome Interpretation (CAMI) challenge has engaged the global developer community to benchmark their programs on highly complex and realistic data sets, generated from ∼700 newly sequenced microorganisms and ∼600 novel viruses and plasmids and representing common experimental setups. Assembly and genome binning programs performed well for species represented by individual genomes but were substantially affected by the presence of related strains. Taxonomic profiling and binning programs were proficient at high taxonomic ranks, with a notable performance decrease below family level. Parameter settings markedly affected performance, underscoring their importance for program reproducibility. The CAMI results highlight current challenges but also provide a roadmap for software selection to answer specific research questions.

Assessment of the specificity of Burkholderia and Pseudomonas qPCR assays for detection of these genera in soil using 454 pyrosequencing.

In this study, two highly specific quantitative PCR assays targeting the bacterial genera Burkholderia and Pseudomonas were developed and evaluated on soil samples. The primers were targeting different multivariate regions of the 16S rRNA gene and designed to be compatible with quantitative PCR and the high throughput 454 pyrosequencing technique. The developed assays were validated using the standard methods. All tests with the new developed assays showed very high specificity. Pyrosequencing was used for direct analysis of the PCR product and applied as a specificity measurement of the primers. The Pseudomonas primers showed a 99% primer specificity, which covered 200 different Pseudomonas sequence clusters in 0.5 g of soil. In contrast to that the same approach using the genus-specific Burkholderia primers showed only 8% primer specificity. This discrepancy in primer specificity between the normal procedures compared with pyrosequencing illustrates that the common validation procedures for quantitative PCR primers may be misleading. Our results exemplify the fact that current 16S RNA gene sequence databases might lack resolution within many taxonomic groups and emphasize the necessity for a standardized and functional primer validation protocol. A possible solution to this could be to supplement the normal verification of quantitative PCR assays with a pyrosequencing approach.