1000 Genomes Project phase 4: The gift that keeps on giving.

In this issue of Cell, Bryska-Bishop et al. report the release of the expanded, high-depth sequencing data that characterize the fourth phase of the 1000 Genomes Project. Using extensive comparisons and benchmarks, they demonstrate how this dataset is positioned to serve as a more comprehensive and accurate resource for global genomics.

Benchmarking Metagenomics Tools for Taxonomic Classification.

Metagenomic sequencing is revolutionizing the detection and characterization of microbial species, and a wide variety of software tools are available to perform taxonomic classification of these data. The fast pace of development of these tools and the complexity of metagenomic data make it important that researchers are able to benchmark their performance. Here, we review current approaches for metagenomic analysis and evaluate the performance of 20 metagenomic classifiers using simulated and experimental datasets. We describe the key metrics used to assess performance, offer a framework for the comparison of additional classifiers, and discuss the future of metagenomic data analysis.

Challenges and best practices in omics benchmarking.

Technological advances enabling massively parallel measurement of biological features - such as microarrays, high-throughput sequencing and mass spectrometry - have ushered in the omics era, now in its third decade. The resulting complex landscape of analytical methods has naturally fostered the growth of an omics benchmarking industry. Benchmarking refers to the process of objectively comparing and evaluating the performance of different computational or analytical techniques when processing and analysing large-scale biological data sets, such as transcriptomics, proteomics and metabolomics. With thousands of omics benchmarking studies published over the past 25 years, the field has matured to the point where the foundations of benchmarking have been established and well described. However, generating meaningful benchmarking data and properly evaluating performance in this complex domain remains challenging. In this Review, we highlight some common oversights and pitfalls in omics benchmarking. We also establish a methodology to bring the issues that can be addressed into focus and to be transparent about those that cannot: this takes the form of a spreadsheet template of guidelines for comprehensive reporting, intended to accompany publications. In addition, a survey of recent developments in benchmarking is provided as well as specific guidance for commonly encountered difficulties.

Scaling neural machine translation to 200 languages.

The development of neural techniques has opened up new avenues for research in machine translation. Today, neural machine translation (NMT) systems can leverage highly multilingual capacities and even perform zero-shot translation, delivering promising results in terms of language coverage and quality. However, scaling quality NMT requires large volumes of parallel bilingual data, which are not equally available for the 7,000+ languages in the world1. Focusing on improving the translation qualities of a relatively small group of high-resource languages comes at the expense of directing research attention to low-resource languages, exacerbating digital inequities in the long run. To break this pattern, here we introduce No Language Left Behind-a single massively multilingual model that leverages transfer learning across languages. We developed a conditional computational model based on the Sparsely Gated Mixture of Experts architecture2-7, which we trained on data obtained with new mining techniques tailored for low-resource languages. Furthermore, we devised multiple architectural and training improvements to counteract overfitting while training on thousands of tasks. We evaluated the performance of our model over 40,000 translation directions using tools created specifically for this purpose-an automatic benchmark (FLORES-200), a human evaluation metric (XSTS) and a toxicity detector that covers every language in our model. Compared with the previous state-of-the-art models, our model achieves an average of 44% improvement in translation quality as measured by BLEU. By demonstrating how to scale NMT to 200 languages and making all contributions in this effort freely available for non-commercial use, our work lays important groundwork for the development of a universal translation system.

A whole-slide foundation model for digital pathology from real-world data.

Digital pathology poses unique computational challenges, as a standard gigapixel slide may comprise tens of thousands of image tiles1-3. Prior models have often resorted to subsampling a small portion of tiles for each slide, thus missing the important slide-level context4. Here we present Prov-GigaPath, a whole-slide pathology foundation model pretrained on 1.3 billion 256 × 256 pathology image tiles in 171,189 whole slides from Providence, a large US health network comprising 28 cancer centres. The slides originated from more than 30,000 patients covering 31 major tissue types. To pretrain Prov-GigaPath, we propose GigaPath, a novel vision transformer architecture for pretraining gigapixel pathology slides. To scale GigaPath for slide-level learning with tens of thousands of image tiles, GigaPath adapts the newly developed LongNet5 method to digital pathology. To evaluate Prov-GigaPath, we construct a digital pathology benchmark comprising 9 cancer subtyping tasks and 17 pathomics tasks, using both Providence and TCGA data6. With large-scale pretraining and ultra-large-context modelling, Prov-GigaPath attains state-of-the-art performance on 25 out of 26 tasks, with significant improvement over the second-best method on 18 tasks. We further demonstrate the potential of Prov-GigaPath on vision-language pretraining for pathology7,8 by incorporating the pathology reports. In sum, Prov-GigaPath is an open-weight foundation model that achieves state-of-the-art performance on various digital pathology tasks, demonstrating the importance of real-world data and whole-slide modelling.

Variant calling and benchmarking in an era of complete human genome sequences.

Genetic variant calling from DNA sequencing has enabled understanding of germline variation in hundreds of thousands of humans. Sequencing technologies and variant-calling methods have advanced rapidly, routinely providing reliable variant calls in most of the human genome. We describe how advances in long reads, deep learning, de novo assembly and pangenomes have expanded access to variant calls in increasingly challenging, repetitive genomic regions, including medically relevant regions, and how new benchmark sets and benchmarking methods illuminate their strengths and limitations. Finally, we explore the possible future of more complete characterization of human genome variation in light of the recent completion of a telomere-to-telomere human genome reference assembly and human pangenomes, and we consider the innovations needed to benchmark their newly accessible repetitive regions and complex variants.

Large language models encode clinical knowledge.

Large language models (LLMs) have demonstrated impressive capabilities, but the bar for clinical applications is high. Attempts to assess the clinical knowledge of models typically rely on automated evaluations based on limited benchmarks. Here, to address these limitations, we present MultiMedQA, a benchmark combining six existing medical question answering datasets spanning professional medicine, research and consumer queries and a new dataset of medical questions searched online, HealthSearchQA. We propose a human evaluation framework for model answers along multiple axes including factuality, comprehension, reasoning, possible harm and bias. In addition, we evaluate Pathways Language Model1 (PaLM, a 540-billion parameter LLM) and its instruction-tuned variant, Flan-PaLM2 on MultiMedQA. Using a combination of prompting strategies, Flan-PaLM achieves state-of-the-art accuracy on every MultiMedQA multiple-choice dataset (MedQA3, MedMCQA4, PubMedQA5 and Measuring Massive Multitask Language Understanding (MMLU) clinical topics6), including 67.6% accuracy on MedQA (US Medical Licensing Exam-style questions), surpassing the prior state of the art by more than 17%. However, human evaluation reveals key gaps. To resolve this, we introduce instruction prompt tuning, a parameter-efficient approach for aligning LLMs to new domains using a few exemplars. The resulting model, Med-PaLM, performs encouragingly, but remains inferior to clinicians. We show that comprehension, knowledge recall and reasoning improve with model scale and instruction prompt tuning, suggesting the potential utility of LLMs in medicine. Our human evaluations reveal limitations of today's models, reinforcing the importance of both evaluation frameworks and method development in creating safe, helpful LLMs for clinical applications.

Aligning climate scenarios to emissions inventories shifts global benchmarks.

Taking stock of global progress towards achieving the Paris Agreement requires consistently measuring aggregate national actions and pledges against modelled mitigation pathways1. However, national greenhouse gas inventories (NGHGIs) and scientific assessments of anthropogenic emissions follow different accounting conventions for land-based carbon fluxes resulting in a large difference in the present emission estimates2,3, a gap that will evolve over time. Using state-of-the-art methodologies4 and a land carbon-cycle emulator5, we align the Intergovernmental Panel on Climate Change (IPCC)-assessed mitigation pathways with the NGHGIs to make a comparison. We find that the key global mitigation benchmarks become harder to achieve when calculated using the NGHGI conventions, requiring both earlier net-zero CO2 timing and lower cumulative emissions. Furthermore, weakening natural carbon removal processes such as carbon fertilization can mask anthropogenic land-based removal efforts, with the result that land-based carbon fluxes in NGHGIs may ultimately become sources of emissions by 2100. Our results are important for the Global Stocktake6, suggesting that nations will need to increase the collective ambition of their climate targets to remain consistent with the global temperature goals.

Non-cell-autonomous cancer progression from chromosomal instability.

Chromosomal instability (CIN) is a driver of cancer metastasis1-4, yet the extent to which this effect depends on the immune system remains unknown. Using ContactTracing-a newly developed, validated and benchmarked tool to infer the nature and conditional dependence of cell-cell interactions from single-cell transcriptomic data-we show that CIN-induced chronic activation of the cGAS-STING pathway promotes downstream signal re-wiring in cancer cells, leading to a pro-metastatic tumour microenvironment. This re-wiring is manifested by type I interferon tachyphylaxis selectively downstream of STING and a corresponding increase in cancer cell-derived endoplasmic reticulum (ER) stress response. Reversal of CIN, depletion of cancer cell STING or inhibition of ER stress response signalling abrogates CIN-dependent effects on the tumour microenvironment and suppresses metastasis in immune competent, but not severely immune compromised, settings. Treatment with STING inhibitors reduces CIN-driven metastasis in melanoma, breast and colorectal cancers in a manner dependent on tumour cell-intrinsic STING. Finally, we show that CIN and pervasive cGAS activation in micronuclei are associated with ER stress signalling, immune suppression and metastasis in human triple-negative breast cancer, highlighting a viable strategy to identify and therapeutically intervene in tumours spurred by CIN-induced inflammation.

Challenges and perspectives in computational deconvolution of genomics data.

Deciphering cell-type heterogeneity is crucial for systematically understanding tissue homeostasis and its dysregulation in diseases. Computational deconvolution is an efficient approach for estimating cell-type abundances from a variety of omics data. Despite substantial methodological progress in computational deconvolution in recent years, challenges are still outstanding. Here we enlist four important challenges related to computational deconvolution: the quality of the reference data, generation of ground truth data, limitations of computational methodologies, and benchmarking design and implementation. Finally, we make recommendations on reference data generation, new directions of computational methodologies, and strategies to promote rigorous benchmarking.

Benchmarking spatial clustering methods with spatially resolved transcriptomics data.

Spatial clustering, which shares an analogy with single-cell clustering, has expanded the scope of tissue physiology studies from cell-centroid to structure-centroid with spatially resolved transcriptomics (SRT) data. Computational methods have undergone remarkable development in recent years, but a comprehensive benchmark study is still lacking. Here we present a benchmark study of 13 computational methods on 34 SRT data (7 datasets). The performance was evaluated on the basis of accuracy, spatial continuity, marker genes detection, scalability, and robustness. We found existing methods were complementary in terms of their performance and functionality, and we provide guidance for selecting appropriate methods for given scenarios. On testing additional 22 challenging datasets, we identified challenges in identifying noncontinuous spatial domains and limitations of existing methods, highlighting their inadequacies in handling recent large-scale tasks. Furthermore, with 145 simulated data, we examined the robustness of these methods against four different factors, and assessed the impact of pre- and postprocessing approaches. Our study offers a comprehensive evaluation of existing spatial clustering methods with SRT data, paving the way for future advancements in this rapidly evolving field.

TISSUE: uncertainty-calibrated prediction of single-cell spatial transcriptomics improves downstream analyses.

Whole-transcriptome spatial profiling of genes at single-cell resolution remains a challenge. To address this limitation, spatial gene expression prediction methods have been developed to infer the spatial expression of unmeasured transcripts, but the quality of these predictions can vary greatly. Here we present Transcript Imputation with Spatial Single-cell Uncertainty Estimation (TISSUE) as a general framework for estimating uncertainty for spatial gene expression predictions and providing uncertainty-aware methods for downstream inference. Leveraging conformal inference, TISSUE provides well-calibrated prediction intervals for predicted expression values across 11 benchmark datasets. Moreover, it consistently reduces the false discovery rate for differential gene expression analysis, improves clustering and visualization of predicted spatial transcriptomics and improves the performance of supervised learning models trained on predicted gene expression profiles. Applying TISSUE to a MERFISH spatial transcriptomics dataset of the adult mouse subventricular zone, we identified subtypes within the neural stem cell lineage and developed subtype-specific regional classifiers.

Unrepresentative big surveys significantly overestimated US vaccine uptake.

Surveys are a crucial tool for understanding public opinion and behaviour, and their accuracy depends on maintaining statistical representativeness of their target populations by minimizing biases from all sources. Increasing data size shrinks confidence intervals but magnifies the effect of survey bias: an instance of the Big Data Paradox1. Here we demonstrate this paradox in estimates of first-dose COVID-19 vaccine uptake in US adults from 9 January to 19 May 2021 from two large surveys: Delphi-Facebook2,3 (about 250,000 responses per week) and Census Household Pulse4 (about 75,000 every two weeks). In May 2021, Delphi-Facebook overestimated uptake by 17 percentage points (14-20 percentage points with 5% benchmark imprecision) and Census Household Pulse by 14 (11-17 percentage points with 5% benchmark imprecision), compared to a retroactively updated benchmark the Centers for Disease Control and Prevention published on 26 May 2021. Moreover, their large sample sizes led to miniscule margins of error on the incorrect estimates. By contrast, an Axios-Ipsos online panel5 with about 1,000 responses per week following survey research best practices6 provided reliable estimates and uncertainty quantification. We decompose observed error using a recent analytic framework1 to explain the inaccuracy in the three surveys. We then analyse the implications for vaccine hesitancy and willingness. We show how a survey of 250,000 respondents can produce an estimate of the population mean that is no more accurate than an estimate from a simple random sample of size 10. Our central message is that data quality matters more than data quantity, and that compensating the former with the latter is a mathematically provable losing proposition.

Diverging neural dynamics for syntactic structure building in naturalistic speaking and listening.

The neural correlates of sentence production are typically studied using task paradigms that differ considerably from the experience of speaking outside of an experimental setting. In this fMRI study, we aimed to gain a better understanding of syntactic processing in spontaneous production versus naturalistic comprehension in three regions of interest (BA44, BA45, and left posterior middle temporal gyrus). A group of participants (n = 16) was asked to speak about the events of an episode of a TV series in the scanner. Another group of participants (n = 36) listened to the spoken recall of a participant from the first group. To model syntactic processing, we extracted word-by-word metrics of phrase-structure building with a top-down and a bottom-up parser that make different hypotheses about the timing of structure building. While the top-down parser anticipates syntactic structure, sometimes before it is obvious to the listener, the bottom-up parser builds syntactic structure in an integratory way after all of the evidence has been presented. In comprehension, neural activity was found to be better modeled by the bottom-up parser, while in production, it was better modeled by the top-down parser. We additionally modeled structure building in production with two strategies that were developed here to make different predictions about the incrementality of structure building during speaking. We found evidence for highly incremental and anticipatory structure building in production, which was confirmed by a converging analysis of the pausing patterns in speech. Overall, this study shows the feasibility of studying the neural dynamics of spontaneous language production.

Platform-independent estimation of human physiological time from single blood samples.

Abundant epidemiological evidence links circadian rhythms to human health, from heart disease to neurodegeneration. Accurate determination of an individual's circadian phase is critical for precision diagnostics and personalized timing of therapeutic interventions. To date, however, we still lack an assay for physiological time that is accurate, minimally burdensome to the patient, and readily generalizable to new data. Here, we present TimeMachine, an algorithm to predict the human circadian phase using gene expression in peripheral blood mononuclear cells from a single blood draw. Once trained on data from a single study, we validated the trained predictor against four independent datasets with distinct experimental protocols and assay platforms, demonstrating that it can be applied generalizably. Importantly, TimeMachine predicted circadian time with a median absolute error ranging from 1.65 to 2.7 h, regardless of systematic differences in experimental protocol and assay platform, without renormalizing the data or retraining the predictor. This feature enables it to be flexibly applied to both new samples and existing data without limitations on the transcriptomic profiling technology (microarray, RNAseq). We benchmark TimeMachine against competing approaches and identify the algorithmic features that contribute to its performance.

Insights into hippocampal perfusion using high-resolution, multi-modal 7T MRI.

We present a comprehensive study on the non-invasive measurement of hippocampal perfusion. Using high-resolution 7 tesla arterial spin labeling (ASL) data, we generated robust perfusion maps and observed significant variations in perfusion among hippocampal subfields, with CA1 exhibiting the lowest perfusion levels. Notably, these perfusion differences were robust and already detectable with 50 perfusion-weighted images per subject, acquired in 5 min. To understand the underlying factors, we examined the influence of image quality metrics, various tissue microstructure and morphometric properties, macrovasculature, and cytoarchitecture. We observed higher perfusion in regions located closer to arteries, demonstrating the influence of vascular proximity on hippocampal perfusion. Moreover, ex vivo cytoarchitectonic features based on neuronal density differences appeared to correlate stronger with hippocampal perfusion than morphometric measures like gray matter thickness. These findings emphasize the interplay between microvasculature, macrovasculature, and metabolic demand in shaping hippocampal perfusion. Our study expands the current understanding of hippocampal physiology and its relevance to neurological disorders. By providing in vivo evidence of perfusion differences between hippocampal subfields, our findings have implications for diagnosis and potential therapeutic interventions. In conclusion, our study provides a valuable resource for extensively characterizing hippocampal perfusion.

An RNA-informed dosage sensitivity map reflects the intrinsic functional nature of genes.

Understanding dosage sensitivity or why Mendelian diseases have dominant vs. recessive modes of inheritance is crucial for uncovering the etiology of human disease. Previous knowledge of dosage sensitivity is mainly based on observations of rare loss-of-function mutations or copy number changes, which are underpowered due to ultra rareness of such variants. Thus, the functional underpinnings of dosage constraint remain elusive. In this study, we aim to systematically quantify dosage perturbations from cis-regulatory variants in the general population to yield a tissue-specific dosage constraint map of genes and further explore their underlying functional logic. We reveal an inherent divergence of dosage constraints in genes by functional categories with signaling genes (transcription factors, protein kinases, ion channels, and cellular machinery) being dosage sensitive, while effector genes (transporters, metabolic enzymes, cytokines, and receptors) are generally dosage resilient. Instead of being a metric of functional dispensability, we show that dosage constraint reflects underlying homeostatic constraints arising from negative feedback. Finally, we employ machine learning to integrate DNA and RNA metrics to generate a comprehensive, tissue-specific map of dosage sensitivity (MoDs) for autosomal genes.

Evolution of Partial Resistance to Artemisinins in Malaria Parasites in Uganda.

BACKGROUND: Partial resistance of Plasmodium falciparum to the artemisinin component of artemisinin-based combination therapies, the most important malaria drugs, emerged in Southeast Asia and now threatens East Africa. Partial resistance, which manifests as delayed clearance after therapy, is mediated principally by mutations in the kelch protein K13 (PfK13). Limited longitudinal data are available on the emergence and spread of artemisinin resistance in Africa. METHODS: We performed annual surveillance among patients who presented with uncomplicated malaria at 10 to 16 sites across Uganda from 2016 through 2022. We sequenced the gene encoding kelch 13 (pfk13) and analyzed relatedness using molecular methods. We assessed malaria metrics longitudinally in eight Ugandan districts from 2014 through 2021. RESULTS: By 2021-2022, the prevalence of parasites with validated or candidate resistance markers reached more than 20% in 11 of the 16 districts where surveillance was conducted. The PfK13 469Y and 675V mutations were seen in far northern Uganda in 2016-2017 and increased and spread thereafter, reaching a combined prevalence of 10 to 54% across much of northern Uganda, with spread to other regions. The 469F mutation reached a prevalence of 38 to 40% in one district in southwestern Uganda in 2021-2022. The 561H mutation, previously described in Rwanda, was first seen in southwestern Uganda in 2021, reaching a prevalence of 23% by 2022. The 441L mutation reached a prevalence of 12 to 23% in three districts in western Uganda in 2022. Genetic analysis indicated local emergence of mutant parasites independent of those in Southeast Asia. The emergence of resistance was observed predominantly in areas where effective malaria control had been discontinued or transmission was unstable. CONCLUSIONS: Data from Uganda showed the emergence of partial resistance to artemisinins in multiple geographic locations, with increasing prevalence and regional spread over time. (Funded by the National Institutes of Health.).

A framework for evaluating the performance of SMLM cluster analysis algorithms.

Single-molecule localization microscopy (SMLM) generates data in the form of coordinates of localized fluorophores. Cluster analysis is an attractive route for extracting biologically meaningful information from such data and has been widely applied. Despite a range of cluster analysis algorithms, there exists no consensus framework for the evaluation of their performance. Here, we use a systematic approach based on two metrics to score the success of clustering algorithms in simulated conditions mimicking experimental data. We demonstrate the framework using seven diverse analysis algorithms: DBSCAN, ToMATo, KDE, FOCAL, CAML, ClusterViSu and SR-Tesseler. Given that the best performer depended on the underlying distribution of localizations, we demonstrate an analysis pipeline based on statistical similarity measures that enables the selection of the most appropriate algorithm, and the optimized analysis parameters for real SMLM data. We propose that these standard simulated conditions, metrics and analysis pipeline become the basis for future analysis algorithm development and evaluation.