From patterns to patients: Advances in clinical machine learning for cancer diagnosis, prognosis, and treatment.

Machine learning (ML) is increasingly used in clinical oncology to diagnose cancers, predict patient outcomes, and inform treatment planning. Here, we review recent applications of ML across the clinical oncology workflow. We review how these techniques are applied to medical imaging and to molecular data obtained from liquid and solid tumor biopsies for cancer diagnosis, prognosis, and treatment design. We discuss key considerations in developing ML for the distinct challenges posed by imaging and molecular data. Finally, we examine ML models approved for cancer-related patient usage by regulatory agencies and discuss approaches to improve the clinical usefulness of ML.

Metabolic analysis as a driver for discovery, diagnosis, and therapy.

Metabolic anomalies contribute to tissue dysfunction. Current metabolism research spans from organelles to populations, and new technologies can accommodate investigation across these scales. Here, we review recent advancements in metabolic analysis, including small-scale metabolomics techniques amenable to organelles and rare cell types, functional screening to explore how cells respond to metabolic stress, and imaging approaches to non-invasively assess metabolic perturbations in diseases. We discuss how metabolomics provides an informative phenotypic dimension that complements genomic analysis in Mendelian and non-Mendelian disorders. We also outline pressing challenges and how addressing them may further clarify the biochemical basis of human disease.

Toward a Common Coordinate Framework for the Human Body.

Understanding the genetic and molecular drivers of phenotypic heterogeneity across individuals is central to biology. As new technologies enable fine-grained and spatially resolved molecular profiling, we need new computational approaches to integrate data from the same organ across different individuals into a consistent reference and to construct maps of molecular and cellular organization at histological and anatomical scales. Here, we review previous efforts and discuss challenges involved in establishing such a common coordinate framework, the underlying map of tissues and organs. We focus on strategies to handle anatomical variation across individuals and highlight the need for new technologies and analytical methods spanning multiple hierarchical scales of spatial resolution.

Identifying Medical Diagnoses and Treatable Diseases by Image-Based Deep Learning.

The implementation of clinical-decision support algorithms for medical imaging faces challenges with reliability and interpretability. Here, we establish a diagnostic tool based on a deep-learning framework for the screening of patients with common treatable blinding retinal diseases. Our framework utilizes transfer learning, which trains a neural network with a fraction of the data of conventional approaches. Applying this approach to a dataset of optical coherence tomography images, we demonstrate performance comparable to that of human experts in classifying age-related macular degeneration and diabetic macular edema. We also provide a more transparent and interpretable diagnosis by highlighting the regions recognized by the neural network. We further demonstrate the general applicability of our AI system for diagnosis of pediatric pneumonia using chest X-ray images. This tool may ultimately aid in expediting the diagnosis and referral of these treatable conditions, thereby facilitating earlier treatment, resulting in improved clinical outcomes. VIDEO ABSTRACT.

A multimodal generative AI copilot for human pathology.

Computational pathology1,2 has witnessed considerable progress in the development of both task-specific predictive models and task-agnostic self-supervised vision encoders3,4. However, despite the explosive growth of generative artificial intelligence (AI), there have been few studies on building general-purpose multimodal AI assistants and copilots5 tailored to pathology. Here we present PathChat, a vision-language generalist AI assistant for human pathology. We built PathChat by adapting a foundational vision encoder for pathology, combining it with a pretrained large language model and fine-tuning the whole system on over 456,000 diverse visual-language instructions consisting of 999,202 question and answer turns. We compare PathChat with several multimodal vision-language AI assistants and GPT-4V, which powers the commercially available multimodal general-purpose AI assistant ChatGPT-4 (ref. 6). PathChat achieved state-of-the-art performance on multiple-choice diagnostic questions from cases with diverse tissue origins and disease models. Furthermore, using open-ended questions and human expert evaluation, we found that overall PathChat produced more accurate and pathologist-preferable responses to diverse queries related to pathology. As an interactive vision-language AI copilot that can flexibly handle both visual and natural language inputs, PathChat may potentially find impactful applications in pathology education, research and human-in-the-loop clinical decision-making.

Integrated lithium niobate microwave photonic processing engine.

Integrated microwave photonics (MWP) is an intriguing technology for the generation, transmission and manipulation of microwave signals in chip-scale optical systems1,2. In particular, ultrafast processing of analogue signals in the optical domain with high fidelity and low latency could enable a variety of applications such as MWP filters3-5, microwave signal processing6-9 and image recognition10,11. An ideal integrated MWP processing platform should have both an efficient and high-speed electro-optic modulation block to faithfully perform microwave-optic conversion at low power and also a low-loss functional photonic network to implement various signal-processing tasks. Moreover, large-scale, low-cost manufacturability is required to monolithically integrate the two building blocks on the same chip. Here we demonstrate such an integrated MWP processing engine based on a 4 inch wafer-scale thin-film lithium niobate platform. It can perform multipurpose tasks with processing bandwidths of up to 67 GHz at complementary metal-oxide-semiconductor (CMOS)-compatible voltages. We achieve ultrafast analogue computation, namely temporal integration and differentiation, at sampling rates of up to 256 giga samples per second, and deploy these functions to showcase three proof-of-concept applications: solving ordinary differential equations, generating ultra-wideband signals and detecting edges in images. We further leverage the image edge detector to realize a photonic-assisted image segmentation model that can effectively outline the boundaries of melanoma lesion in medical diagnostic images. Our ultrafast lithium niobate MWP engine could provide compact, low-latency and cost-effective solutions for future wireless communications, high-resolution radar and photonic artificial intelligence.

Foundation models for generalist medical artificial intelligence.

The exceptionally rapid development of highly flexible, reusable artificial intelligence (AI) models is likely to usher in newfound capabilities in medicine. We propose a new paradigm for medical AI, which we refer to as generalist medical AI (GMAI). GMAI models will be capable of carrying out a diverse set of tasks using very little or no task-specific labelled data. Built through self-supervision on large, diverse datasets, GMAI will flexibly interpret different combinations of medical modalities, including data from imaging, electronic health records, laboratory results, genomics, graphs or medical text. Models will in turn produce expressive outputs such as free-text explanations, spoken recommendations or image annotations that demonstrate advanced medical reasoning abilities. Here we identify a set of high-impact potential applications for GMAI and lay out specific technical capabilities and training datasets necessary to enable them. We expect that GMAI-enabled applications will challenge current strategies for regulating and validating AI devices for medicine and will shift practices associated with the collection of large medical datasets.

HCR spectral imaging: 10-plex, quantitative, high-resolution RNA and protein imaging in highly autofluorescent samples.

Signal amplification based on the mechanism of hybridization chain reaction (HCR) provides a unified framework for multiplex, quantitative, high-resolution imaging of RNA and protein targets in highly autofluorescent samples. With conventional bandpass imaging, multiplexing is typically limited to four or five targets owing to the difficulty in separating signals generated by fluorophores with overlapping spectra. Spectral imaging has offered the conceptual promise of higher levels of multiplexing, but it has been challenging to realize this potential in highly autofluorescent samples, including whole-mount vertebrate embryos. Here, we demonstrate robust HCR spectral imaging with linear unmixing, enabling simultaneous imaging of ten RNA and/or protein targets in whole-mount zebrafish embryos and mouse brain sections. Further, we demonstrate that the amplified and unmixed signal in each of the ten channels is quantitative, enabling accurate and precise relative quantitation of RNA and/or protein targets with subcellular resolution, and RNA absolute quantitation with single-molecule resolution, in the anatomical context of highly autofluorescent samples.

Mapping enzyme activity in living systems by real-time mid-infrared photothermal imaging of nitrile chameleons.

Simultaneous spatial mapping of the activity of multiple enzymes in a living system can elucidate their functions in health and disease. However, methods based on monitoring fluorescent substrates are limited. Here, we report the development of nitrile (C≡N)-tagged enzyme activity reporters, named nitrile chameleons, for the peak shift between substrate and product. To image these reporters in real time, we developed a laser-scanning mid-infrared photothermal imaging system capable of imaging the enzymatic substrates and products at a resolution of 300 nm. We show that when combined, these tools can map the activity distribution of different enzymes and measure their relative catalytic efficiency in living systems such as cancer cells, Caenorhabditis elegans, and brain tissues, and can be used to directly visualize caspase-phosphatase interactions during apoptosis. Our method is generally applicable to a broad category of enzymes and will enable new analyses of enzymes in their native context.

Organ Mapping Antibody Panels: a community resource for standardized multiplexed tissue imaging.

Multiplexed antibody-based imaging enables the detailed characterization of molecular and cellular organization in tissues. Advances in the field now allow high-parameter data collection (>60 targets); however, considerable expertise and capital are needed to construct the antibody panels employed by these methods. Organ mapping antibody panels are community-validated resources that save time and money, increase reproducibility, accelerate discovery and support the construction of a Human Reference Atlas.

DNA-barcoded signal amplification for imaging mass cytometry enables sensitive and highly multiplexed tissue imaging.

Imaging mass cytometry (IMC) is a highly multiplexed, antibody-based imaging method that captures heterogeneous spatial protein expression patterns at subcellular resolution. Here we report the extension of IMC to low-abundance markers through incorporation of the DNA-based signal amplification by exchange reaction, immuno-SABER. We applied SABER-IMC to image the tumor immune microenvironment in human melanoma by simultaneous imaging of 18 markers with immuno-SABER and 20 markers without amplification. SABER-IMC enabled the identification of immune cell phenotypic markers, such as T cell co-receptors and their ligands, that are not detectable with IMC.

Optimizing multiplexed imaging experimental design through tissue spatial segregation estimation.

Recent advances in multiplexed imaging methods allow simultaneous detection of dozens of proteins and hundreds of RNAs, enabling deep spatial characterization of both healthy and diseased tissues. Parameters for the design of optimal multiplex imaging studies, especially those estimating how much area has to be imaged to capture all cell phenotype clusters, are lacking. Here, using a spatial transcriptomic atlas of healthy and tumor human tissues, we developed a statistical framework that determines the number and area of fields of view necessary to accurately identify all cell phenotypes that are part of a tissue. Using this strategy on imaging mass cytometry data, we identified a measurement of tissue spatial segregation that enables optimal experimental design. This strategy will enable an improved design of multiplexed imaging studies.

Accurate classification of major brain cell types using in vivo imaging and neural network processing.

Comprehensive analysis of tissue cell type composition using microscopic techniques has primarily been confined to ex vivo approaches. Here, we introduce NuCLear (Nucleus-instructed tissue composition using deep learning), an approach combining in vivo two-photon imaging of histone 2B-eGFP-labeled cell nuclei with subsequent deep learning-based identification of cell types from structural features of the respective cell nuclei. Using NuCLear, we were able to classify almost all cells per imaging volume in the secondary motor cortex of the mouse brain (0.25 mm3 containing approximately 25,000 cells) and to identify their position in 3D space in a noninvasive manner using only a single label throughout multiple imaging sessions. Twelve weeks after baseline, cell numbers did not change yet astrocytic nuclei significantly decreased in size. NuCLear opens a window to study changes in relative density and location of different cell types in the brains of individual mice over extended time periods, enabling comprehensive studies of changes in cell type composition in physiological and pathophysiological conditions.

Artificial intelligence in cancer imaging: Clinical challenges and applications.

Judgement, as one of the core tenets of medicine, relies upon the integration of multilayered data with nuanced decision making. Cancer offers a unique context for medical decisions given not only its variegated forms with evolution of disease but also the need to take into account the individual condition of patients, their ability to receive treatment, and their responses to treatment. Challenges remain in the accurate detection, characterization, and monitoring of cancers despite improved technologies. Radiographic assessment of disease most commonly relies upon visual evaluations, the interpretations of which may be augmented by advanced computational analyses. In particular, artificial intelligence (AI) promises to make great strides in the qualitative interpretation of cancer imaging by expert clinicians, including volumetric delineation of tumors over time, extrapolation of the tumor genotype and biological course from its radiographic phenotype, prediction of clinical outcome, and assessment of the impact of disease and treatment on adjacent organs. AI may automate processes in the initial interpretation of images and shift the clinical workflow of radiographic detection, management decisions on whether or not to administer an intervention, and subsequent observation to a yet to be envisioned paradigm. Here, the authors review the current state of AI as applied to medical imaging of cancer and describe advances in 4 tumor types (lung, brain, breast, and prostate) to illustrate how common clinical problems are being addressed. Although most studies evaluating AI applications in oncology to date have not been vigorously validated for reproducibility and generalizability, the results do highlight increasingly concerted efforts in pushing AI technology to clinical use and to impact future directions in cancer care.

Nanocrystals for Deep-Tissue In Vivo Luminescence Imaging in the Near-Infrared Region.

In vivo imaging technologies have emerged as a powerful tool for both fundamental research and clinical practice. In particular, luminescence imaging in the tissue-transparent near-infrared (NIR, 700-1700 nm) region offers tremendous potential for visualizing biological architectures and pathophysiological events in living subjects with deep tissue penetration and high imaging contrast owing to the reduced light-tissue interactions of absorption, scattering, and autofluorescence. The distinctive quantum effects of nanocrystals have been harnessed to achieve exceptional photophysical properties, establishing them as a promising category of luminescent probes. In this comprehensive review, the interactions between light and biological tissues, as well as the advantages of NIR light for in vivo luminescence imaging, are initially elaborated. Subsequently, we focus on achieving deep tissue penetration and improved imaging contrast by optimizing the performance of nanocrystal fluorophores. The ingenious design strategies of NIR nanocrystal probes are discussed, along with their respective biomedical applications in versatile in vivo luminescence imaging modalities. Finally, thought-provoking reflections on the challenges and prospects for future clinical translation of nanocrystal-based in vivo luminescence imaging in the NIR region are wisely provided.

Carbon Nanomaterial Fluorescent Probes and Their Biological Applications.

Fluorescent carbon nanomaterials have broadly useful chemical and photophysical attributes that are conducive to applications in biology. In this review, we focus on materials whose photophysics allow for the use of these materials in biomedical and environmental applications, with emphasis on imaging, biosensing, and cargo delivery. The review focuses primarily on graphitic carbon nanomaterials including graphene and its derivatives, carbon nanotubes, as well as carbon dots and carbon nanohoops. Recent advances in and future prospects of these fields are discussed at depth, and where appropriate, references to reviews pertaining to older literature are provided.

Activity-Based Fluorescent Probes for Hydrogen Sulfide and Related Reactive Sulfur Species.

Hydrogen sulfide (H2S) is not only a well-established toxic gas but also an important small molecule bioregulator in all kingdoms of life. In contemporary biology, H2S is often classified as a "gasotransmitter," meaning that it is an endogenously produced membrane permeable gas that carries out essential cellular processes. Fluorescent probes for H2S and related reactive sulfur species (RSS) detection provide an important cornerstone for investigating the multifaceted roles of these important small molecules in complex biological systems. A now common approach to develop such tools is to develop "activity-based probes" that couple a specific H2S-mediated chemical reaction to a fluorescent output. This Review covers the different types of such probes and also highlights the chemical mechanisms by which each probe type is activated by specific RSS. Common examples include reduction of oxidized nitrogen motifs, disulfide exchange, electrophilic reactions, metal precipitation, and metal coordination. In addition, we also outline complementary activity-based probes for imaging reductant-labile and sulfane sulfur species, including persulfides and polysulfides. For probes highlighted in this Review, we focus on small molecule systems with demonstrated compatibility in cellular systems or related applications. Building from breadth of reported activity-based strategies and application, we also highlight key unmet challenges and future opportunities for advancing activity-based probes for H2S and related RSS.

Cryo-EM structure determination of small therapeutic protein targets at 3 Å-resolution using a rigid imaging scaffold.

Cryoelectron microscopy (Cryo-EM) has enabled structural determination of proteins larger than about 50 kDa, including many intractable by any other method, but it has largely failed for smaller proteins. Here, we obtain structures of small proteins by binding them to a rigid molecular scaffold based on a designed protein cage, revealing atomic details at resolutions reaching 2.9 Å. We apply this system to the key cancer signaling protein KRAS (19 kDa in size), obtaining four structures of oncogenic mutational variants by cryo-EM. Importantly, a structure for the key G12C mutant bound to an inhibitor drug (AMG510) reveals significant conformational differences compared to prior data in the crystalline state. The findings highlight the promise of cryo-EM scaffolds for advancing the design of drug molecules against small therapeutic protein targets in cancer and other human diseases.

Orange/far-red hybrid voltage indicators with reduced phototoxicity enable reliable long-term imaging in neurons and cardiomyocytes.

Hybrid voltage indicators (HVIs) are chemogenetic sensors that combines the superior photophysical properties of organic dyes and the genetic targetability of protein sensors to report transient membrane voltage changes. They exhibit boosted sensitivity in excitable cells such as neurons and cardiomyocytes. However, the voltage signals recorded during long-term imaging are severely diminished or distorted due to phototoxicity and photobleaching issues. To capture stable electrophysiological activities over a long time, we employ cyanine dyes conjugated with a cyclooctatetraene (COT) molecule as the fluorescence reporter of HVI. The resulting orange-emitting HVI-COT-Cy3 enables high-fidelity voltage imaging for up to 30 min in cultured primary neurons with a sensitivity of ~ -30% ΔF/F0 per action potential (AP). It also maximally preserves the signal of individual APs in cardiomyocytes. The far-red-emitting HVI-COT-Cy5 allows two-color voltage/calcium imaging with GCaMP6s in neurons and cardiomyocytes for 15 min. We leverage the HVI-COT series with reduced phototoxicity and photobleaching to evaluate the impact of drug candidates on the electrophysiology of excitable cells.

Understanding immunity in a tissue-centric context: Combining novel imaging methods and mathematics to extract new insights into function and dysfunction.

A central question in immunology is what features allow the immune system to respond in a timely manner to a variety of pathogens encountered at unanticipated times and diverse body sites. Two decades of advanced and static dynamic imaging methods have now revealed several major principles facilitating host defense. Suborgan spatial prepositioning of distinct cells promotes time-efficient interactions upon pathogen sensing. Such pre-organization also provides an effective barrier to movement of pathogens from parenchymal tissues into the blood circulation. Various molecular mechanisms maintain effective intercellular communication among otherwise rapidly moving cells. These and related discoveries have benefited from recent increases in the number of parameters that can be measured simultaneously in a single tissue section and the extension of such multiplex analyses to 3D tissue volumes. The application of new computational methods to such imaging data has provided a quantitative, in vivo context for cell trafficking and signaling pathways traditionally explored in vitro or with dissociated cell preparations. Here, we summarize our efforts to devise and employ diverse imaging tools to probe immune system organization and function, concluding with a commentary on future developments, which we believe will reveal even more about how the immune system operates in health and disease.