Tuberculosis Chest X-Ray Image Retrieval System Using Deep Learning Based Biomarker Predictions.

The world health organization's global tuberculosis (TB) report for 2022 identifies TB, with an estimated 1.6 million, as a leading cause of death. The number of new cases has risen since 2020, particularly the number of new drug-resistant cases, estimated at 450,000 in 2021. This is concerning, as treatment of patients with drug resistant TB is complex and may not always be successful. The NIAID TB Portals program is an international consortium with a primary focus on patient centric data collection and analysis for drug resistant TB. The data includes images, their associated radiological findings, clinical records, and socioeconomic information. This work describes a TB Portals' Chest X-ray based image retrieval system which enables precision medicine. An input image is used to retrieve similar images and the associated patient specific information, thus facilitating inspection of outcomes and treatment regimens from comparable patients. Image similarity is defined using clinically relevant biomarkers: gender, age, body mass index (BMI), and the percentage of lung affected per sextant. The biomarkers are predicted using variations of the DenseNet169 convolutional neural network. A multi-task approach is used to predict gender, age and BMI incorporating transfer learning from an initial training on the NIH Clinical Center CXR dataset to the TB portals dataset. The resulting gender AUC, age and BMI mean absolute errors were 0.9854, 4.03years and 1.67kgm2. For the percentage of sextant affected by lesions the mean absolute errors ranged between 7% to 12% with higher error values in the middle and upper sextants which exhibit more variability than the lower sextants. The retrieval system is currently available from https://rap.tbportals.niaid.nih.gov/find_similar_cxr.

Multi-omic profiling of follicular lymphoma reveals changes in tissue architecture and enhanced stromal remodeling in high-risk patients.

Follicular lymphoma (FL) is a generally incurable malignancy that evolves from developmentally blocked germinal center (GC) B cells. To promote survival and immune escape, tumor B cells undergo significant genetic changes and extensively remodel the lymphoid microenvironment. Dynamic interactions between tumor B cells and the tumor microenvironment (TME) are hypothesized to contribute to the broad spectrum of clinical behaviors observed among FL patients. Despite the urgent need, existing clinical tools do not reliably predict disease behavior. Using a multi-modal strategy, we examined cell-intrinsic and -extrinsic factors governing progression and therapeutic outcomes in FL patients enrolled onto a prospective clinical trial. By leveraging the strengths of each platform, we identify several tumor-specific features and microenvironmental patterns enriched in individuals who experience early relapse, the most high-risk FL patients. These features include stromal desmoplasia and changes to the follicular growth pattern present 20 months before first progression and first relapse.

IBEX: an iterative immunolabeling and chemical bleaching method for high-content imaging of diverse tissues.

High-content imaging is needed to catalog the variety of cellular phenotypes and multicellular ecosystems present in metazoan tissues. We recently developed iterative bleaching extends multiplexity (IBEX), an iterative immunolabeling and chemical bleaching method that enables multiplexed imaging (>65 parameters) in diverse tissues, including human organs relevant for international consortia efforts. IBEX is compatible with >250 commercially available antibodies and 16 unique fluorophores, and can be easily adopted to different imaging platforms using slides and nonproprietary imaging chambers. The overall protocol consists of iterative cycles of antibody labeling, imaging and chemical bleaching that can be completed at relatively low cost in 2-5 d by biologists with basic laboratory skills. To support widespread adoption, we provide extensive details on tissue processing, curated lists of validated antibodies and tissue-specific panels for multiplex imaging. Furthermore, instructions are included on how to automate the method using competitively priced instruments and reagents. Finally, we present a software solution for image alignment that can be executed by individuals without programming experience using open-source software and freeware. In summary, IBEX is a noncommercial method that can be readily implemented by academic laboratories and scaled to achieve high-content mapping of diverse tissues in support of a Human Reference Atlas or other such applications.

IBEX: A versatile multiplex optical imaging approach for deep phenotyping and spatial analysis of cells in complex tissues.

The diverse composition of mammalian tissues poses challenges for understanding the cell-cell interactions required for organ homeostasis and how spatial relationships are perturbed during disease. Existing methods such as single-cell genomics, lacking a spatial context, and traditional immunofluorescence, capturing only two to six molecular features, cannot resolve these issues. Imaging technologies have been developed to address these problems, but each possesses limitations that constrain widespread use. Here we report a method that overcomes major impediments to highly multiplex tissue imaging. "Iterative bleaching extends multiplexity" (IBEX) uses an iterative staining and chemical bleaching method to enable high-resolution imaging of >65 parameters in the same tissue section without physical degradation. IBEX can be employed with various types of conventional microscopes and permits use of both commercially available and user-generated antibodies in an "open" system to allow easy adjustment of staining panels based on ongoing marker discovery efforts. We show how IBEX can also be used with amplified staining methods for imaging strongly fixed tissues with limited epitope retention and with oligonucleotide-based staining, allowing potential cross-referencing between flow cytometry, cellular indexing of transcriptomes and epitopes by sequencing, and IBEX analysis of the same tissue. To facilitate data processing, we provide an open-source platform for automated registration of iterative images. IBEX thus represents a technology that can be rapidly integrated into most current laboratory workflows to achieve high-content imaging to reveal the complex cellular landscape of diverse organs and tissues.

Correction to: SimpleITK Image-Analysis Notebooks: a Collaborative Environment for Education and Reproducible Research.

This paper had published originally without open access, but has since been republished with open access.

Image Segmentation, Registration and Characterization in R with SimpleITK.

Many types of medical and scientific experiments acquire raw data in the form of images. Various forms of image processing and image analysis are used to transform the raw image data into quantitative measures that are the basis of subsequent statistical analysis. In this article we describe the SimpleITK R package. SimpleITK is a simplified interface to the insight segmentation and registration toolkit (ITK). ITK is an open source C++ toolkit that has been actively developed over the past 18 years and is widely used by the medical image analysis community. SimpleITK provides packages for many interpreter environments, including R. Currently, it includes several hundred classes for image analysis including a wide range of image input and output, filtering operations, and higher level components for segmentation and registration. Using SimpleITK, development of complex combinations of image and statistical analysis procedures is feasible. This article includes several examples of computational image analysis tasks implemented using SimpleITK, including spherical marker localization, multi-modal image registration, segmentation evaluation, and cell image analysis.

SimpleITK Image-Analysis Notebooks: a Collaborative Environment for Education and Reproducible Research.

Modern scientific endeavors increasingly require team collaborations to construct and interpret complex computational workflows. This work describes an image-analysis environment that supports the use of computational tools that facilitate reproducible research and support scientists with varying levels of software development skills. The Jupyter notebook web application is the basis of an environment that enables flexible, well-documented, and reproducible workflows via literate programming. Image-analysis software development is made accessible to scientists with varying levels of programming experience via the use of the SimpleITK toolkit, a simplified interface to the Insight Segmentation and Registration Toolkit. Additional features of the development environment include user friendly data sharing using online data repositories and a testing framework that facilitates code maintenance. SimpleITK provides a large number of examples illustrating educational and research-oriented image analysis workflows for free download from GitHub under an Apache 2.0 license: github.com/InsightSoftwareConsortium/SimpleITK-Notebooks .

Multi-resolution correlative focused ion beam scanning electron microscopy: applications to cell biology.

Efficient correlative imaging of small targets within large fields is a central problem in cell biology. Here, we demonstrate a series of technical advances in focused ion beam scanning electron microscopy (FIB-SEM) to address this issue. We report increases in the speed, robustness and automation of the process, and achieve consistent z slice thickness of ∼3 nm. We introduce "keyframe imaging" as a new approach to simultaneously image large fields of view and obtain high-resolution 3D images of targeted sub-volumes. We demonstrate application of these advances to image post-fusion cytoplasmic intermediates of the HIV core. Using fluorescently labeled cell membranes, proteins and HIV cores, we first produce a "target map" of an HIV infected cell by fluorescence microscopy. We then generate a correlated 3D EM volume of the entire cell as well as high-resolution 3D images of individual HIV cores, achieving correlative imaging across a volume scale of 10(9) in a single automated experimental run.

The Design of SimpleITK.

SimpleITK is a new interface to the Insight Segmentation and Registration Toolkit (ITK) designed to facilitate rapid prototyping, education and scientific activities via high level programming languages. ITK is a templated C++ library of image processing algorithms and frameworks for biomedical and other applications, and it was designed to be generic, flexible and extensible. Initially, ITK provided a direct wrapping interface to languages such as Python and Tcl through the WrapITK system. Unlike WrapITK, which exposed ITK's complex templated interface, SimpleITK was designed to provide an easy to use and simplified interface to ITK's algorithms. It includes procedural methods, hides ITK's demand driven pipeline, and provides a template-less layer. Also SimpleITK provides practical conveniences such as binary distribution packages and overloaded operators. Our user-friendly design goals dictated a departure from the direct interface wrapping approach of WrapITK, toward a new facade class structure that only exposes the required functionality, hiding ITK's extensive template use. Internally SimpleITK utilizes a manual description of each filter with code-generation and advanced C++ meta-programming to provide the higher-level interface, bringing the capabilities of ITK to a wider audience. SimpleITK is licensed as open source software library under the Apache License Version 2.0 and more information about downloading it can be found at http://www.simpleitk.org.

Visualizing cells and humans in 3D: biomedical image analysis at nanometer and meter scales.

Researchers analyzed and presented volume data from the Visible Human Project (VHP) and data from high-resolution 3D ion-abrasion scanning electron microscopy (IA-SEM). They acquired the VHP data using cryosectioning, a destructive approach to 3D human anatomical imaging resulting in whole-body images with a field of view approaching 2 meters and a minimum resolvable feature size of 300 microns. IA-SEM is a type of block-face imaging microscopy, a destructive approach to microscopic 3D imaging of cells. The field of view of IA-SEM data is on the order of 10 microns (whole cell) with a minimum resolvable feature size of 15 nanometers (single-slice thickness). Despite the difference in subject and scale, the analysis and modeling methods were remarkably similar. They are derived from image processing, computer vision, and computer graphics techniques. Moreover, together we are employing medical illustration, visualization, and rapid prototyping to inform and inspire biomedical science. By combining graphics and biology, we are imaging across nine orders of magnitude of space to better promote public health through research.

Correlative 3D imaging of whole mammalian cells with light and electron microscopy.

We report methodological advances that extend the current capabilities of ion-abrasion scanning electron microscopy (IA-SEM), also known as focused ion beam scanning electron microscopy, a newly emerging technology for high resolution imaging of large biological specimens in 3D. We establish protocols that enable the routine generation of 3D image stacks of entire plastic-embedded mammalian cells by IA-SEM at resolutions of ∼10-20nm at high contrast and with minimal artifacts from the focused ion beam. We build on these advances by describing a detailed approach for carrying out correlative live confocal microscopy and IA-SEM on the same cells. Finally, we demonstrate that by combining correlative imaging with newly developed tools for automated image processing, small 100nm-sized entities such as HIV-1 or gold beads can be localized in SEM image stacks of whole mammalian cells. We anticipate that these methods will add to the arsenal of tools available for investigating mechanisms underlying host-pathogen interactions, and more generally, the 3D subcellular architecture of mammalian cells and tissues.

3D visualization of HIV transfer at the virological synapse between dendritic cells and T cells.

The efficiency of HIV infection is greatly enhanced when the virus is delivered at conjugates between CD4+ T cells and virus-bearing antigen-presenting cells such as macrophages or dendritic cells via specialized structures known as virological synapses. Using ion abrasion SEM, electron tomography, and superresolution light microscopy, we have analyzed the spatial architecture of cell-cell contacts and distribution of HIV virions at virological synapses formed between mature dendritic cells and T cells. We demonstrate the striking envelopment of T cells by sheet-like membrane extensions derived from mature dendritic cells, resulting in a shielded region for formation of virological synapses. Within the synapse, filopodial extensions emanating from CD4+ T cells make contact with HIV virions sequestered deep within a 3D network of surface-accessible compartments in the dendritic cell. Viruses are detected at the membrane surfaces of both dendritic cells and T cells, but virions are not released passively at the synapse; instead, virus transfer requires the engagement of T-cell CD4 receptors. The relative seclusion of T cells from the extracellular milieu, the burial of the site of HIV transfer, and the receptor-dependent initiation of virion transfer by T cells highlight unique aspects of cell-cell HIV transmission.

Ion-abrasion scanning electron microscopy reveals distorted liver mitochondrial morphology in murine methylmalonic acidemia.

Methylmalonic acidemia is a lethal inborn error of metabolism that causes mitochondrial impairment, multi-organ dysfunction and a shortened lifespan. Previous transmission electron microscope studies of thin sections from normal (Mut(+/+)) and diseased (Mut(-/-)) tissue found that the mitochondria appear to occupy a progressively larger volume of mutant cells with age, becoming megamitochondria. To assess changes in shape and volume of mitochondria resulting from the mutation, we carried out ion-abrasion scanning electron microscopy (IA-SEM), a method for 3D imaging that involves the iterative use of a focused gallium ion beam to abrade the surface of the specimen, and a scanning electron beam to image the newly exposed surface. Using IA-SEM, we show that mitochondria are more convoluted and have a broader distribution of sizes in the mutant tissue. Compared to normal cells, mitochondria from mutant cells have a larger surface-area-to-volume ratio, which can be attributed to their convoluted shape and not to their elongation or reduced volume. The 3D imaging approach and image analysis described here could therefore be useful as a diagnostic tool for the evaluation of disease progression in aberrant cells at resolutions higher than that currently achieved using confocal light microscopy.

Ion-abrasion scanning electron microscopy reveals surface-connected tubular conduits in HIV-infected macrophages.

HIV-1-containing internal compartments are readily detected in images of thin sections from infected cells using conventional transmission electron microscopy, but the origin, connectivity, and 3D distribution of these compartments has remained controversial. Here, we report the 3D distribution of viruses in HIV-1-infected primary human macrophages using cryo-electron tomography and ion-abrasion scanning electron microscopy (IA-SEM), a recently developed approach for nanoscale 3D imaging of whole cells. Using IA-SEM, we show the presence of an extensive network of HIV-1-containing tubular compartments in infected macrophages, with diameters of approximately 150-200 nm, and lengths of up to approximately 5 microm that extend to the cell surface from vesicular compartments that contain assembling HIV-1 virions. These types of surface-connected tubular compartments are not observed in T cells infected with the 29/31 KE Gag-matrix mutant where the virus is targeted to multi-vesicular bodies and released into the extracellular medium. IA-SEM imaging also allows visualization of large sheet-like structures that extend outward from the surfaces of macrophages, which may bend and fold back to allow continual creation of viral compartments and virion-lined channels. This potential mechanism for efficient virus trafficking between the cell surface and interior may represent a subversion of pre-existing vesicular machinery for antigen capture, processing, sequestration, and presentation.