Denoising diffusion probabilistic models for generation of realistic fully-annotated microscopy image datasets.

Recent advances in computer vision have led to significant progress in the generation of realistic image data, with denoising diffusion probabilistic models proving to be a particularly effective method. In this study, we demonstrate that diffusion models can effectively generate fully-annotated microscopy image data sets through an unsupervised and intuitive approach, using rough sketches of desired structures as the starting point. The proposed pipeline helps to reduce the reliance on manual annotations when training deep learning-based segmentation approaches and enables the segmentation of diverse datasets without the need for human annotations. We demonstrate that segmentation models trained with a small set of synthetic image data reach accuracy levels comparable to those of generalist models trained with a large and diverse collection of manually annotated image data, thereby offering a streamlined and specialized application of segmentation models.

Surface-Patterned DNA Origami Rulers Reveal Nanoscale Distance Dependency of the Epidermal Growth Factor Receptor Activation.

The nanoscale arrangement of ligands can have a major effect on the activation of membrane receptor proteins and thus cellular communication mechanisms. Here we report on the technological development and use of tailored DNA origami-based molecular rulers to fabricate "Multiscale Origami Structures As Interface for Cells" (MOSAIC), to enable the systematic investigation of the effect of the nanoscale spacing of epidermal growth factor (EGF) ligands on the activation of the EGF receptor (EGFR). MOSAIC-based analyses revealed that EGF distances of about 30-40 nm led to the highest response in EGFR activation of adherent MCF7 and Hela cells. Our study emphasizes the significance of DNA-based platforms for the detailed investigation of the molecular mechanisms of cellular signaling cascades.

Multimodal Deep Learning for Integrating Chest Radiographs and Clinical Parameters: A Case for Transformers.

Background Clinicians consider both imaging and nonimaging data when diagnosing diseases; however, current machine learning approaches primarily consider data from a single modality. Purpose To develop a neural network architecture capable of integrating multimodal patient data and compare its performance to models incorporating a single modality for diagnosing up to 25 pathologic conditions. Materials and Methods In this retrospective study, imaging and nonimaging patient data were extracted from the Medical Information Mart for Intensive Care (MIMIC) database and an internal database comprised of chest radiographs and clinical parameters inpatients in the intensive care unit (ICU) (January 2008 to December 2020). The MIMIC and internal data sets were each split into training (n = 33 893, n = 28 809), validation (n = 740, n = 7203), and test (n = 1909, n = 9004) sets. A novel transformer-based neural network architecture was trained to diagnose up to 25 conditions using nonimaging data alone, imaging data alone, or multimodal data. Diagnostic performance was assessed using area under the receiver operating characteristic curve (AUC) analysis. Results The MIMIC and internal data sets included 36 542 patients (mean age, 63 years ± 17 [SD]; 20 567 male patients) and 45 016 patients (mean age, 66 years ± 16; 27 577 male patients), respectively. The multimodal model showed improved diagnostic performance for all pathologic conditions. For the MIMIC data set, the mean AUC was 0.77 (95% CI: 0.77, 0.78) when both chest radiographs and clinical parameters were used, compared with 0.70 (95% CI: 0.69, 0.71; P < .001) for only chest radiographs and 0.72 (95% CI: 0.72, 0.73; P < .001) for only clinical parameters. These findings were confirmed on the internal data set. Conclusion A model trained on imaging and nonimaging data outperformed models trained on only one type of data for diagnosing multiple diseases in patients in an ICU setting. © RSNA, 2023 Supplemental material is available for this article. See also the editorial by Kitamura and Topol in this issue.

Artificial Intelligence for Clinical Interpretation of Bedside Chest Radiographs.

Background Supine chest radiography for bedridden patients in intensive care units (ICUs) is one of the most frequently ordered imaging studies worldwide. Purpose To evaluate the diagnostic performance of a neural network-based model that is trained on structured semiquantitative radiologic reports of bedside chest radiographs. Materials and Methods For this retrospective single-center study, children and adults in the ICU of a university hospital who had been imaged using bedside chest radiography from January 2009 to December 2020 were reported by using a structured and itemized template. Ninety-eight radiologists rated the radiographs semiquantitatively for the severity of disease patterns. These data were used to train a neural network to identify cardiomegaly, pulmonary congestion, pleural effusion, pulmonary opacities, and atelectasis. A held-out internal test set (100 radiographs from 100 patients) that was assessed independently by an expert panel of six radiologists provided the ground truth. Individual assessments by each of these six radiologists, by two nonradiologist physicians in the ICU, and by the neural network were compared with the ground truth. Separately, the nonradiologist physicians assessed the images without and with preliminary readings provided by the neural network. The weighted Cohen κ coefficient was used to measure agreement between the readers and the ground truth. Results A total of 193 566 radiographs in 45 016 patients (mean age, 66 years ± 16 [SD]; 61% men) were included and divided into training (n = 122 294; 64%), validation (n = 31 243; 16%), and test (n = 40 029; 20%) sets. The neural network exhibited higher agreement with a majority vote of the expert panel (κ = 0.86) than each individual radiologist compared with the majority vote of the expert panel (κ = 0.81 to ≤0.84). When the neural network provided preliminary readings, the reports of the nonradiologist physicians improved considerably (aided vs unaided, κ = 0.87 vs 0.79, respectively; P < .001). Conclusion A neural network trained with structured semiquantitative bedside chest radiography reports allowed nonradiologist physicians improved interpretations compared with the consensus reading of expert radiologists. © RSNA, 2022 Supplemental material is available for this article. See also the editorial by Wielpütz in this issue.

Pax6 organizes the anterior eye segment by guiding two distinct neural crest waves.

Cranial neural crest (NC) contributes to the developing vertebrate eye. By multidimensional, quantitative imaging, we traced the origin of the ocular NC cells to two distinct NC populations that differ in the maintenance of sox10 expression, Wnt signalling, origin, route, mode and destination of migration. The first NC population migrates to the proximal and the second NC cell group populates the distal (anterior) part of the eye. By analysing zebrafish pax6a/b compound mutants presenting anterior segment dysgenesis, we demonstrate that Pax6a/b guide the two NC populations to distinct proximodistal locations. We further provide evidence that the lens whose formation is pax6a/b-dependent and lens-derived TGFß signals contribute to the building of the anterior segment. Taken together, our results reveal multiple roles of Pax6a/b in the control of NC cells during development of the anterior segment.

Definition and Quantification of Three-Dimensional Imaging Targets to Phenotype Pre-Eclampsia Subtypes: An Exploratory Study.

Pre-eclampsia is a severe placenta-related complication of pregnancy with limited early diagnostic and therapeutic options. Aetiological knowledge is controversial, and there is no universal consensus on what constitutes the early and late phenotypes of pre-eclampsia. Phenotyping of native placental three-dimensional (3D) morphology offers a novel approach to improve our understanding of the structural placental abnormalities in pre-eclampsia. Healthy and pre-eclamptic placental tissues were imaged with multiphoton microscopy (MPM). Imaging based on inherent signal (collagen, and cytoplasm) and fluorescent staining (nuclei, and blood vessels) enabled the visualization of placental villous tissue with subcellular resolution. Images were analysed with a combination of open source (FIJI, VMTK, Stardist, MATLAB, DBSCAN), and commercially (MATLAB) available software. Trophoblast organization, 3D-villous tree structure, syncytial knots, fibrosis, and 3D-vascular networks were identified as quantifiable imaging targets. Preliminary data indicate increased syncytial knot density with characteristic elongated shape, higher occurrence of paddle-like villous sprouts, abnormal villous volume-to-surface ratio, and decreased vascular density in pre-eclampsia compared to control placentas. The preliminary data presented indicate the potential of quantifying 3D microscopic images for identifying different morphological features and phenotyping pre-eclampsia in placental villous tissue.

The HMG box transcription factors Sox1a and Sox1b specify a new class of glycinergic interneuron in the spinal cord of zebrafish embryos.

Specification of neurons in the spinal cord relies on extrinsic and intrinsic signals, which in turn are interpreted by expression of transcription factors. V2 interneurons develop from the ventral aspects of the spinal cord. We report here a novel neuronal V2 subtype, named V2s, in zebrafish embryos. Formation of these neurons depends on the transcription factors sox1a and sox1b. They develop from common gata2a- and gata3-dependent precursors co-expressing markers of V2b and V2s interneurons. Chemical blockage of Notch signalling causes a decrease in V2s and an increase in V2b cells. Our results are consistent with the existence of at least two types of precursor arranged in a hierarchical manner in the V2 domain. V2s neurons grow long ipsilateral descending axonal projections with a short branch at the ventral midline. They acquire a glycinergic neurotransmitter type during the second day of development. Unilateral ablation of V2s interneurons causes a delay in touch-provoked escape behaviour, suggesting that V2s interneurons are involved in fast motor responses.

Third-generation in situ hybridization chain reaction: multiplexed, quantitative, sensitive, versatile, robust.

In situ hybridization based on the mechanism of the hybridization chain reaction (HCR) has addressed multi-decade challenges that impeded imaging of mRNA expression in diverse organisms, offering a unique combination of multiplexing, quantitation, sensitivity, resolution and versatility. Here, with third-generation in situ HCR, we augment these capabilities using probes and amplifiers that combine to provide automatic background suppression throughout the protocol, ensuring that reagents will not generate amplified background even if they bind non-specifically within the sample. Automatic background suppression dramatically enhances performance and robustness, combining the benefits of a higher signal-to-background ratio with the convenience of using unoptimized probe sets for new targets and organisms. In situ HCR v3.0 enables three multiplexed quantitative analysis modes: (1) qHCR imaging - analog mRNA relative quantitation with subcellular resolution in the anatomical context of whole-mount vertebrate embryos; (2) qHCR flow cytometry - analog mRNA relative quantitation for high-throughput expression profiling of mammalian and bacterial cells; and (3) dHCR imaging - digital mRNA absolute quantitation via single-molecule imaging in thick autofluorescent samples.

BeadNet: deep learning-based bead detection and counting in low-resolution microscopy images.

MOTIVATION: An automated counting of beads is required for many high-throughput experiments such as studying mimicked bacterial invasion processes. However, state-of-the-art algorithms under- or overestimate the number of beads in low-resolution images. In addition, expert knowledge is needed to adjust parameters. RESULTS: In combination with our image labeling tool, BeadNet enables biologists to easily annotate and process their data reducing the expertise required in many existing image analysis pipelines. BeadNet outperforms state-of-the-art-algorithms in terms of missing, added and total amount of beads. AVAILABILITY AND IMPLEMENTATION: BeadNet (software, code and dataset) is available at https://bitbucket.org/t_scherr/beadnet. The image labeling tool is available at https://bitbucket.org/abartschat/imagelabelingtool. SUPPLEMENTARY INFORMATION: Supplementary data are available at Bioinformatics online.

An objective comparison of cell-tracking algorithms.

We present a combined report on the results of three editions of the Cell Tracking Challenge, an ongoing initiative aimed at promoting the development and objective evaluation of cell segmentation and tracking algorithms. With 21 participating algorithms and a data repository consisting of 13 data sets from various microscopy modalities, the challenge displays today's state-of-the-art methodology in the field. We analyzed the challenge results using performance measures for segmentation and tracking that rank all participating methods. We also analyzed the performance of all of the algorithms in terms of biological measures and practical usability. Although some methods scored high in all technical aspects, none obtained fully correct solutions. We found that methods that either take prior information into account using learning strategies or analyze cells in a global spatiotemporal video context performed better than other methods under the segmentation and tracking scenarios included in the challenge.

Microfluidic Chips for Life Sciences-A Comparison of Low Entry Manufacturing Technologies.

Microfluidic water-in-oil droplets are a versatile tool for biological and biochemical applications due to the advantages of extremely small monodisperse reaction vessels in the pL-nL range. A key factor for the successful dissemination of this technology to life science laboratory users is the ability to produce microfluidic droplet generators and related accessories by low-entry barrier methods, which enable rapid prototyping and manufacturing of devices with low instrument and material costs. The direct, experimental side-by-side comparison of three commonly used additive manufacturing (AM) methods, namely fused deposition modeling (FDM), inkjet printing (InkJ), and stereolithography (SLA), is reported. As a benchmark, micromilling (MM) is used as an established method. To demonstrate which of these methods can be easily applied by the non-expert to realize applications in topical fields of biochemistry and microbiology, the methods are evaluated with regard to their limits for the minimum structure resolution in all three spatial directions. The suitability of functional SLA and MM chips to replace classic SU-8 prototypes is demonstrated on the basis of representative application cases.

EmbryoMiner: A new framework for interactive knowledge discovery in large-scale cell tracking data of developing embryos.

State-of-the-art light-sheet and confocal microscopes allow recording of entire embryos in 3D and over time (3D+t) for many hours. Fluorescently labeled structures can be segmented and tracked automatically in these terabyte-scale 3D+t images, resulting in thousands of cell migration trajectories that provide detailed insights to large-scale tissue reorganization at the cellular level. Here we present EmbryoMiner, a new interactive open-source framework suitable for in-depth analyses and comparisons of entire embryos, including an extensive set of trajectory features. Starting at the whole-embryo level, the framework can be used to iteratively focus on a region of interest within the embryo, to investigate and test specific trajectory-based hypotheses and to extract quantitative features from the isolated trajectories. Thus, the new framework provides a valuable new way to quantitatively compare corresponding anatomical regions in different embryos that were manually selected based on biological prior knowledge. As a proof of concept, we analyzed 3D+t light-sheet microscopy images of zebrafish embryos, showcasing potential user applications that can be performed using the new framework.

XPIWIT--an XML pipeline wrapper for the Insight Toolkit.

UNLABELLED: The Insight Toolkit offers plenty of features for multidimensional image analysis. Current implementations, however, often suffer either from a lack of flexibility due to hard-coded C++ pipelines for a certain task or by slow execution times, e.g. caused by inefficient implementations or multiple read/write operations for separate filter execution. We present an XML-based wrapper application for the Insight Toolkit that combines the performance of a pure C++ implementation with an easy-to-use graphical setup of dynamic image analysis pipelines. Created XML pipelines can be interpreted and executed by XPIWIT in console mode either locally or on large clusters. We successfully applied the software tool for the automated analysis of terabyte-scale, time-resolved 3D image data of zebrafish embryos. AVAILABILITY AND IMPLEMENTATION: XPIWIT is implemented in C++ using the Insight Toolkit and the Qt SDK. It has been successfully compiled and tested under Windows and Unix-based systems. Software and documentation are distributed under Apache 2.0 license and are publicly available for download at https://bitbucket.org/jstegmaier/xpiwit/downloads/. CONTACT: johannes.stegmaier@kit.edu SUPPLEMENTARY INFORMATION: Supplementary data are available at Bioinformatics online.

Automated prior knowledge-based quantification of neuronal patterns in the spinal cord of zebrafish.

MOTIVATION: To reliably assess the effects of unknown chemicals on the development of fluorescently labeled sensory-, moto- and interneuron populations in the spinal cord of zebrafish, automated data analysis is essential. RESULTS: For the evaluation of a high-throughput screen of a large chemical library, we developed a new method for the automated extraction of quantitative information from green fluorescent protein (eGFP) and red fluorescent protein (RFP) labeled spinal cord neurons in double-transgenic zebrafish embryos. The methodology comprises region of interest detection, intensity profiling with reference comparison and neuron distribution histograms. All methods were validated on a manually evaluated pilot study using a Notch inhibitor dose-response experiment. The automated evaluation showed superior performance to manual investigation regarding time consumption, information detail and reproducibility. AVAILABILITY AND IMPLEMENTATION: Being part of GNU General Public Licence (GNU-GPL) licensed open-source MATLAB toolbox Gait-CAD, an implementation of the presented methods is publicly available for download at http://sourceforge.net/projects/zebrafishimage/.

Automatic detection of cell-cycle stages using recurrent neural networks.

Mitosis is the process by which eukaryotic cells divide to produce two similar daughter cells with identical genetic material. Research into the process of mitosis is therefore of critical importance both for the basic understanding of cell biology and for the clinical approach to manifold pathologies resulting from its malfunctioning, including cancer. In this paper, we propose an approach to study mitotic progression automatically using deep learning. We used neural networks to predict different mitosis stages. We extracted video sequences of cells undergoing division and trained a Recurrent Neural Network (RNN) to extract image features. The use of RNN enabled better extraction of features. The RNN-based approach gave better performance compared to classifier based feature extraction methods which do not use time information. Evaluation of precision, recall, and F-score indicates the superiority of the proposed model compared to the baseline. To study the loss in performance due to confusion between adjacent classes, we plotted the confusion matrix as well. In addition, we visualized the feature space to understand why RNNs are better at classifying the mitosis stages than other classifier models, which indicated the formation of strong clusters for the different classes, clearly confirming the advantage of the proposed RNN-based approach.

A Deep Learning-based in silico Framework for Optimization on Retinal Prosthetic Stimulation.

We propose a neural network-based framework to optimize the perceptions simulated by the in silico retinal implant model pulse2percept. The overall pipeline consists of a trainable encoder, a pre-trained retinal implant model and a pre-trained evaluator. The encoder is a U-Net, which takes the original image and outputs the stimulus. The pre-trained retinal implant model is also a U-Net, which is trained to mimic the biomimetic perceptual model implemented in pulse2percept. The evaluator is a shallow VGG classifier, which is trained with original images. Based on 10,000 test images from the MNIST dataset, we show that the convolutional neural network-based encoder performs significantly better than the trivial downsampling approach, yielding a boost in the weighted F1-Score by 36.17% in the pre-trained classifier with 6×10 electrodes. With this fully neural network-based encoder, the quality of the downstream perceptions can be fine-tuned using gradient descent in an end-to-end fashion.

A Deep-Learning-Computed Cancer Score for the Identification of Human Hepatocellular Carcinoma Area Based on a Six-Colour Multiplex Immunofluorescence Panel.

Liver cancer is one of the most frequently diagnosed and fatal cancers worldwide, with hepatocellular carcinoma (HCC) being the most common primary liver cancer. Hundreds of studies involving thousands of patients have now been analysed across different cancer types, including HCC, regarding the effects of immune infiltrates on the prognosis of cancer patients. However, for these analyses, an unambiguous delineation of the cancer area is paramount, which is difficult due to the strong heterogeneity and considerable inter-operator variability induced by qualitative visual assessment and manual assignment. Nowadays, however, multiplex analyses allow the simultaneous evaluation of multiple protein markers, which, in conjunction with recent machine learning approaches, may offer great potential for the objective, enhanced identification of cancer areas with further in situ analysis of prognostic immune parameters. In this study, we, therefore, used an exemplary five-marker multiplex immunofluorescence panel of commonly studied markers for prognosis (CD3 T, CD4 T helper, CD8 cytotoxic T, FoxP3 regulatory T, and PD-L1) and DAPI to assess which analytical approach is best suited to combine morphological and immunohistochemical data into a cancer score to identify the cancer area that best matches an independent pathologist's assignment. For each cell, a total of 68 individual cell features were determined, which were used as input for 4 different approaches for computing a cancer score: a correlation-based selection of individual cell features, a MANOVA-based selection of features, a multilayer perceptron, and a convolutional neural network (a U-net). Accuracy was used to evaluate performance. With a mean accuracy of 75%, the U-net was best capable of identifying the cancer area. Although individual cell features showed a strong heterogeneity between patients, the spatial representations obtained with the computed cancer scores delineate HCC well from non-cancer liver tissues. Future analyses with larger sample sizes will help to improve the model and enable direct, in-depth investigations of prognostic parameters, ultimately enabling precision medicine.

Medical transformer for multimodal survival prediction in intensive care: integration of imaging and non-imaging data.

When clinicians assess the prognosis of patients in intensive care, they take imaging and non-imaging data into account. In contrast, many traditional machine learning models rely on only one of these modalities, limiting their potential in medical applications. This work proposes and evaluates a transformer-based neural network as a novel AI architecture that integrates multimodal patient data, i.e., imaging data (chest radiographs) and non-imaging data (clinical data). We evaluate the performance of our model in a retrospective study with 6,125 patients in intensive care. We show that the combined model (area under the receiver operating characteristic curve [AUROC] of 0.863) is superior to the radiographs-only model (AUROC = 0.811, p < 0.001) and the clinical data-only model (AUROC = 0.785, p < 0.001) when tasked with predicting in-hospital survival per patient. Furthermore, we demonstrate that our proposed model is robust in cases where not all (clinical) data points are available.

Denoising diffusion probabilistic models for 3D medical image generation.

Recent advances in computer vision have shown promising results in image generation. Diffusion probabilistic models have generated realistic images from textual input, as demonstrated by DALL-E 2, Imagen, and Stable Diffusion. However, their use in medicine, where imaging data typically comprises three-dimensional volumes, has not been systematically evaluated. Synthetic images may play a crucial role in privacy-preserving artificial intelligence and can also be used to augment small datasets. We show that diffusion probabilistic models can synthesize high-quality medical data for magnetic resonance imaging (MRI) and computed tomography (CT). For quantitative evaluation, two radiologists rated the quality of the synthesized images regarding "realistic image appearance", "anatomical correctness", and "consistency between slices". Furthermore, we demonstrate that synthetic images can be used in self-supervised pre-training and improve the performance of breast segmentation models when data is scarce (Dice scores, 0.91 [without synthetic data], 0.95 [with synthetic data]).

LiveCellMiner: A new tool to analyze mitotic progression.

Live-cell imaging has become state of the art to accurately identify the nature of mitotic and cell cycle defects. Low- and high-throughput microscopy setups have yield huge data amounts of cells recorded in different experimental and pathological conditions. Tailored semi-automated and automated image analysis approaches allow the analysis of high-content screening data sets, saving time and avoiding bias. However, they were mostly designed for very specific experimental setups, which restricts their flexibility and usability. The general need for dedicated experiment-specific user-annotated training sets and experiment-specific user-defined segmentation parameters remains a major bottleneck for fully automating the analysis process. In this work we present LiveCellMiner, a highly flexible open-source software tool to automatically extract, analyze and visualize both aggregated and time-resolved image features with potential biological relevance. The software tool allows analysis across high-content data sets obtained in different platforms, in a quantitative and unbiased manner. As proof of principle application, we analyze here the dynamic chromatin and tubulin cytoskeleton features in human cells passing through mitosis highlighting the versatile and flexible potential of this tool set.