Deep learning-enhanced snapshot hyperspectral confocal microscopy imaging system.

Laser-scanning confocal hyperspectral microscopy is a powerful technique to identify the different sample constituents and their spatial distribution in three-dimensional (3D). However, it suffers from low imaging speed because of the mechanical scanning methods. To overcome this challenge, we propose a snapshot hyperspectral confocal microscopy imaging system (SHCMS). It combined coded illumination microscopy based on a digital micromirror device (DMD) with a snapshot hyperspectral confocal neural network (SHCNet) to realize single-shot confocal hyperspectral imaging. With SHCMS, high-contrast 160-bands confocal hyperspectral images of potato tuber autofluorescence can be collected by only single-shot, which is almost 5 times improvement in the number of spectral channels than previously reported methods. Moreover, our approach can efficiently record hyperspectral volumetric imaging due to the optical sectioning capability. This fast high-resolution hyperspectral imaging method may pave the way for real-time highly multiplexed biological imaging.

Domain generalization across tumor types, laboratories, and species - Insights from the 2022 edition of the Mitosis Domain Generalization Challenge.

Recognition of mitotic figures in histologic tumor specimens is highly relevant to patient outcome assessment. This task is challenging for algorithms and human experts alike, with deterioration of algorithmic performance under shifts in image representations. Considerable covariate shifts occur when assessment is performed on different tumor types, images are acquired using different digitization devices, or specimens are produced in different laboratories. This observation motivated the inception of the 2022 challenge on MItosis Domain Generalization (MIDOG 2022). The challenge provided annotated histologic tumor images from six different domains and evaluated the algorithmic approaches for mitotic figure detection provided by nine challenge participants on ten independent domains. Ground truth for mitotic figure detection was established in two ways: a three-expert majority vote and an independent, immunohistochemistry-assisted set of labels. This work represents an overview of the challenge tasks, the algorithmic strategies employed by the participants, and potential factors contributing to their success. With an F1 score of 0.764 for the top-performing team, we summarize that domain generalization across various tumor domains is possible with today's deep learning-based recognition pipelines. However, we also found that domain characteristics not present in the training set (feline as new species, spindle cell shape as new morphology and a new scanner) led to small but significant decreases in performance. When assessed against the immunohistochemistry-assisted reference standard, all methods resulted in reduced recall scores, with only minor changes in the order of participants in the ranking.

Complete genome assembly provides a high-quality skeleton for pan-NLRome construction in melon.

Melon (Cucumis melo L.), being under intensive domestication and selective breeding, displays an abundant phenotypic diversity. Wild germplasm with tolerance to stress represents an untapped genetic resource for discovery of disease-resistance genes. To comprehensively characterize resistance genes in melon, we generate a telomere-to-telomere (T2T) and gap-free genome of wild melon accession PI511890 (C. melo var. chito) with a total length of 375.0 Mb and a contig N50 of 31.24 Mb. The complete genome allows us to dissect genome architecture and identify resistance gene analogs. We construct a pan-NLRome using seven melon genomes, which include 208 variable and 18 core nucleotide-binding leucine-rich repeat receptors (NLRs). Multiple disease-related transcriptome analyses indicate that most up-regulated NLRs induced by pathogens are shell or cloud NLRs. The T2T gap-free assembly and the pan-NLRome not only serve as essential resources for genomic studies and molecular breeding of melon but also provide insights into the genome architecture and NLR diversity.

A two-stage fine-tuning method for low-resource cross-lingual summarization.

Cross-lingual summarization (CLS) is the task of condensing lengthy source language text into a concise summary in a target language. This presents a dual challenge, demanding both cross-language semantic understanding (i.e., semantic alignment) and effective information compression capabilities. Traditionally, researchers have tackled these challenges using two types of methods: pipeline methods (e.g., translate-then-summarize) and end-to-end methods. The former is intuitive but prone to error propagation, particularly for low-resource languages. The later has shown an impressive performance, due to multilingual pre-trained models (mPTMs). However, mPTMs (e.g., mBART) are primarily trained on resource-rich languages, thereby limiting their semantic alignment capabilities for low-resource languages. To address these issues, this paper integrates the intuitiveness of pipeline methods and the effectiveness of mPTMs, and then proposes a two-stage fine-tuning method for low-resource cross-lingual summarization (TFLCLS). In the first stage, by recognizing the deficiency in the semantic alignment for low-resource languages in mPTMs, a semantic alignment fine-tuning method is employed to enhance the mPTMs' understanding of such languages. In the second stage, while considering that mPTMs are not originally tailored for information compression and CLS demands the model to simultaneously align and compress, an adaptive joint fine-tuning method is introduced. This method further enhances the semantic alignment and information compression abilities of mPTMs that were trained in the first stage. To evaluate the performance of TFLCLS, a low-resource CLS dataset, named Vi2ZhLow, is constructed from scratch; moreover, two additional low-resource CLS datasets, En2ZhLow and Zh2EnLow, are synthesized from widely used large-scale CLS datasets. Experimental results show that TFCLS outperforms state-of-the-art methods by 18.88%, 12.71% and 16.91% in ROUGE-2 on the three datasets, respectively, even when limited with only 5,000 training samples.

Integrated full-length transcriptome and metabolome analysis reveals the defence response of melon to gummy stem blight.

Gummy stem blight (GSB), a widespread disease causing great loss to cucurbit production, has become a major threat to melon cultivation. However, the melon-GSB interaction remains largely unknown. Here, full-length transcriptome and widely targeted metabolome were used to investigate the defence responses of resistant (PI511089) and susceptible (Payzawat) melon accessions to GSB pathogen infection at 24 h. The biosynthesis of secondary metabolites and MAPK signalling pathway were specifically enriched for differentially expressed genes in PI511890, while carbohydrate metabolism and amino acid metabolism were specifically enriched in Payzawat. More than 1000 novel genes were identified and MAPK signalling pathway was specifically enriched for them in PI511890. There were 11 793 alternative splicing events involving in the defence response to GSB. Totally, 910 metabolites were identified in Payzawat and PI511890, and flavonoids were the dominant metabolites. Integrated full-length transcriptome and metabolome analysis showed eriodictyol and oxalic acid were the potential marker metabolites for GSB resistance in melon. Moreover, posttranscription regulation was widely involved in the defence response of melon to GSB pathogen infection. These results not only improve our understanding on the interaction between melon and GSB, but also facilitate the genetic improvement of melon with GSB resistance.

Unsupervised Multi-Domain Progressive Stain Transfer Guided by Style Encoding Dictionary.

In histopathology, the tissue slides are usually stained by common H&E stain or special stains (MAS, PAS, and PASM, etc.) to clearly show specific tissue structures. The rapid development of deep learning provides a good solution to generate virtual staining images to significantly reduce the time and labor costs associated with histochemical staining. However, most existing methods need to train a special model for every two stains, which consumes a lot of computing resources with the increasing of staining types. To address this problem, we propose an unsupervised multi-domain stain transfer method, GramGAN, which realizes the progressive transfer through cascaded Style-Guided blocks. For each Style-Guided block, we design a style encoding dictionary to characterize and store all the staining style information. In addition, we propose a Rényi entropy-based regularization term to improve the discrimination ability of different styles. The experimental results show that our method can realize accurate transferring among multiple staining styles with better performance. Furthermore, we build and publish a special stained image dataset suitable for glomeruli segmentation (including H&E staining), where the accuracy of glomeruli detection and segmentation can be significantly improved after transferring H&E-stained images to PAS-stained and PASM-stained ones by our method. The code is publicly available at: https://github.com/xianchaoguan/GramGAN.

AutoUnmix: an autoencoder-based spectral unmixing method for multi-color fluorescence microscopy imaging.

Multiplexed fluorescence microscopy imaging is widely used in biomedical applications. However, simultaneous imaging of multiple fluorophores can result in spectral leaks and overlapping, which greatly degrades image quality and subsequent analysis. Existing popular spectral unmixing methods are mainly based on computational intensive linear models, and the performance is heavily dependent on the reference spectra, which may greatly preclude its further applications. In this paper, we propose a deep learning-based blindly spectral unmixing method, termed AutoUnmix, to imitate the physical spectral mixing process. A transfer learning framework is further devised to allow our AutoUnmix to adapt to a variety of imaging systems without retraining the network. Our proposed method has demonstrated real-time unmixing capabilities, surpassing existing methods by up to 100-fold in terms of unmixing speed. We further validate the reconstruction performance on both synthetic datasets and biological samples. The unmixing results of AutoUnmix achieve the highest SSIM of 0.99 in both three- and four-color imaging, with nearly up to 20% higher than other popular unmixing methods. For experiments where spectral profiles and morphology are akin to simulated data, our method realizes the quantitative performance demonstrated above. Due to the desirable property of data independency and superior blind unmixing performance, we believe AutoUnmix is a powerful tool for studying the interaction process of different organelles labeled by multiple fluorophores.

Generalizability of Self-Supervised Training Models for Digital Pathology: A Multicountry Comparison in Colorectal Cancer.

PURPOSE: In this multicountry study, we aim to explore the effectiveness of self-supervised learning (SSL) in colorectal cancer (CRC)-related predictive tasks using large amount of unlabeled digital pathology imaging data. METHODS: We adopted SimSiam to conduct self-supervised pretraining on two large whole-slide image CRC data sets from the United States and Australia. The SSL pretrained encoder is then used in several predictive tasks, including supervised predictive tasks (tissue classification, microsatellite instability v microsatellite stability classification), and weakly supervised predictive tasks (polyp type classification and adenoma grading, and 5-year survival prediction). Performance on the tasks was compared between models using SSL pretraining and those using ImageNet pretraining, and performance for one-country pretraining was compared with two-country pretraining. RESULTS: We demonstrate that SSL pretraining outperforms ImageNet pretraining in predictive tasks, that is, SSL pretraining outperforms the ImageNet pretraining by 3.01% of F1 score on average over supervised predictive tasks and 1.53% of AUC on average over weakly supervised predictive tasks. Furthermore, two-country SSL pretraining has shown more stable performance than single-country pretraining, that is, two-country pretraining outperforms at least one of the single-country pretrainings by 1.93% of F1 on average over supervised predictive tasks and 1.36% of AUC on average over weakly-supervised predictive tasks. CONCLUSION: We find that using unlabeled image data for SSL pretraining in CRC related tasks is more effective than using ImageNet pretraining. Furthermore, SSL pretraining using data from multiple countries achieve more stable performance and better generalization than single-country pretraining.

dMIL-Transformer: Multiple Instance Learning Via Integrating Morphological and Spatial Information for Lymph Node Metastasis Classification.

Automated classification of lymph node metastasis (LNM) plays an important role in the diagnosis and prognosis. However, it is very challenging to achieve satisfactory performance in LNM classification, because both the morphology and spatial distribution of tumor regions should be taken into account. To address this problem, this article proposes a two-stage dMIL-Transformer framework, which integrates both the morphological and spatial information of the tumor regions based on the theory of multiple instance learning (MIL). In the first stage, a double Max-Min MIL (dMIL) strategy is devised to select the suspected top-K positive instances from each input histopathology image, which contains tens of thousands of patches (primarily negative). The dMIL strategy enables a better decision boundary for selecting the critical instances compared with other methods. In the second stage, a Transformer-based MIL aggregator is designed to integrate all the morphological and spatial information of the selected instances from the first stage. The self-attention mechanism is further employed to characterize the correlation between different instances and learn the bag-level representation for predicting the LNM category. The proposed dMIL-Transformer can effectively deal with the thorny classification in LNM with great visualization and interpretability. We conduct various experiments over three LNM datasets, and achieve 1.79%-7.50% performance improvement compared with other state-of-the-art methods.

Analysis of Shear Performance of Multi-Bolt Shear Connectors.

Bolt shear connectors used in prefabricated steel-concrete composite beams can be arranged into a group to enhance the construction efficiency, which will cause the multi-bolt effect and further affect the shear performance of bolt connectors. This paper developed three-dimensional finite element models of push-out specimens to investigate the shear performance of multi-bolt connectors. Numerical results showed that the friction coefficient at the interfaces between the steel girders and precast concrete (PC) slabs and bolt preload dramatically improved the initial stiffness of bolts; when the longitudinal spacing of bolts was reduced from 100 mm to 60 mm, the decrease in the average peak load per bolt was 3.5%, 9.2%, and 11.4% for bolts of 16 mm, 20 mm, and 24 mm diameters. A modified calculation method for the shear resistance of multi-bolt shear connectors was proposed based on the numerical analysis, and a simplified model of shear load versus relative slip was further developed.

RCMNet: A deep learning model assists CAR-T therapy for leukemia.

Acute leukemia is a type of blood cancer with a high mortality rate. Current therapeutic methods include bone marrow transplantation, supportive therapy, and chemotherapy. Although a satisfactory remission of the disease can be achieved, the risk of recurrence is still high. Therefore, novel treatments are demanding. Chimeric antigen receptor-T (CAR-T) therapy has emerged as a promising approach to treating and curing acute leukemia. To harness the therapeutic potential of CAR-T cell therapy for blood diseases, reliable cell morphological identification is crucial. Nevertheless, the identification of CAR-T cells is a big challenge posed by their phenotypic similarity with other blood cells. To address this substantial clinical challenge, herein we first construct a CAR-T dataset with 500 original microscopy images after staining. Following that, we create a novel integrated model called RCMNet (ResNet18 with Convolutional Block Attention Module and Multi-Head Self-Attention) that combines the convolutional neural network (CNN) and Transformer. The model shows 99.63% top-1 accuracy on the public dataset. Compared with previous reports, our model obtains satisfactory results for image classification. Although testing on the CAR-T cell dataset, a decent performance is observed, which is attributed to the limited size of the dataset. Transfer learning is adapted for RCMNet and a maximum of 83.36% accuracy is achieved, which is higher than that of other state-of-the-art models. This study evaluates the effectiveness of RCMNet on a big public dataset and translates it to a clinical dataset for diagnostic applications.

Molecular mapping for fruit-related traits, and joint identification of candidate genes and selective sweeps for seed size in melon.

Melon is a popular fruit vegetable crop worldwide with diverse morphological variation. We report a high-density genetic map of melon and nine major QTLs with physical region ranging from 43.47 kb to 1.89 Mb. Importantly, two seed-related trait QTLs were repeatedly detected in two environments, and the mapping region was narrowed to 522 kb according to a regional linkage analysis. A total of 40 annotated genes were screened for nonsynonymous variations, of which EVM0009818, involved in cytokinin-activated signaling, was differentially expressed in the young fruits of parents based on RNA-seq. Selective sweep analysis identified 152 sweep signals for seed size, including the two seed-related QTLs and nine homologs that have been verified to regulate seed size in Arabidopsis or rice. This work illustrates the power of a joint analysis combining resequencing-based genetic map for QTL mapping and a combination of KASP genotyping and RNA-seq analysis to facilitate QTL fine mapping.

Deep-3D microscope: 3D volumetric microscopy of thick scattering samples using a wide-field microscope and machine learning.

Confocal microscopy is a standard approach for obtaining volumetric images of a sample with high axial and lateral resolution, especially when dealing with scattering samples. Unfortunately, a confocal microscope is quite expensive compared to traditional microscopes. In addition, the point scanning in confocal microscopy leads to slow imaging speed and photobleaching due to the high dose of laser energy. In this paper, we demonstrate how the advances in machine learning can be exploited to "teach" a traditional wide-field microscope, one that's available in every lab, into producing 3D volumetric images like a confocal microscope. The key idea is to obtain multiple images with different focus settings using a wide-field microscope and use a 3D generative adversarial network (GAN) based neural network to learn the mapping between the blurry low-contrast image stacks obtained using a wide-field microscope and the sharp, high-contrast image stacks obtained using a confocal microscope. After training the network with widefield-confocal stack pairs, the network can reliably and accurately reconstruct 3D volumetric images that rival confocal images in terms of its lateral resolution, z-sectioning and image contrast. Our experimental results demonstrate generalization ability to handle unseen data, stability in the reconstruction results, high spatial resolution even when imaging thick (∼40 microns) highly-scattering samples. We believe that such learning-based microscopes have the potential to bring confocal imaging quality to every lab that has a wide-field microscope.

Deep learning-based single-shot autofocus method for digital microscopy.

Digital pathology is being transformed by artificial intelligence (AI)-based pathological diagnosis. One major challenge for correct AI diagnoses is to ensure the focus quality of captured images. Here, we propose a deep learning-based single-shot autofocus method for microscopy. We use a modified MobileNetV3, a lightweight network, to predict the defocus distance with a single-shot microscopy image acquired at an arbitrary image plane without secondary camera or additional optics. The defocus prediction takes only 9 ms with a focusing error of only ∼1/15 depth of field. We also provide implementation examples for the augmented reality microscope and the whole slide imaging (WSI) system. Our proposed technique can perform real-time and accurate autofocus which will not only support pathologists in their daily work, but also provide potential applications in the life sciences, material research, and industrial automatic detection.

Predicting Neurological Deterioration after Moderate Traumatic Brain Injury: Development and Validation of a Prediction Model Based on Data Collected on Admission.

Moderate traumatic brain injury (mTBI) is a heterogeneous entity that is poorly defined in the literature. Patients with mTBI have a high rate of neurological deterioration (ND), which is usually accompanied by poor prognosis and no definitive methods to predict. The purpose of this study is to develop and validate a prediction model that estimates the ND risk in patients with mTBI using data collected on admission. Data for 479 patients with mTBI collected retrospectively in our department were analyzed by logistic regression models. Bivariable logistic regression identified variables with a p < 0.05. Multi-variable logistic regression modeling with backward stepwise elimination was used to determine reduced parameters and establish a prediction model. The discrimination efficacy, calibration efficacy, and clinical utility of the prediction model were evaluated. The prediction model was validated using data for 176 patients collected from another hospital. Eight independent prognostic factors were identified: hypertension, Marshall scale (types III and IV), subdural hemorrhage (SDH), location of contusion (frontal and temporal contusions), Injury Severity Score >13, D-dimer level >11.4 mg/L, Glasgow Coma Scale score ≤10, and platelet count ≤152 × 109/L. A prediction model was established and was shown as a nomogram. Using bootstrapping, internal validation showed that the C-statistic of the prediction model was 0.881 (95% confidence interval [CI]: 0.849-0.909). The results of external validation showed that the nomogram could predict ND with an area under the curve of 0.827 (95% CI: 0.763-0.880). The present model, based on simple parameters collected on admission, can predict the risk of ND in patients with mTBI accurately. The high discriminative ability indicates the potential of this model for classifying patients with mTBI according to ND risk.

Ptychography-based high-throughput lensless on-chip microscopy via incremental proximal algorithms.

Ptychography-based lensless on-chip microscopy enables high-throughput imaging by retrieving the missing phase information from intensity measurements. Numerous reconstruction algorithms for ptychography have been proposed, yet only a few incremental algorithms can be extended to lensless on-chip microscopy because of large-scale datasets but limited computational efficiency. In this paper, we propose the use of accelerated proximal gradient methods for blind ptychographic phase retrieval in lensless on-chip microscopy. Incremental gradient approaches are adopted in the reconstruction routine. Our algorithms divide the phase retrieval problem into sub-problems involving the evaluation of proximal operator, stochastic gradient descent, and Wirtinger derivatives. We benchmark the performances of accelerated proximal gradient, extended ptychographic iterative engine, and alternating direction method of multipliers, and discuss their convergence and accuracy in both noisy and noiseless cases. We also validate our algorithms using experimental datasets, where full field of view measurements are captured to recover the high-resolution complex samples. Among these algorithms, accelerated proximal gradient presents the overall best performance regarding accuracy and convergence rate. The proposed methods may find applications in ptychographic reconstruction, especially for cases where a wide field of view and high resolution are desired at the same time.

Low-cost whole slide imaging system with single-shot autofocusing based on color-multiplexed illumination and deep learning.

Recent research on whole slide imaging (WSI) has greatly promoted the development of digital pathology. However, accurate autofocusing is still the main challenge for WSI acquisition and automated digital microscope. To address this problem, this paper describes a low cost WSI system and proposes a fast, robust autofocusing method based on deep learning. We use a programmable LED array for sample illumination. Before the brightfield image acquisition, we turn on a red and a green LED, and capture a color-multiplexed image, which is fed into a neural network for defocus distance estimation. After the focus tracking process, we employ a low-cost DIY adaptor to digitally adjust the photographic lens instead of the mechanical stage to perform axial position adjustment, and acquire the in-focus image under brightfield illumination. To ensure the calculation speed and image quality, we build a network model based on a 'light weight' backbone network architecture-MobileNetV3. Since the color-multiplexed coherent illuminated images contain abundant information about the defocus orientation, the proposed method enables high performance of autofocusing. Experimental results show that the proposed method can accurately predict the defocus distance of various types of samples and has good generalization ability for new types of samples. In the case of using GPU, the processing time for autofocusing is less than 0.1 second for each field of view, indicating that our method can further speed up the acquisition of whole slide images.

Neural network model assisted Fourier ptychography with Zernike aberration recovery and total variation constraint.

SIGNIFICANCE: Fourier ptychography (FP) is a computational imaging approach that achieves high-resolution reconstruction. Inspired by neural networks, many deep-learning-based methods are proposed to solve FP problems. However, the performance of FP still suffers from optical aberration, which needs to be considered. AIM: We present a neural network model for FP reconstructions that can make proper estimation toward aberration and achieve artifact-free reconstruction. APPROACH: Inspired by the iterative reconstruction of FP, we design a neural network model that mimics the forward imaging process of FP via TensorFlow. The sample and aberration are considered as learnable weights and optimized through back-propagation. Especially, we employ the Zernike terms instead of aberration to decrease the optimization freedom of pupil recovery and perform a high-accuracy estimation. Owing to the auto-differentiation capabilities of the neural network, we additionally utilize total variation regularization to improve the visual quality. RESULTS: We validate the performance of the reported method via both simulation and experiment. Our method exhibits higher robustness against sophisticated optical aberrations and achieves better image quality by reducing artifacts. CONCLUSIONS: The forward neural network model can jointly recover the high-resolution sample and optical aberration in iterative FP reconstruction. We hope our method that can provide a neural-network perspective to solve iterative-based coherent or incoherent imaging problems.

Neural Architecture Search for compressed sensing Magnetic Resonance image reconstruction.

Recent works have demonstrated that deep learning (DL) based compressed sensing (CS) implementation can accelerate Magnetic Resonance (MR) Imaging by reconstructing MR images from sub-sampled k-space data. However, network architectures adopted in previous methods are all designed by handcraft. Neural Architecture Search (NAS) algorithms can automatically build neural network architectures which have outperformed human designed ones in several vision tasks. Inspired by this, here we proposed a novel and efficient network for the MR image reconstruction problem via NAS instead of manual attempts. Particularly, a specific cell structure, which was integrated into the model-driven MR reconstruction pipeline, was automatically searched from a flexible pre-defined operation search space in a differentiable manner. Experimental results show that our searched network can produce better reconstruction results compared to previous state-of-the-art methods in terms of PSNR and SSIM with 4∼6 times fewer computation resources. Extensive experiments were conducted to analyze how hyper-parameters affect reconstruction performance and the searched structures. The generalizability of the searched architecture was also evaluated on different organ MR datasets. Our proposed method can reach a better trade-off between computation cost and reconstruction performance for MR reconstruction problem with good generalizability and offer insights to design neural networks for other medical image applications. The evaluation code will be available at https://github.com/yjump/NAS-for-CSMRI.

A High-Quality Melon Genome Assembly Provides Insights into Genetic Basis of Fruit Trait Improvement.

Accurate reference genomes have become indispensable tools for characterization of genetic and functional variations. Here we generated a high-quality assembly of the melon Payzawat using a combination of short-read sequencing, single-molecule real-time sequencing, Hi-C, and a high-density genetic map. The final 12 chromosome-level scaffolds cover ∼94.13% of the estimated genome (398.57 Mb). Compared with the published DHL92 genome, our assembly exhibits a 157-fold increase in contig length and remarkable improvements in the assembly of centromeres and telomeres. Six genes within STHQF12.4 on pseudochromosome 12, identified from whole-genome comparison between Payzawat and DHL92, may explain a considerable proportion of the skin thickness. In addition, our population study showed that melon domesticated at multiple times from whole-genome perspective and melons in China are introduced from different routes. Selective sweeps underlying the genes related to desirable traits, haplotypes of alleles associated with agronomic traits, and the variants from resequencing data enable efficient breeding.