Deep reinforcement learning with significant multiplications inference.

We propose a sparse computation method for optimizing the inference of neural networks in reinforcement learning (RL) tasks. Motivated by the processing abilities of the brain, this method combines simple neural network pruning with a delta-network algorithm to account for the input data correlations. The former mimics neuroplasticity by eliminating inefficient connections; the latter makes it possible to update neuron states only when their changes exceed a certain threshold. This combination significantly reduces the number of multiplications during the neural network inference for fast neuromorphic computing. We tested the approach in popular deep RL tasks, yielding up to a 100-fold reduction in the number of required multiplications without substantial performance loss (sometimes, the performance even improved).

Optimal MRI undersampling patterns for ultimate benefit of medical vision tasks.

Compressed sensing is commonly concerned with optimizing the image quality after a partial undersampling of the measurable k-space to accelerate MRI. In this article, we propose to change the focus from the quality of the reconstructed image to the quality of the downstream image analysis outcome. Specifically, we propose to optimize the patterns according to how well a sought-after pathology could be detected or localized in the reconstructed images. We find the optimal undersampling patterns in k-space that maximize target value functions of interest in commonplace medical vision problems (reconstruction, segmentation, and classification) and propose a new iterative gradient sampling routine universally suitable for these tasks. We validate the proposed MRI acceleration paradigm on three classical medical datasets, demonstrating a noticeable improvement of the target metrics at the high acceleration factors (for the segmentation problem at ×16 acceleration, we report up to 12% improvement in Dice score over the other undersampling patterns).

Medical image captioning via generative pretrained transformers.

The proposed model for automatic clinical image caption generation combines the analysis of radiological scans with structured patient information from the textual records. It uses two language models, the Show-Attend-Tell and the GPT-3, to generate comprehensive and descriptive radiology records. The generated textual summary contains essential information about pathologies found, their location, along with the 2D heatmaps that localize each pathology on the scans. The model has been tested on two medical datasets, the Open-I, MIMIC-CXR, and the general-purpose MS-COCO, and the results measured with natural language assessment metrics demonstrated its efficient applicability to chest X-ray image captioning.

K2S Challenge: From Undersampled K-Space to Automatic Segmentation.

Magnetic Resonance Imaging (MRI) offers strong soft tissue contrast but suffers from long acquisition times and requires tedious annotation from radiologists. Traditionally, these challenges have been addressed separately with reconstruction and image analysis algorithms. To see if performance could be improved by treating both as end-to-end, we hosted the K2S challenge, in which challenge participants segmented knee bones and cartilage from 8× undersampled k-space. We curated the 300-patient K2S dataset of multicoil raw k-space and radiologist quality-checked segmentations. 87 teams registered for the challenge and there were 12 submissions, varying in methodologies from serial reconstruction and segmentation to end-to-end networks to another that eschewed a reconstruction algorithm altogether. Four teams produced strong submissions, with the winner having a weighted Dice Similarity Coefficient of 0.910 ± 0.021 across knee bones and cartilage. Interestingly, there was no correlation between reconstruction and segmentation metrics. Further analysis showed the top four submissions were suitable for downstream biomarker analysis, largely preserving cartilage thicknesses and key bone shape features with respect to ground truth. K2S thus showed the value in considering reconstruction and image analysis as end-to-end tasks, as this leaves room for optimization while more realistically reflecting the long-term use case of tools being developed by the MR community.

Evaluation of an Integrated Spectroscopy and Classification Platform for Point-of-Care Core Needle Biopsy Assessment: Performance Characteristics from Ex Vivo Renal Mass Biopsies.

PURPOSE: To evaluate a transmission optical spectroscopy instrument for rapid ex vivo assessment of core needle cancer biopsies (CNBs) at the point of care. MATERIALS AND METHODS: CNBs from surgically resected renal tumors and nontumor regions were scanned on their sampling trays with a custom spectroscopy instrument. After extracting principal spectral components, machine learning was used to train logistic regression, support vector machines, and random decision forest (RF) classifiers on 80% of randomized and stratified data. The algorithms were evaluated on the remaining 20% of the data set held out during training. Binary classification (tumor/nontumor) was performed based on a decision threshold. Multinomial classification was also performed to differentiate between the subtypes of renal cell carcinoma (RCC) and account for potential confounding effects from fat, blood, and necrotic tissue. Classifiers were compared based on sensitivity, specificity, and positive predictive value (PPV) relative to a histopathologic standard. RESULTS: A total of 545 CNBs from 102 patients were analyzed, yielding 5,583 spectra after outlier exclusion. At the individual spectra level, the best performing algorithm was RF with sensitivities of 96% and 92% and specificities of 90% and 89%, for the binary and multiclass analyses, respectively. At the full CNB level, RF algorithm also showed the highest sensitivity and specificity (93% and 91%, respectively). For RCC subtypes, the highest sensitivity and PPV were attained for clear cell (93.5%) and chromophobe (98.2%) subtypes, respectively. CONCLUSIONS: Ex vivo spectroscopy imaging paired with machine learning can accurately characterize renal mass CNB at the time of tissue acquisition.

Infrared Laser-Based Single Cell Permeabilization by Plasma Membrane Temperature Gradients.

Single cell microinjection provides precise tuning of the volume and timing of delivery into the treated cells; however, it also introduces workflow complexity that requires highly skilled operators and specialized equipment. Laser-based microinjection provides an alternative method for targeting a single cell using a common laser and a workflow that may be readily standardized. This paper presents experiments using a 1550 nm, 100 fs pulse duration laser with a repetition rate of 20 ns for laser-based microinjection and calculations of the hypothesized physical mechanism responsible for the experimentally observed permeabilization. Chinese Hamster Ovarian (CHO) cells exposed to this laser underwent propidium iodide uptake, demonstrating the potential for selective cell permeabilization. The agreement between the experimental conditions and the electropermeabilization threshold based on estimated changes in the transmembrane potential induced by a laser-induced plasma membrane temperature gradient, even without accounting for enhancement due to traditional electroporation, strengthens the hypothesis of this mechanism for the experimental observations. Compared to standard 800 nm lasers, 1550 nm fs lasers may ultimately provide a lower cost microinjection method that readily interfaces with a microscope and is agnostic to operator skill, while inducing fewer deleterious effects (e.g., temperature rise, shockwaves, and cavitation bubbles).

Tubular shape aware data generation for segmentation in medical imaging.

PURPOSE: Chest X-ray is one of the most widespread examinations of the human body. In interventional radiology, its use is frequently associated with the need to visualize various tube-like objects, such as puncture needles, guiding sheaths, wires, and catheters. Detection and precise localization of these tube-like objects in the X-ray images are, therefore, of utmost value, catalyzing the development of accurate target-specific segmentation algorithms. Similar to the other medical imaging tasks, the manual pixel-wise annotation of the tubes is a resource-consuming process. METHODS: In this work, we aim to alleviate the lack of annotated images by using artificial data. Specifically, we present an approach for synthetic generation of the tube-shaped objects, with a generative adversarial network being regularized with a prior-shape constraint. Namely, our model uses Frangi-based regularization to draw synthetic tubes in the predefined fake mask regions and, then, uses the adversarial component to preserve the global realistic appearance of the synthesized image. RESULTS: Our method eliminates the need for the paired image-mask data and requires only a weakly labeled dataset, with fine-tuning on a small paired sample (10-20 images) proving sufficient to reach the accuracy of the fully supervised models. CONCLUSION: We report the applicability of the approach for the task of segmenting tubes and catheters in the X-ray images, whereas the results should also hold for the other acquisition modalities and image computing applications that contain tubular objects.

MOOD 2020: A Public Benchmark for Out-of-Distribution Detection and Localization on Medical Images.

Detecting Out-of-Distribution (OoD) data is one of the greatest challenges in safe and robust deployment of machine learning algorithms in medicine. When the algorithms encounter cases that deviate from the distribution of the training data, they often produce incorrect and over-confident predictions. OoD detection algorithms aim to catch erroneous predictions in advance by analysing the data distribution and detecting potential instances of failure. Moreover, flagging OoD cases may support human readers in identifying incidental findings. Due to the increased interest in OoD algorithms, benchmarks for different domains have recently been established. In the medical imaging domain, for which reliable predictions are often essential, an open benchmark has been missing. We introduce the Medical-Out-Of-Distribution-Analysis-Challenge (MOOD) as an open, fair, and unbiased benchmark for OoD methods in the medical imaging domain. The analysis of the submitted algorithms shows that performance has a strong positive correlation with the perceived difficulty, and that all algorithms show a high variance for different anomalies, making it yet hard to recommend them for clinical practice. We also see a strong correlation between challenge ranking and performance on a simple toy test set, indicating that this might be a valuable addition as a proxy dataset during anomaly detection algorithm development.

3D denoised completion network for deep single-pixel reconstruction of hyperspectral images.

Single-pixel imaging acquires an image by measuring its coefficients in a transform domain, thanks to a spatial light modulator. However, as measurements are sequential, only a few coefficients can be measured in the real-time applications. Therefore, single-pixel reconstruction is usually an underdetermined inverse problem that requires regularization to obtain an appropriate solution. Combined with a spectral detector, the concept of single-pixel imaging allows for hyperspectral imaging. While each channel can be reconstructed independently, we propose to exploit the spectral redundancy between channels to regularize the reconstruction problem. In particular, we introduce a denoised completion network that includes 3D convolution filters. Contrary to black-box approaches, our network combines the classical Tikhonov theory with the deep learning methodology, leading to an explainable network. Considering both simulated and experimental data, we demonstrate that the proposed approach yields hyperspectral images with higher quantitative metrics than the approaches developed for grayscale images.

Deep negative volume segmentation.

Clinical examination of three-dimensional image data of compound anatomical objects, such as complex joints, remains a tedious process, demanding the time and the expertise of physicians. For instance, automation of the segmentation task of the TMJ (temporomandibular joint) has been hindered by its compound three-dimensional shape, multiple overlaid textures, an abundance of surrounding irregularities in the skull, and a virtually omnidirectional range of the jaw's motion-all of which extend the manual annotation process to more than an hour per patient. To address the challenge, we invent a new workflow for the 3D segmentation task: namely, we propose to segment empty spaces between all the tissues surrounding the object-the so-called negative volume segmentation. Our approach is an end-to-end pipeline that comprises a V-Net for bone segmentation, a 3D volume construction by inflation of the reconstructed bone head in all directions along the normal vector to its mesh faces. Eventually confined within the skull bones, the inflated surface occupies the entire "negative" space in the joint, effectively providing a geometrical/topological metric of the joint's health. We validate the idea on the CT scans in a 50-patient dataset, annotated by experts in maxillofacial medicine, quantitatively compare the asymmetry given the left and the right negative volumes, and automate the entire framework for clinical adoption.

Optical Mapping-Validated Machine Learning Improves Atrial Fibrillation Driver Detection by Multi-Electrode Mapping.

BACKGROUND: Atrial fibrillation (AF) can be maintained by localized intramural reentrant drivers. However, AF driver detection by clinical surface-only multielectrode mapping (MEM) has relied on subjective interpretation of activation maps. We hypothesized that application of machine learning to electrogram frequency spectra may accurately automate driver detection by MEM and add some objectivity to the interpretation of MEM findings. METHODS: Temporally and spatially stable single AF drivers were mapped simultaneously in explanted human atria (n=11) by subsurface near-infrared optical mapping (NIOM; 0.3 mm2 resolution) and 64-electrode MEM (higher density or lower density with 3 and 9 mm2 resolution, respectively). Unipolar MEM and NIOM recordings were processed by Fourier transform analysis into 28 407 total Fourier spectra. Thirty-five features for machine learning were extracted from each Fourier spectrum. RESULTS: Targeted driver ablation and NIOM activation maps efficiently defined the center and periphery of AF driver preferential tracks and provided validated annotations for driver versus nondriver electrodes in MEM arrays. Compared with analysis of single electrogram frequency features, averaging the features from each of the 8 neighboring electrodes, significantly improved classification of AF driver electrograms. The classification metrics increased when less strict annotation, including driver periphery electrodes, were added to driver center annotation. Notably, f1-score for the binary classification of higher-density catheter data set was significantly higher than that of lower-density catheter (0.81±0.02 versus 0.66±0.04, P<0.05). The trained algorithm correctly highlighted 86% of driver regions with higher density but only 80% with lower-density MEM arrays (81% for lower-density+higher-density arrays together). CONCLUSIONS: The machine learning model pretrained on Fourier spectrum features allows efficient classification of electrograms recordings as AF driver or nondriver compared with the NIOM gold-standard. Future application of NIOM-validated machine learning approach may improve the accuracy of AF driver detection for targeted ablation treatment in patients.

Reinforcement learning for suppression of collective activity in oscillatory ensembles.

We present the use of modern machine learning approaches to suppress self-sustained collective oscillations typically signaled by ensembles of degenerative neurons in the brain. The proposed hybrid model relies on two major components: an environment of oscillators and a policy-based reinforcement learning block. We report a model-agnostic synchrony control based on proximal policy optimization and two artificial neural networks in an Actor-Critic configuration. A class of physically meaningful reward functions enabling the suppression of collective oscillatory mode is proposed. The synchrony suppression is demonstrated for two models of neuronal populations-for the ensembles of globally coupled limit-cycle Bonhoeffer-van der Pol oscillators and for the bursting Hindmarsh-Rose neurons using rectangular and charge-balanced stimuli.

Fluorescence Phenomena in Nerve-Labeling Styryl-Type Dyes.

Several classes of diversely substituted styryl type dyes have been synthesized with the goal of extending their expected fluorescent properties as much towards red as possible given the constraint that they maintain drug-like properties and retain high affinity binding to their biological target. We report on the synthesis, optical properties of a series of styryl dyes ( d1-d14 ), and the anomalous photophysical behavior of several of these Donor-Acceptor pairs separated by long conjugated π-systems ( d7-d10 ). We further describe an unusual dual emission behavior with two distinct ground state conformers which could be individually excited to locally excited (LE) and twisted intramolecular charge transfer (TICT) excited state in push-pull dye systems ( d7 , d9 and d10 ). Additionally, unexpected emission behavior in dye systems d7 and d8 wherein the amino- derivative d7 displayed a dual emission in polar medium while the N,N-dimethyl derivative d8 and other methylated derivatives d12-d14 showed only LE emission but did not show any TICT emission. Based on photophysical and nerve binding studies, we down selected compounds that exhibited the most robust fluorescent staining of nerve tissue sections. These dyes ( d7 , d9 , and d10 ) were subsequently selected for in-vivo fluorescence imaging studies in rodents using the small animal multispectral imaging instrument and the dual-mode laparoscopic instrument developed in-house.

Plasma membrane temperature gradients and multiple cell permeabilization induced by low peak power density femtosecond lasers.

Calculations indicate that selectively heating the extracellular media induces membrane temperature gradients that combine with electric fields and a temperature-induced reduction in the electropermeabilization threshold to potentially facilitate exogenous molecular delivery. Experiments by a wide-field, pulsed femtosecond laser with peak power density far below typical single cell optical delivery systems confirmed this hypothesis. Operating this laser in continuous wave mode at the same average power permeabilized many fewer cells, suggesting that bulk heating alone is insufficient and temperature gradients are crucial for permeabilization. This work suggests promising opportunities for a high throughput, low cost, contactless method for laser mediated exogenous molecule delivery without the complex optics of typical single cell optoinjection, for potential integration into microscope imaging and microfluidic systems.

Quantitative amplification of weak images by nonlinear propagation.

We demonstrate quantitative nonlinear recovery of images that have been hidden by the addition of partially coherent light. The method assumes a simple model for spatial nonlinearity that allows direct Laplacian inversion based on intensity transport.

Spectral dynamics of spatially incoherent modulation instability.

To date, all experiments in nonlinear statistical optics have relied on beams whose transverse spatial statistics were Gaussian. Here, we present a new technique to generalize these studies by using a spatial light modulator to create spatially incoherent beams with arbitrary spectral distributions. As a specific example of the new dynamics possible, we consider the spatial modulation instability of a partially coherent beam. We show that, for statistical beams of uniform intensity and equal correlation length, the underlying spectral shape determines the threshold and visibility of intensity modulations as well as the spectral profile of the growing sidebands. We demonstrate the behavior using statistical light, but the results will hold for any wave-kinetic system, such as plasma, ultracold gases, and turbulent acoustic waves.

Instability-driven recovery of diffused images.

We demonstrate the nonlinear recovery of diffused images in a self-focusing photorefractive medium. The method is based on the convolution property of nonlinearity, in which related modes reinforce each other as they propagate. The resulting mode coupling enables energy transfer from the scattered light to the underlying signal. The dynamics is well described by a model in which the signal seeds a modulation instability in the diffused background.

Diffraction from an edge in a self-focusing medium.

We experimentally demonstrate diffraction from a straight edge in a medium with self-focusing nonlinearity. Diffraction into the shadow region is suppressed with increasing nonlinearity, but mode coupling leads to excitations and traveling waves on the high-intensity side. Theoretically, we interpret these modulations as spatially dispersive shock waves with negative pressure.

Modulation instability of a coherent-incoherent mixture.

We examine the nonlinear coupling and modulation instability of a coherent beam with one that is partially spatially incoherent. Using a mutual coherence approach, we derive the growth rate for perturbations and show that the presence of any amount of coherent component eliminates the nonlinear threshold for instability. The fraction of coherent light is shown to determine the gain and characteristic period of the resulting patterns. Theoretical considerations are confirmed by numerical simulation and by experimental observations in a self-focusing photorefractive crystal.

Nonlinear focusing and defocusing of partially coherent spatial beams.

We consider the propagation of a partially coherent spatial beam in both self-focusing and self-defocusing nonlinear media. Using a Gaussian-Schell model, we derive an equation governing the width of highly incoherent beams as they propagate in both types of media and confirm its validity by using numerical simulations. Experiments performed in a biased photorefractive crystal match the predicted scaling.