Sharing massive biomedical data at magnitudes lower bandwidth using implicit neural function.

Efficient storage and sharing of massive biomedical data would open up their wide accessibility to different institutions and disciplines. However, compressors tailored for natural photos/videos are rapidly limited for biomedical data, while emerging deep learning-based methods demand huge training data and are difficult to generalize. Here, we propose to conduct Biomedical data compRession with Implicit nEural Function (BRIEF) by representing the target data with compact neural networks, which are data specific and thus have no generalization issues. Benefiting from the strong representation capability of implicit neural function, BRIEF achieves 2[Formula: see text]3 orders of magnitude compression on diverse biomedical data at significantly higher fidelity than existing techniques. Besides, BRIEF is of consistent performance across the whole data volume, and supports customized spatially varying fidelity. BRIEF's multifold advantageous features also serve reliable downstream tasks at low bandwidth. Our approach will facilitate low-bandwidth data sharing and promote collaboration and progress in the biomedical field.

iSMOD: an integrative browser for image-based single-cell multi-omics data.

Genomic and transcriptomic image data, represented by DNA and RNA fluorescence in situ hybridization (FISH), respectively, together with proteomic data, particularly that related to nuclear proteins, can help elucidate gene regulation in relation to the spatial positions of chromatins, messenger RNAs, and key proteins. However, methods for image-based multi-omics data collection and analysis are lacking. To this end, we aimed to develop the first integrative browser called iSMOD (image-based Single-cell Multi-omics Database) to collect and browse comprehensive FISH and nucleus proteomics data based on the title, abstract, and related experimental figures, which integrates multi-omics studies focusing on the key players in the cell nucleus from 20 000+ (still growing) published papers. We have also provided several exemplar demonstrations to show iSMOD's wide applications-profiling multi-omics research to reveal the molecular target for diseases; exploring the working mechanism behind biological phenomena using multi-omics interactions, and integrating the 3D multi-omics data in a virtual cell nucleus. iSMOD is a cornerstone for delineating a global view of relevant research to enable the integration of scattered data and thus provides new insights regarding the missing components of molecular pathway mechanisms and facilitates improved and efficient scientific research.

Snapshot compressive imaging based digital image correlation: temporally super-resolved full-resolution deformation measurement.

The limited throughput of a digital image correlation (DIC) system hampers measuring deformations at both high spatial resolution and high temporal resolution. To address this dilemma, in this paper we propose to integrate snapshot compressive imaging (SCI)-a recently proposed computational imaging approach-into DIC for high-speed, high-resolution deformation measurement. Specifically, an SCI-DIC system is established to encode a sequence of fast changing speckle patterns into a snapshot and a high-accuracy speckle decompress SCI (Sp-DeSCI) algorithm is proposed for computational reconstruction of the speckle sequence. To adapt SCI reconstruction to the unique characteristics of speckle patterns, we propose three techniques under SCI reconstruction framework to secure high-precision reconstruction, including the normalized sum squared difference criterion, speckle-adaptive patch search strategy, and adaptive group aggregation. For efficacy validation of the proposed Sp-DeSCI, we conducted extensive simulated experiments and a four-point bending SCI-DIC experiment on real data. Both simulation and real experiments verify that the Sp-DeSCI successfully removes the deviations of reconstructed speckles in DeSCI and provides the highest displacement accuracy among existing algorithms. The SCI-DIC system together with the Sp-DeSCI algorithm can offer temporally super-resolved deformation measurement at full spatial resolution, and can potentially replace conventional high-speed DIC in real measurements.

Weighted sampling-adaptive single-pixel sensing.

The novel single-pixel sensing technique that uses an end-to-end neural network for joint optimization achieves high-level semantic sensing, which is effective but computation-consuming for varied sampling rates. In this Letter, we report a weighted optimization technique for sampling-adaptive single-pixel sensing, which only needs to train the network once for any dynamic sampling rate. Specifically, we innovatively introduce a weighting scheme in the encoding process to characterize different patterns' modulation efficiencies, in which the modulation patterns and their corresponding weights are updated iteratively. The optimal pattern series with the highest weights is employed for light modulation in the experimental implementation, thus achieving highly efficient sensing. Experiments validated that once the network is trained with a sampling rate of 1, the single-target classification accuracy reaches up to 95.00% at a sampling rate of 0.03 on the MNIST dataset and 90.20% at a sampling rate of 0.07 on the CCPD dataset for multi-target sensing.

Plug-and-play pixel super-resolution phase retrieval for digital holography.

In order to increase signal-to-noise ratio in optical imaging, most detectors sacrifice resolution to increase pixel size in a confined area, which impedes further development of high throughput holographic imaging. Although the pixel super-resolution technique (PSR) enables resolution enhancement, it suffers from the trade-off between reconstruction quality and super-resolution ratio. In this work, we report a high-fidelity PSR phase retrieval method with plug-and-play optimization, termed PNP-PSR. It decomposes PSR reconstruction into independent sub-problems based on generalized alternating projection framework. An alternating projection operator and an enhancing neural network are employed to tackle the measurement fidelity and statistical prior regularization, respectively. PNP-PSR incorporates the advantages of individual operators, achieving both high efficiency and noise robustness. Extensive experiments show that PNP-PSR outperforms the existing techniques in both resolution enhancement and noise suppression.

High-axial-resolution single-molecule localization under dense excitation with a multi-channel deep U-Net.

Single-molecule localization microscopy (SMLM) can bypass the diffraction limit of optical microscopes and greatly improve the resolution in fluorescence microscopy. By introducing the point spread function (PSF) engineering technique, we can customize depth varying PSF to achieve higher axial resolution. However, most existing 3D single-molecule localization algorithms require excited fluorescent molecules to be sparse and captured at high signal-to-noise ratios, which results in a long acquisition time and precludes SMLM's further applications in many potential fields. To address this problem, we propose a novel 3D single-molecular localization method based on a multi-channel neural network based on U-Net. By leveraging the deep network's great advantages in feature extraction, the proposed network can reliably discriminate dense fluorescent molecules with overlapped PSFs and corrupted by sensor noise. Both simulated and real experiments demonstrate its superior performance in PSF engineered microscopes with short exposure and dense excitations, which holds great potential in fast 3D super-resolution microscopy.

Single-pixel ptychography.

Ptychography is a predominant non-interferometric technique to image large complex fields but with quite a narrow working spectrum, because diffraction measurements require dense array detection with an ultra-high dynamic range. Here we report a single-pixel ptychography technique that realizes non-interferometric and non-scanning complex-field imaging in a wide waveband, where 2D dense detector arrays are not available. A single-pixel detector is placed in the far field to record the DC-only component of the diffracted wavefront scattered from the target field, which is illuminated by a sequence of binary modulation patterns. This decreases the measurements' dynamic range by several orders of magnitude. We employ an efficient single-pixel phase-retrieval algorithm to jointly recover the field's 2D amplitude and phase maps from the 1D intensity-only measurement sequence. No a priori object information is needed in the recovery process. We validate the technique's quantitative phase imaging nature using both calibrated phase objects and biological samples and demonstrate its wide working spectrum with both 488-nm visible light and 980-nm near-infrared light.

Probabilistic natural mapping of gene-level tests for genome-wide association studies.

Genome-wide association studies (GWASs) generally focus on a single marker, which limits the elucidation of the genetic architecture of complex traits. Herein, we present a new computational framework, termed probabilistic natural mapping (PALM), for performing gene-level association tests. PALM robustly reveals the inherent genomic structures of genes and generates feature representations that can be seamlessly incorporated into conventional statistic tests. Our approach substantially improves the effectiveness of uncovering associations derived from a subgroup of variants with weak effects, which represents a known challenge associated with existing methods. We applied PALM in a gastric cancer GWAS and identified two additional gastric cancer-associated susceptibility genes, NOC3L and RUNDC2A. The robust susceptibility discoveries of PALM are widely supported by existing studies from other biological perspectives. PALM will be useful for further GWAS analytical strategies that use gene-level analyses.

Single-shot compressed ultrafast photography based on U-net network.

The compressive ultrafast photography (CUP) has achieved real-time femtosecond imaging based on the compressive-sensing methods. However, the reconstruction performance usually suffers from artifacts brought by strong noise, aberration, and distortion, which prevents its applications. We propose a deep compressive ultrafast photography (DeepCUP) method. Various numerical simulations have been demonstrated on both the MNIST and UCF-101 datasets and compared with other state-of-the-art algorithms. The result shows that our DeepCUP has a superior performance in both PSNR and SSIM compared to previous compressed-sensing methods. We also illustrate the outstanding performance of the proposed method under system errors and noise in comparison to other methods.

Single-pixel phase and fluorescence microscope.

Multimodal microscopes either use multiple cameras or a single camera to multiplex different modes spatially. The former needs expertise demanding alignment and the latter suffers from limited spatial resolution. Here, we report an alignment-free full-resolution simultaneous fluorescence and phase imaging approach using single-pixel detectors. By combining reference-free interferometry with single-pixel imaging scheme, we employ structured illumination to encode the phase and fluorescence of the sample into two single-pixel detection arms, and then conduct reconstruction computationally from the illumination patterns and recorded correlated measurements. The recovered fluorescence and phase images are inherently aligned thanks to single-pixel imaging scheme. To validate the proposed method, we built a proof-of-concept setup for first imaging the phase of an etched glass with given etching depth and then imaging the phase and fluorescence of the quantum dot sample. This method holds great potential for multispectral fluorescence microscopy with additional single-pixel detectors or a spectrometer. Besides, this cost-efficient multimodal system might find broad applications in biomedical science and material science.

Doubling the pixel count limitation of single-pixel imaging via sinusoidal amplitude modulation.

We demonstrate a single-pixel imaging (SPI) method that can achieve pixel resolution beyond the physical limitation of the spatial light modulator (SLM), by adopting sinusoidal amplitude modulation and frequency filtering. Through light field analysis, we observe that the induced intensity with a squared value of the amplitude contains higher frequency components. By filtering out the zero frequency of the sinusoidal amplitude in the Fourier domain, we can separate out the higher frequency components, which enables SPI with higher resolving ability and thus beyond the limitation of the SLM. Further, to address the speed issue in grayscale spatial light modulation, we propose a fast implementation scheme with tens-of-kilohertz refresh rate. Specifically, we use a digital micromirror device (DMD) working at the full frame rate to conduct binarized sinusoidal patterning in the spatial domain and pinhole filtering eliminating the binarization error in the Fourier domain. For experimental validation, we build a single-pixel microscope to retrieve 1200 × 1200-pixel images via a sub-megapixel DMD, and the setup achieves comparable performance to array sensor microscopy and provides additional sectioning ability.

Snapshot quantitative phase microscopy with a printed film.

This paper proposes a low-cost snapshot quantitative phase imaging approach. The setup is simple and adds only a printed film to a conventional microscope. The phase of a sample is regarded as an additional aberration of the optical imaging system. And the image captured through a phase object is modeled as the distorted version of a projected pattern. An optimization algorithm is utilized to recover the phase information via distortion estimation. We demonstrate our method on various samples such as a micro-lens array, IMR90 cells and the dynamic evaporation process of a water drop, and our approach has a capability of real-time phase imaging for highly dynamic phenomenon using a traditional microscope.

Snapshot hyperspectral imaging via spectral basis multiplexing in Fourier domain.

Hyperspectral imaging is an important tool having been applied in various fields, but still limited in observation of dynamic scenes. In this paper, we propose a snapshot hyperspectral imaging technique which exploits both spectral and spatial sparsity of natural scenes. Under the computational imaging scheme, we conduct spectral dimension reduction and spatial frequency truncation to the hyperspectral data cube and snapshot it in a low cost manner. Specifically, we modulate the spectral variations by several broadband spectral filters, and then map these modulated images into different regions in the Fourier domain. The encoded image compressed in both spectral and spatial are finally collected by a monochrome detector. Correspondingly, the reconstruction is essentially a Fourier domain extraction and spectral dimensional back projection with low computational load. This Fourier-spectral multiplexing in a 2D sensor simplifies both the encoding and decoding process, and makes hyperspectral data captured in a low cost manner. We demonstrate the high performance of our method by quantitative evaluation on simulation data and build a prototype system experimentally for further validation.

Single-shot lensless imaging via simultaneous multi-angle LED illumination.

Lensless imaging is a technique that records diffraction patterns without using lenses and recovers the complex field of object via phase retrieval. Robust lensless phase retrieval process usually requires multiple measurements with defocus variation, transverse translation or angle-varied illumination. However, making such diverse measurements is time-consuming and limits the application of lensless setup for dynamic samples. In this paper, we propose a single-shot lensless imaging scheme via simultaneous multi-angle LED illumination. Diffraction patterns under multi-angle lights are recorded by different areas of the sensor within a single shot. An optimization algorithm is applied to utilize the single-shot measurement and retrieve the aliasing information for reconstruction. We first use numerical simulations to evaluate the proposed scheme quantitatively by comparisons with the multi-acquisition case. Then a proof-of-concept lensless setup is built to validate the method by imaging a resolution chart and biological samples, achieving ∼ 4.92 µm half-pitch resolution and ∼ 1.202mm2 field of view (FOV). We also discuss different design tradeoffs and present a 4-frame acquisition scheme (with ∼ 3.48 µm half-pitch resolution and ∼ 2.35 × 2.55 mm2 FOV) to show the flexibility of performance enhancement by capturing more measurements.

Adaptive polarization-difference transient imaging for depth estimation in scattering media.

Introducing polarization into transient imaging improves depth estimation in participating media, by discriminating reflective from scattered light transport and calculating depth from the former component only. Previous works have leveraged this approach under the assumption of uniform polarization properties. However, the orientation and intensity of polarization inside scattering media is nonuniform, both in the spatial and temporal domains. As a result of this simplifying assumption, the accuracy of the estimated depth worsens significantly as the optical thickness of the medium increases. In this Letter, we introduce a novel adaptive polarization-difference method for transient imaging, taking into account the nonuniform nature of polarization in scattering media. Our results demonstrate a superior performance for impulse-based transient imaging over previous unpolarized or uniform approaches.

Experimental comparison of single-pixel imaging algorithms.

Single-pixel imaging (SPI) is a novel technique that captures 2D images using a photodiode, instead of conventional 2D array sensors. SPI has high signal-to-noise ratio, wide spectral range, low cost, and robustness to light scattering. Various algorithms have been proposed for SPI reconstruction, including linear correlation methods, the alternating projection (AP) method, and compressive sensing (CS) based methods. However, there has been no comprehensive review discussing respective advantages, which is important for SPI's further applications and development. In this paper, we review and compare these algorithms in a unified reconstruction framework. We also propose two other SPI algorithms, including a conjugate gradient descent (CGD) based method and a Poisson maximum-likelihood-based method. Both simulations and experiments validate the following conclusions: to obtain comparable reconstruction accuracy, the CS-based total variation (TV) regularization method requires the fewest measurements and consumes the least running time for small-scale reconstruction, the CGD and AP methods run fastest in large-scale cases, and the TV and AP methods are the most robust to measurement noise. In a word, there are trade-offs in capture efficiency, computational complexity, and robustness to noise among different SPI algorithms. We have released our source code for non-commercial use.

Contrast and resolution enhanced optical sectioning in scattering tissue using line-scanning two-photon structured illumination microscopy.

Optical sectioning imaging with high spatial resolution deep inside scattering samples such as mammalian brain is of great interest in biological study. Conventional two-photon microscopy deteriorates in focus when light scattering increases. Here we develop an optical sectioning enhanced two-photon technique which incorporates structured illumination into line-scanning spatial-temporal focusing microscopy (LTSIM), and generate patterned illumination via laser intensity modulation synchronized with scanning. LTSIM brings scattering background elimination and in-focus contrast enhancement, and realizes nearly 2-fold increase in spatial resolution to ∼208 nm laterally and ∼0.94 µm axially. In addition, the intensity modulated line-scanning implementation of LTSIM enables fast and flexible generation of structured illumination, permitting adjustable spatial frequency profiles to optimize image contrast. The highly qualified optical sectioning ability of our system is demonstrated on samples including tissue phantom, C. elegans and mouse brain at depths over hundreds of microns.

Content-adaptive ghost imaging of dynamic scenes.

Limited by long acquisition time of 2D ghost imaging, current ghost imaging systems are so far inapplicable for dynamic scenes. However, it's been demonstrated that nature images are spatiotemporally redundant and the redundancy is scene dependent. Inspired by that, we propose a content-adaptive computational ghost imaging approach to achieve high reconstruction quality under a small number of measurements, and thus achieve ghost imaging of dynamic scenes. To utilize content-adaptive inter-frame redundancy, we put the reconstruction under an iterative reweighted optimization, with non-uniform weight computed from temporal-correlated frame sequences. The proposed approach can achieve dynamic imaging at 16fps with 64×64-pixel resolution.

Scattering robust 3D reconstruction via polarized transient imaging.

Reconstructing 3D structure of scenes in the scattering medium is a challenging task with great research value. Existing techniques often impose strong assumptions on the scattering behaviors and are of limited performance. Recently, a low-cost transient imaging system has provided a feasible way to resolve the scene depth, by detecting the reflection instant on the time profile of a surface point. However, in cases with scattering medium, the rays are both reflected and scattered during transmission, and the depth calculated from the time profile largely deviates from the true value. To handle this problem, we used the different polarization behaviors of the reflection and scattering components, and introduced active polarization to separate the reflection component to estimate the scattering robust depth. Our experiments have demonstrated that our approach can accurately reconstruct the 3D structure underlying the scattering medium.

Illumination estimation from specular highlight in a multi-spectral image.

The reflection spectrum of an object characterizes its surface material, but for non-Lambertian scenes, the recorded spectrum often deviates owing to specular contamination. To compensate for this deviation, the illumination spectrum is required, and it can be estimated from specularity. However, existing illumination-estimation methods often degenerate in challenging cases, especially when only weak specularity exists. By adopting the dichromatic reflection model, which formulates a specular-influenced image as a linear combination of diffuse and specular components, this paper explores two individual priors and one mutual prior upon these two components: (i) The chromaticity of a specular component is identical over all the pixels. (ii) The diffuse component of a specular-contaminated pixel can be reconstructed using its specular-free counterpart describing the same material. (iii) The spectrum of illumination usually has low correlation with that of diffuse reflection. A general optimization framework is proposed to estimate the illumination spectrum from the specular component robustly and accurately. The results of both simulation and real experiments demonstrate the robustness and accuracy of our method.