Adaptive real-time single-pixel imaging.

For most imaging systems, there is a trade-off between spatial resolution, temporal resolution, and signal-to-noise ratio. Such a trade-off is particularly severe in single-pixel imaging systems, given the limited throughput of the only one available pixel. Here we report a real-time single-pixel imaging method that can adaptively balance the spatial resolution, temporal resolution, and signal-to-noise ratio of the imaging system according to the changes in the target scene. When scene changes are detected, the dynamic imaging mode will be activated. The temporal resolution will be given high priority and real-time single-pixel imaging will be conducted at a video frame rate (30 frames/s) to visualize the object motion. When no scene changes are detected, the static imaging mode will be activated. The spatial resolution and the signal-to-noise ratio will be progressively built up to resolve fine structures and to improve image quality. The proposed method not only adds practicability to single-pixel imaging, but also generates a new, to the best of our knowledge, insight in data redundancy reduction and information capacity improvement for other computational imaging schemes.

Prior-free 3D tracking of a fast-moving object at 6667 frames per second with single-pixel detectors.

Real-time tracking and 3D trajectory computation of fast-moving objects is a promising technology, especially in the field of autonomous driving. However, existing image-based tracking methods face significant challenges when it comes to real-time tracking, primarily due to the limitation of storage space and computational resources. Here, we propose a novel approach that enables real-time 3D tracking of a fast-moving object without any prior motion information and at a very low computational cost. To enable 3D coordinate synthesis with a space-efficient optical setup, geometric moment patterns are projected on two non-orthogonal planes with a spatial resolution of 125 µm. Our experiment demonstrates an impressive tracking speed of 6667 frames per second (FPS) with a 20 kHz digital micromirror device (DMD), which is more than 200 times faster than the widely adopted video-based tracking methods. To the best of our knowledge, this is the highest tracking speed record in the field of single-pixel 3D trajectory tracking. This method promotes the development of real-time tracking techniques with single-pixel imaging (SPI).

Single-shot differential phase contrast microscopy using ring-shaped polarisation multiplexing illumination.

We propose a differential phase contrast microscopy that enables single-shot phase imaging for unstained biological samples. The proposed approach employs a ring-shaped LED array for polarisation multiplexing illumination and a polarisation camera for image acquisition. As such, multiple images of different polarisation angles can be simultaneously captured with a single shot. Through polarisation demultiplexing, the sample phase can therefore be recovered from the single-shot measurement. Both simulations and experiments demonstrate the effectiveness of the approach. We also demonstrate that ring-shaped illumination enables higher contrast and lower-distortion imaging results than disk-shaped illumination does. The proposed single-shot approach potentially enables phase contrast imaging for live cell samples in vitro. Lay Description: We propose a microscopy that enables imaging of transparent samples, unstained cells, etc. We demonstrate that the proposed method enables higher contrast and lower-distortion imaging results than conventional methods, and significantly improves imaging efficiency. The proposed method potentially enables dynamic imaging for live cell samples in vitro.

Light-field photography using differential high-speed aperture coding.

Programmable aperture light-field photography enables the acquisition of angular information without compromising spatial resolution. However, direct current (DC) background noise is unavoidable in images recorded by programmable aperture light-field photography, leading to reducing the contrast of reconstructed images. In addition, it requires sacrificing temporal resolution to obtain angular information, making it a challenge to capture dynamic scenes. In this paper, we propose programmable aperture light-field photography using differential high-speed aperture coding. This method effectively reduces DC noise and produces high-contrast refocused images. Furthermore, we build a light-field camera based on a 1250 Hz spatial light modulator and a 1250 fps high-speed camera, achieving dynamic light-field photography at 1110(H)×800(V) resolution and 24 fps. Our results demonstrate significant improvements in image contrast and exhibit considerable promise for diverse applications.

Autofocus Fourier single-pixel microscopy.

Single-pixel microscopy enables observation of micro samples in invisible wave bands. Finding the focus position is essential to capture a clear image of a sample but could be difficult for single-pixel microscopy particularly in invisible wave bands. It is because the structured patterns projected onto the sample would be invisible and searching for the focus position manually could be exhausting. Here, we report an autofocus method for Fourier single-pixel microscopy. The reported method allows one to find the focus position without recording or reconstructing a complete image. The focus position is determined by the magnitude summation of a small number of Fourier coefficients, which enables fast autofocus. The reported method is experimentally demonstrated in imaging various objects in both visible and near-infrared wave bands. The method adds practicability to a single-pixel microscopy.

Full-resolution, full-field-of-view, and high-quality fast Fourier single-pixel imaging.

Fourier single-pixel imaging (FSI) uses Fourier basis patterns for spatial light modulation to acquire the Fourier spectrum of the object image. The object image can be reconstructed via an inverse Fourier transform. However, the Fourier basis patterns are inherently gray scale, which results in the difficulty that the patterns can hardly be generated at a high speed by using a commonly used spatial light modulator-digital micromirrors device. To tackle this problem, fast FSI, which uses upsampled and dithered Fourier basis patterns to approximate the gray scale patterns, has been reported, but the achievable spatial resolution has to be sacrificed in the pattern upsampling process. Here we propose a method that can achieve not only full-resolution but also full-field-of-view and high-quality FSI. The key to the proposed method is to use a new, to the best of our knowledge, error diffusion dithering algorithm combined with two different scanning strategies to generate two sets of binarized Fourier basis patterns for spatial light modulation. As a result, two images with a sub-pixel shift from each other are reconstructed. It results in the final high-quality reconstruction by synthesizing the two images. We experimentally demonstrate the method can produce a high-quality 1024 × 768-pixel and full resolution image with a digital micromirror device with 1024 × 768 micromirrors.

Remote referencing strategy for high-resolution coded ptychographic imaging.

The applications of conventional ptychography are limited by its relatively low resolution and throughput in the visible light regime. The new development of coded ptychography (CP) has addressed these issues and achieved the highest numerical aperture for large-area optical imaging in a lensless configuration. A high-quality reconstruction of CP relies on precise tracking of the coded sensor's positional shifts. The coded layer on the sensor, however, prevents the use of cross correlation analysis for motion tracking. Here we derive and analyze the motion tracking model of CP. A novel, to the best of our knowledge, remote referencing scheme and its subsequent refinement pipeline are developed for blind image acquisition. By using this approach, we can suppress the correlation peak caused by the coded surface and recover the positional shifts with deep sub-pixel accuracy. In contrast with common positional refinement methods, the reported approach can be disentangled from the iterative phase retrieval process and is computationally efficient. It allows blind image acquisition without motion feedback from the scanning process. It also provides a robust and reliable solution for implementing ptychography with high imaging throughput. We validate this approach by performing high-resolution whole slide imaging of bio-specimens.

Image-free active autofocusing with dual modulation and its application to Fourier single-pixel imaging.

Autofocusing is widely used in applications where sharp image acquisition or projection is needed. Here we report an active autofocusing method for sharp image projection. The method works with wide-field structured illumination and single-pixel detection. To find the focus position, the method illuminates the target object with a set of 3-step phase-shifting Fourier basis patterns repeatedly and collects the backscattered light by using a single-pixel detector through a grating. Dual modulation-dynamic modulation by the time-varying structured illumination and static modulation by the grating-embeds the depth information for the target object in the resulting single-pixel measurements. As such, the focus position can be determined by recovering the Fourier coefficients from the single-pixel measurements and searching for the coefficient with the maximum magnitude. High-speed spatial light modulation not only enables rapid autofocusing but also makes the method work even when the lens system is in continuous motion or the focal length of the lens is continuously adjusted. We experimentally validate the reported method in a self-built digital projector and demonstrate the application of the method in Fourier single-pixel imaging.

Contrast-enhanced, single-shot LED array microscopy based on Fourier ptychographic algorithm and deep learning.

LED array microscopes have the advantages of miniaturisation and low cost. It has been demonstrated that LED array microscopes outperform Köhler illumination microscopes in some applications. A LED array allows for a large numerical aperture of illumination. The larger numerical aperture of illumination brings the higher spatial resolution, but the lower image contrast as well. Therefore, there is a tradeoff between resolution and contrast for LED array microscopes. The Fourier ptychographic algorithm can overcome this tradeoff by increasing image contrast without sacrificing spatial resolution. However, the Fourier ptychographic algorithm requires acquisition of multiple images, which is time-consuming and results in live sample imaging challenging. To solve this problem, we develop contrast-enhanced, single-shot LED array microscopy based on the Fourier ptychographic algorithm and deep learning. The sample to be imaged is under illumination by all LEDs of the array simultaneously. The image captured is fed to several trained convolutional neural networks to generate the same number of images that are required by the Fourier ptychographic algorithm. We experimentally present that the image contrast of the final reconstruction is remarkably improved in comparison with the image captured. The proposed method can also produce chromatic-aberration-free results, even when an objective without aberration correction is used. We believe the method might provide live sample imaging with a low-cost approach.

Ringing-free fast Fourier single-pixel imaging.

Fourier single-pixel imaging (FSI) allows an image to be reconstructed by acquiring the Fourier spectrum of the image using a single-pixel detector. Fast FSI is typically achieved by acquiring a truncated Fourier spectrum, that is, only low-frequency Fourier coefficients are acquired, with the high-frequency coefficients discarded. However, the truncation of the Fourier spectrum leads to undesirable ringing artifacts in the resulting image. Ringing artifacts produce false edges in the image and reduce the image contrast, resulting in image quality degeneration. The artifact is particularly severe in dynamic FSI, where the sampling ratio is generally ultra-low. We propose an effective and fast deringing algorithm to achieve ringing-free fast FSI. The algorithm eliminates ringing artifacts through 2D sub-pixel shifting and preserves image details through image fusion. Both static and dynamic imaging results demonstrate that the proposed method can reconstruct ringing-free images from under-sampled data in FSI. The deringing algorithm not only provides FSI with the capability of fast high-quality single-pixel imaging but also might prove its applicability in other areas, such as Fourier-based data compression algorithms.

Robust multi-angle structured illumination lensless microscopy via illumination angle calibration.

Multi-angle structured illumination lensless (MASIL) microscopy enables high-resolution image recovery over a large field of view. Successful image recovery of MASIL microscopy, however, relies on an accurate knowledge of the multi-angle illumination. System misalignments and slight deviations from the true illumination angle may result in image artifacts in reconstruction. Here we report a MASIL microscopy system that is robust against illumination misalignment. To calibrate the illumination angles, we design and use a double-sided mask, which is a glass wafer fabricated with a ring-array pattern on the upper surface and a disk-array pattern on the lower surface. As such, the illumination angles can be decoded from the captured images by estimating the relative displacement of the two patterns. We experimentally demonstrate that this system can achieve successful image recovery without any prior knowledge of the illumination angles. The reported approach provides a simple yet robust resolution for wide-field lensless microscopy. It can solve the LED array misalignment problem and calibrate angle-varied illumination for a variety of applications.

Mask-modulated lensless imaging via translated structured illumination.

Lensless microscopy technique enables high-resolution image recovery over a large field of view. By integrating the concept of phase retrieval, it can also retrieve the lost phase information from intensity-only measurements. Here we report a mask-modulated lensless imaging platform based on translated structured illumination. In the reported platform, we sandwich the object in-between a coded mask and a naked image sensor for lensless data acquisition. An LED array is used to provide angle-varied illumination for projecting a translated structured pattern without involving mechanical scanning. For different LED elements, we acquire the lensless intensity data for recovering the complex-valued object. In the reconstruction process, we employ the regularized ptychographic iterative engine and implement an up-sampling process in the reciprocal space. As demonstrated by experimental results, the reported platform is able to recover complex-valued object images with higher resolution and better quality than previous implementations. Our approach may provide a cost-effective solution for high-resolution and wide field-of-view ptychographic imaging without involving mechanical scanning.

Deep learning-enabled whole slide imaging (DeepWSI): oil-immersion quality using dry objectives, longer depth of field, higher system throughput, and better functionality.

Whole slide imaging (WSI) has moved the traditional manual slide inspection process to the era of digital pathology. A typical WSI system translates the sample to different positions and captures images using a high numerical aperture (NA) objective lens. Performing oil-immersion microscopy is a major obstacle for WSI as it requires careful liquid handling during the scanning process. Switching between dry objective and oil-immersion lens is often impossible as it disrupts the acquisition process. For a high-NA objective lens, the sub-micron depth of field also poses a challenge to acquiring in-focus images of samples with uneven topography. Additionally, it implies a small field of view for each tile, thus limiting the system throughput and resulting in a long acquisition time. Here we report a deep learning-enabled WSI platform, termed DeepWSI, to substantially improve the system performance and imaging throughput. With this platform, we show that images captured with a regular dry objective lens can be transformed into images comparable to that of a 1.4-NA oil immersion lens. Blurred images with defocus distance from -5 µm to +5 µm can be virtually refocused to the in-focus plane post measurement. We demonstrate an equivalent data throughput of >2 gigapixels per second, the highest among existing WSI systems. Using the same deep neural network, we also report a high-resolution virtual staining strategy and demonstrate it for Fourier ptychographic WSI. The DeepWSI platform may provide a turnkey solution for developing high-performance diagnostic tools for digital pathology.

High-throughput lensless whole slide imaging via continuous height-varying modulation of a tilted sensor.

We report a new, to the best of our knowledge, lensless microscopy configuration by integrating the concepts of transverse translational ptychography and defocus multi-height phase retrieval. In this approach, we place a tilted image sensor under the specimen for introducing linearly increasing phase modulation along one lateral direction. Similar to the operation of ptychography, we laterally translate the specimen and acquire the diffraction images for reconstruction. Since the axial distance between the specimen and the sensor varies at different lateral positions, laterally translating the specimen effectively introduces defocus multi-height measurements while eliminating axial scanning. Lateral translation further introduces sub-pixel shift for pixel super-resolution imaging and naturally expands the field of view for rapid whole slide imaging. We show that the equivalent height variation can be precisely estimated from the lateral shift of the specimen, thereby addressing the challenge of precise axial positioning in conventional multi-height phase retrieval. Using a sensor with 1.67 µm pixel size, our low-cost and field-portable prototype can resolve the 690 nm linewidth on the resolution target. We show that a whole slide image of a blood smear with a 120mm2 field of view can be acquired in 18 s. We also demonstrate accurate automatic white blood cell counting from the recovered image. The reported approach may provide a turnkey solution for addressing point-of-care and telemedicine-related challenges.

Image-free classification of fast-moving objects using "learned" structured illumination and single-pixel detection.

Object classification generally relies on image acquisition and subsequent analysis. Real-time classification of fast-moving objects is a challenging task. Here we propose an approach for real-time classification of fast-moving objects without image acquisition. The key to the approach is to use structured illumination and single-pixel detection to acquire the object features directly. A convolutional neural network (CNN) is trained to learn the object features. The "learned" object features are then used as structured patterns for structured illumination. Object classification can be achieved by picking up the resulting light signals by a single-pixel detector and feeding the single-pixel measurements to the trained CNN. In our experiments, we show that accurate and real-time classification of fast-moving objects can be achieved. Potential applications of the proposed approach include rapid classification of flowing cells, assembly-line inspection, and aircraft classification in defense applications. Benefiting from the use of a single-pixel detector, the approach might be applicable for hidden moving object classification.

Image-free real-time 3-D tracking of a fast-moving object using dual-pixel detection.

Real-time 3-D tracking of a fast-moving object has found important applications in industry, traffic control, sports, biomedicine, defense, etc. However, it is difficult to adopt typical image-based object tracking systems in a fast-moving object tracking in real time and for a long duration, because reliable and robust image processing and analysis algorithms are often computationally exhausted, and limited storage and bandwidth can hardly fulfill the great demand of high-speed photography. Here we report an image-free 3-D tracking approach. The approach uses only two single-pixel detectors and a high-speed spatial light modulator for data acquisition. By illuminating the target moving object with six single-period Fourier basis patterns, the approach is able to analytically calculate the position of the object with the corresponding single-pixel measurements. The approach is low-cost, and data- and computation-efficient. We experimentally demonstrate that the proposed approach can detect and track a fast-moving object at a frame rate of 1666 frames per second by using a 10,000 Hz digital micromirror device. Benefiting from the wide working spectrum of single-pixel detectors, the reported approach might be applicable for hidden fast-moving object tracking.

Efficient full-color single-pixel imaging based on the human vision property-"giving in to the blues".

Single-pixel imaging is a novel, to the best of our knowledge, computational imaging scheme, but a large number of measurements are typically required in data acquisition. Full-color single-pixel imaging takes many more measurements than does monochromatic single-pixel imaging. Utilizing the fact that human eyes have a poorer spatial resolution to blues than reds and greens, we propose to sample the blue component of color images with an ultra-low sampling ratio so as to reduce the number of measurements. We demonstrate our method with simulations and experiments, concluding that 95% of the measurements can be reduced in the acquisition of the blue component of natural color images in the size of 256×256 pixels, and the resulting images are without remarkable visual loss. Moreover, utilizing the sparsity of natural images, the sampling ratios of the red and green components can be reduced to 15% and 50%, respectively. This Letter may generate a new insight of how to optimize the imaging efficiency by utilizing human vision properties. The proposed method can be adopted by other full-color computational imaging techniques.

Experimental observation of differential self-mixing interference signals using a randomly polarized laser: a differential self-mixing interferometry.

Differential measurement has strong anti-interference ability. We report the first, to the best of our knowledge, experimental demonstration of differential self-mixing interference signals using a randomly polarized laser for differential self-mixing interferometry (SMI). In the differential SMI system, the detection light can be divided into interference signals of $p$p-polarized and $s$s-polarized light with identical intensity and pi-shift phase difference. By exploiting such a new experimental phenomenon, we propose a noise-robust and low-cost self-mixing interferometer. Experiments show that the proposed approach can effectively suppress both periodic and aperiodic noise. Thus, the reported phenomenon and approach has a good application prospect in self-mixing interferometers.

Super-resolved multispectral lensless microscopy via angle-tilted, wavelength-multiplexed ptychographic modulation.

We report an angle-tilted, wavelength-multiplexed ptychographic modulation approach for multispectral lensless on-chip microscopy. In this approach, we illuminate the specimen with lights at five wavelengths simultaneously. A prism is added at the illumination path for spectral dispersion. Thus, lightwaves at different wavelengths hit the specimen at slightly different incident angles, breaking the ambiguities in mixed-state ptychographic reconstruction. At the detection path, we place a thin diffuser between the specimen and the monochromatic image sensor for encoding the spectral information into 2D intensity measurements. By scanning the sample to different x-y positions, we acquire a sequence of monochromatic images for reconstructing the five complex object profiles at the five wavelengths. An up-sampling procedure is integrated into the recovery process to bypass the resolution limit imposed by the imager pixel size. We demonstrate a half-pitch resolution of 0.55 µm using an image sensor with 1.85 µm pixel size. We also demonstrate quantitative and high-quality multispectral reconstructions of stained tissue sections for digital pathology applications.

Image-free real-time detection and tracking of fast moving object using a single-pixel detector.

Real-time detection and tracking for fast moving object has important applications in various fields. However, available methods, especially low-cost ones, can hardly achieve real-time and long-duration object detection and tracking. Here we report an image-free and cost-effective method for detecting and tracking a fast moving object in real time and for long duration. The method employs a spatial light modulator and a single-pixel detector for data acquisition. It uses Fourier basis patterns to illuminate the target moving object and collects the resulting light signal with a single-pixel detector. The proposed method is able to detect and track the object with the single-pixel measurements directly without image reconstruction. The detection and tracking algorithm of the proposed method is computationally efficient. We experimentally demonstrate that the method can achieve a temporal resolution of 1,666 frames per second by using a 10,000 Hz digital micro-mirror device. The latency time of the method is on the order of microseconds. Additionally, the method acquires only 600 bytes of data for each frame. The method therefore allows fast moving object detection and tracking in real time and for long duration. This image-free approach might open up a new avenue for spatial information acquisition in a highly efficient manner.