Prior-free 3D tracking of a fast-moving object at 6667 frames per second with single-pixel detectors.

Real-time tracking and 3D trajectory computation of fast-moving objects is a promising technology, especially in the field of autonomous driving. However, existing image-based tracking methods face significant challenges when it comes to real-time tracking, primarily due to the limitation of storage space and computational resources. Here, we propose a novel approach that enables real-time 3D tracking of a fast-moving object without any prior motion information and at a very low computational cost. To enable 3D coordinate synthesis with a space-efficient optical setup, geometric moment patterns are projected on two non-orthogonal planes with a spatial resolution of 125 µm. Our experiment demonstrates an impressive tracking speed of 6667 frames per second (FPS) with a 20 kHz digital micromirror device (DMD), which is more than 200 times faster than the widely adopted video-based tracking methods. To the best of our knowledge, this is the highest tracking speed record in the field of single-pixel 3D trajectory tracking. This method promotes the development of real-time tracking techniques with single-pixel imaging (SPI).

Light-field photography using differential high-speed aperture coding.

Programmable aperture light-field photography enables the acquisition of angular information without compromising spatial resolution. However, direct current (DC) background noise is unavoidable in images recorded by programmable aperture light-field photography, leading to reducing the contrast of reconstructed images. In addition, it requires sacrificing temporal resolution to obtain angular information, making it a challenge to capture dynamic scenes. In this paper, we propose programmable aperture light-field photography using differential high-speed aperture coding. This method effectively reduces DC noise and produces high-contrast refocused images. Furthermore, we build a light-field camera based on a 1250 Hz spatial light modulator and a 1250 fps high-speed camera, achieving dynamic light-field photography at 1110(H)×800(V) resolution and 24 fps. Our results demonstrate significant improvements in image contrast and exhibit considerable promise for diverse applications.

Single-shot differential phase contrast microscopy using ring-shaped polarisation multiplexing illumination.

We propose a differential phase contrast microscopy that enables single-shot phase imaging for unstained biological samples. The proposed approach employs a ring-shaped LED array for polarisation multiplexing illumination and a polarisation camera for image acquisition. As such, multiple images of different polarisation angles can be simultaneously captured with a single shot. Through polarisation demultiplexing, the sample phase can therefore be recovered from the single-shot measurement. Both simulations and experiments demonstrate the effectiveness of the approach. We also demonstrate that ring-shaped illumination enables higher contrast and lower-distortion imaging results than disk-shaped illumination does. The proposed single-shot approach potentially enables phase contrast imaging for live cell samples in vitro. Lay Description: We propose a microscopy that enables imaging of transparent samples, unstained cells, etc. We demonstrate that the proposed method enables higher contrast and lower-distortion imaging results than conventional methods, and significantly improves imaging efficiency. The proposed method potentially enables dynamic imaging for live cell samples in vitro.

Adaptive real-time single-pixel imaging.

For most imaging systems, there is a trade-off between spatial resolution, temporal resolution, and signal-to-noise ratio. Such a trade-off is particularly severe in single-pixel imaging systems, given the limited throughput of the only one available pixel. Here we report a real-time single-pixel imaging method that can adaptively balance the spatial resolution, temporal resolution, and signal-to-noise ratio of the imaging system according to the changes in the target scene. When scene changes are detected, the dynamic imaging mode will be activated. The temporal resolution will be given high priority and real-time single-pixel imaging will be conducted at a video frame rate (30 frames/s) to visualize the object motion. When no scene changes are detected, the static imaging mode will be activated. The spatial resolution and the signal-to-noise ratio will be progressively built up to resolve fine structures and to improve image quality. The proposed method not only adds practicability to single-pixel imaging, but also generates a new, to the best of our knowledge, insight in data redundancy reduction and information capacity improvement for other computational imaging schemes.

Autofocus Fourier single-pixel microscopy.

Single-pixel microscopy enables observation of micro samples in invisible wave bands. Finding the focus position is essential to capture a clear image of a sample but could be difficult for single-pixel microscopy particularly in invisible wave bands. It is because the structured patterns projected onto the sample would be invisible and searching for the focus position manually could be exhausting. Here, we report an autofocus method for Fourier single-pixel microscopy. The reported method allows one to find the focus position without recording or reconstructing a complete image. The focus position is determined by the magnitude summation of a small number of Fourier coefficients, which enables fast autofocus. The reported method is experimentally demonstrated in imaging various objects in both visible and near-infrared wave bands. The method adds practicability to a single-pixel microscopy.

Contrast-enhanced, single-shot LED array microscopy based on Fourier ptychographic algorithm and deep learning.

LED array microscopes have the advantages of miniaturisation and low cost. It has been demonstrated that LED array microscopes outperform Köhler illumination microscopes in some applications. A LED array allows for a large numerical aperture of illumination. The larger numerical aperture of illumination brings the higher spatial resolution, but the lower image contrast as well. Therefore, there is a tradeoff between resolution and contrast for LED array microscopes. The Fourier ptychographic algorithm can overcome this tradeoff by increasing image contrast without sacrificing spatial resolution. However, the Fourier ptychographic algorithm requires acquisition of multiple images, which is time-consuming and results in live sample imaging challenging. To solve this problem, we develop contrast-enhanced, single-shot LED array microscopy based on the Fourier ptychographic algorithm and deep learning. The sample to be imaged is under illumination by all LEDs of the array simultaneously. The image captured is fed to several trained convolutional neural networks to generate the same number of images that are required by the Fourier ptychographic algorithm. We experimentally present that the image contrast of the final reconstruction is remarkably improved in comparison with the image captured. The proposed method can also produce chromatic-aberration-free results, even when an objective without aberration correction is used. We believe the method might provide live sample imaging with a low-cost approach.

Image-free active autofocusing with dual modulation and its application to Fourier single-pixel imaging.

Autofocusing is widely used in applications where sharp image acquisition or projection is needed. Here we report an active autofocusing method for sharp image projection. The method works with wide-field structured illumination and single-pixel detection. To find the focus position, the method illuminates the target object with a set of 3-step phase-shifting Fourier basis patterns repeatedly and collects the backscattered light by using a single-pixel detector through a grating. Dual modulation-dynamic modulation by the time-varying structured illumination and static modulation by the grating-embeds the depth information for the target object in the resulting single-pixel measurements. As such, the focus position can be determined by recovering the Fourier coefficients from the single-pixel measurements and searching for the coefficient with the maximum magnitude. High-speed spatial light modulation not only enables rapid autofocusing but also makes the method work even when the lens system is in continuous motion or the focal length of the lens is continuously adjusted. We experimentally validate the reported method in a self-built digital projector and demonstrate the application of the method in Fourier single-pixel imaging.

Remote referencing strategy for high-resolution coded ptychographic imaging.

The applications of conventional ptychography are limited by its relatively low resolution and throughput in the visible light regime. The new development of coded ptychography (CP) has addressed these issues and achieved the highest numerical aperture for large-area optical imaging in a lensless configuration. A high-quality reconstruction of CP relies on precise tracking of the coded sensor's positional shifts. The coded layer on the sensor, however, prevents the use of cross correlation analysis for motion tracking. Here we derive and analyze the motion tracking model of CP. A novel, to the best of our knowledge, remote referencing scheme and its subsequent refinement pipeline are developed for blind image acquisition. By using this approach, we can suppress the correlation peak caused by the coded surface and recover the positional shifts with deep sub-pixel accuracy. In contrast with common positional refinement methods, the reported approach can be disentangled from the iterative phase retrieval process and is computationally efficient. It allows blind image acquisition without motion feedback from the scanning process. It also provides a robust and reliable solution for implementing ptychography with high imaging throughput. We validate this approach by performing high-resolution whole slide imaging of bio-specimens.

Full-resolution, full-field-of-view, and high-quality fast Fourier single-pixel imaging.

Fourier single-pixel imaging (FSI) uses Fourier basis patterns for spatial light modulation to acquire the Fourier spectrum of the object image. The object image can be reconstructed via an inverse Fourier transform. However, the Fourier basis patterns are inherently gray scale, which results in the difficulty that the patterns can hardly be generated at a high speed by using a commonly used spatial light modulator-digital micromirrors device. To tackle this problem, fast FSI, which uses upsampled and dithered Fourier basis patterns to approximate the gray scale patterns, has been reported, but the achievable spatial resolution has to be sacrificed in the pattern upsampling process. Here we propose a method that can achieve not only full-resolution but also full-field-of-view and high-quality FSI. The key to the proposed method is to use a new, to the best of our knowledge, error diffusion dithering algorithm combined with two different scanning strategies to generate two sets of binarized Fourier basis patterns for spatial light modulation. As a result, two images with a sub-pixel shift from each other are reconstructed. It results in the final high-quality reconstruction by synthesizing the two images. We experimentally demonstrate the method can produce a high-quality 1024 × 768-pixel and full resolution image with a digital micromirror device with 1024 × 768 micromirrors.

Robust multi-angle structured illumination lensless microscopy via illumination angle calibration.

Multi-angle structured illumination lensless (MASIL) microscopy enables high-resolution image recovery over a large field of view. Successful image recovery of MASIL microscopy, however, relies on an accurate knowledge of the multi-angle illumination. System misalignments and slight deviations from the true illumination angle may result in image artifacts in reconstruction. Here we report a MASIL microscopy system that is robust against illumination misalignment. To calibrate the illumination angles, we design and use a double-sided mask, which is a glass wafer fabricated with a ring-array pattern on the upper surface and a disk-array pattern on the lower surface. As such, the illumination angles can be decoded from the captured images by estimating the relative displacement of the two patterns. We experimentally demonstrate that this system can achieve successful image recovery without any prior knowledge of the illumination angles. The reported approach provides a simple yet robust resolution for wide-field lensless microscopy. It can solve the LED array misalignment problem and calibrate angle-varied illumination for a variety of applications.

Ringing-free fast Fourier single-pixel imaging.

Fourier single-pixel imaging (FSI) allows an image to be reconstructed by acquiring the Fourier spectrum of the image using a single-pixel detector. Fast FSI is typically achieved by acquiring a truncated Fourier spectrum, that is, only low-frequency Fourier coefficients are acquired, with the high-frequency coefficients discarded. However, the truncation of the Fourier spectrum leads to undesirable ringing artifacts in the resulting image. Ringing artifacts produce false edges in the image and reduce the image contrast, resulting in image quality degeneration. The artifact is particularly severe in dynamic FSI, where the sampling ratio is generally ultra-low. We propose an effective and fast deringing algorithm to achieve ringing-free fast FSI. The algorithm eliminates ringing artifacts through 2D sub-pixel shifting and preserves image details through image fusion. Both static and dynamic imaging results demonstrate that the proposed method can reconstruct ringing-free images from under-sampled data in FSI. The deringing algorithm not only provides FSI with the capability of fast high-quality single-pixel imaging but also might prove its applicability in other areas, such as Fourier-based data compression algorithms.

Ptychographic sensor for large-scale lensless microbial monitoring with high spatiotemporal resolution.

Traditional microbial detection methods often rely on the overall property of microbial cultures and cannot resolve individual growth event at high spatiotemporal resolution. As a result, they require bacteria to grow to confluence and then interpret the results. Here, we demonstrate the application of an integrated ptychographic sensor for lensless cytometric analysis of microbial cultures over a large scale and with high spatiotemporal resolution. The reported device can be placed within a regular incubator or used as a standalone incubating unit for long-term microbial monitoring. For longitudinal study where massive data are acquired at sequential time points, we report a new temporal-similarity constraint to increase the temporal resolution of ptychographic reconstruction by 7-fold. With this strategy, the reported device achieves a centimeter-scale field of view, a half-pitch spatial resolution of 488 nm, and a temporal resolution of 15-s intervals. For the first time, we report the direct observation of bacterial growth in a 15-s interval by tracking the phase wraps of the recovered images, with high phase sensitivity like that in interferometric measurements. We also characterize cell growth via longitudinal dry mass measurement and perform rapid bacterial detection at low concentrations. For drug-screening application, we demonstrate proof-of-concept antibiotic susceptibility testing and perform single-cell analysis of antibiotic-induced filamentation. The combination of high phase sensitivity, high spatiotemporal resolution, and large field of view is unique among existing microscopy techniques. As a quantitative and miniaturized platform, it can improve studies with microorganisms and other biospecimens at resource-limited settings.

Quantitative multi-height phase retrieval via a coded image sensor.

Multi-height phase retrieval introduces different object-to-detector distances for obtaining phase diversity measurements. In the acquisition process, the slow-varying phase information, however, cannot be converted to intensity variations for detection. Therefore, the low-frequency contents of the phase profile are lost during acquisition and cannot be properly restored via phase retrieval. Here, we demonstrate the use of a coded image sensor for addressing this challenge in multi-height phase retrieval. In our scheme, we add a coded layer on top of the image sensor for encoding the slow-varying complex wavefronts into intensity variations of the modulated patterns. Inspired by the concept of blind ptychography, we report a reconstruction scheme to jointly recover the complex object and the unknown coded layer using multi-height measurements. With both simulation and experimental results, we show that the recovered phase is quantitative and the slow-varying phase profiles can be properly restored using lensless multi-height measurements. We also show that the image quality using the coded sensor is better than that of a regular image sensor. For demonstrations, we validate the reported scheme with various biospecimens and compare the results to those of regular lensless multi-height phase retrieval. The use of a coded image sensor may enable true quantitative phase imaging for the lensless multi-height, multi-wavelength, and transport-of-intensity equation approaches.

Deep learning-enabled whole slide imaging (DeepWSI): oil-immersion quality using dry objectives, longer depth of field, higher system throughput, and better functionality.

Whole slide imaging (WSI) has moved the traditional manual slide inspection process to the era of digital pathology. A typical WSI system translates the sample to different positions and captures images using a high numerical aperture (NA) objective lens. Performing oil-immersion microscopy is a major obstacle for WSI as it requires careful liquid handling during the scanning process. Switching between dry objective and oil-immersion lens is often impossible as it disrupts the acquisition process. For a high-NA objective lens, the sub-micron depth of field also poses a challenge to acquiring in-focus images of samples with uneven topography. Additionally, it implies a small field of view for each tile, thus limiting the system throughput and resulting in a long acquisition time. Here we report a deep learning-enabled WSI platform, termed DeepWSI, to substantially improve the system performance and imaging throughput. With this platform, we show that images captured with a regular dry objective lens can be transformed into images comparable to that of a 1.4-NA oil immersion lens. Blurred images with defocus distance from -5 µm to +5 µm can be virtually refocused to the in-focus plane post measurement. We demonstrate an equivalent data throughput of >2 gigapixels per second, the highest among existing WSI systems. Using the same deep neural network, we also report a high-resolution virtual staining strategy and demonstrate it for Fourier ptychographic WSI. The DeepWSI platform may provide a turnkey solution for developing high-performance diagnostic tools for digital pathology.

Optofluidic ptychography on a chip.

We report the implementation of a fully on-chip, lensless microscopy technique termed optofluidic ptychography. This imaging modality complements the miniaturization provided by microfluidics and allows the integration of ptychographic microscopy into various lab-on-a-chip devices. In our prototype, we place a microfluidic channel on the top surface of a coverslip and coat the bottom surface with a scattering layer. The channel and the coated coverslip substrate are then placed on top of an image sensor for diffraction data acquisition. Similar to the operation of a flow cytometer, the device utilizes microfluidic flow to deliver specimens across the channel. The diffracted light from the flowing objects is modulated by the scattering layer and recorded by the image sensor for ptychographic reconstruction, where high-resolution quantitative complex images are recovered from the diffraction measurements. By using an image sensor with a 1.85 µm pixel size, our device can resolve the 550 nm linewidth on the resolution target. We validate the device by imaging different types of biospecimens, including C. elegans, yeast cells, paramecium, and closterium sp. We also demonstrate a high-resolution ptychographic reconstruction at a video framerate of 30 frames per second. The reported technique can address a wide range of biomedical needs and engenders new ptychographic imaging innovations in a flow cytometer configuration.

High-throughput lensless whole slide imaging via continuous height-varying modulation of a tilted sensor.

We report a new, to the best of our knowledge, lensless microscopy configuration by integrating the concepts of transverse translational ptychography and defocus multi-height phase retrieval. In this approach, we place a tilted image sensor under the specimen for introducing linearly increasing phase modulation along one lateral direction. Similar to the operation of ptychography, we laterally translate the specimen and acquire the diffraction images for reconstruction. Since the axial distance between the specimen and the sensor varies at different lateral positions, laterally translating the specimen effectively introduces defocus multi-height measurements while eliminating axial scanning. Lateral translation further introduces sub-pixel shift for pixel super-resolution imaging and naturally expands the field of view for rapid whole slide imaging. We show that the equivalent height variation can be precisely estimated from the lateral shift of the specimen, thereby addressing the challenge of precise axial positioning in conventional multi-height phase retrieval. Using a sensor with 1.67 µm pixel size, our low-cost and field-portable prototype can resolve the 690 nm linewidth on the resolution target. We show that a whole slide image of a blood smear with a 120mm2 field of view can be acquired in 18 s. We also demonstrate accurate automatic white blood cell counting from the recovered image. The reported approach may provide a turnkey solution for addressing point-of-care and telemedicine-related challenges.

Mask-modulated lensless imaging via translated structured illumination.

Lensless microscopy technique enables high-resolution image recovery over a large field of view. By integrating the concept of phase retrieval, it can also retrieve the lost phase information from intensity-only measurements. Here we report a mask-modulated lensless imaging platform based on translated structured illumination. In the reported platform, we sandwich the object in-between a coded mask and a naked image sensor for lensless data acquisition. An LED array is used to provide angle-varied illumination for projecting a translated structured pattern without involving mechanical scanning. For different LED elements, we acquire the lensless intensity data for recovering the complex-valued object. In the reconstruction process, we employ the regularized ptychographic iterative engine and implement an up-sampling process in the reciprocal space. As demonstrated by experimental results, the reported platform is able to recover complex-valued object images with higher resolution and better quality than previous implementations. Our approach may provide a cost-effective solution for high-resolution and wide field-of-view ptychographic imaging without involving mechanical scanning.

Image-free real-time 3-D tracking of a fast-moving object using dual-pixel detection.

Real-time 3-D tracking of a fast-moving object has found important applications in industry, traffic control, sports, biomedicine, defense, etc. However, it is difficult to adopt typical image-based object tracking systems in a fast-moving object tracking in real time and for a long duration, because reliable and robust image processing and analysis algorithms are often computationally exhausted, and limited storage and bandwidth can hardly fulfill the great demand of high-speed photography. Here we report an image-free 3-D tracking approach. The approach uses only two single-pixel detectors and a high-speed spatial light modulator for data acquisition. By illuminating the target moving object with six single-period Fourier basis patterns, the approach is able to analytically calculate the position of the object with the corresponding single-pixel measurements. The approach is low-cost, and data- and computation-efficient. We experimentally demonstrate that the proposed approach can detect and track a fast-moving object at a frame rate of 1666 frames per second by using a 10,000 Hz digital micromirror device. Benefiting from the wide working spectrum of single-pixel detectors, the reported approach might be applicable for hidden fast-moving object tracking.

Automatic image annotation method based on a convolutional neural network with threshold optimization.

In this study, a convolutional neural network with threshold optimization (CNN-THOP) is proposed to solve the issue of overlabeling or downlabeling arising during the multilabel image annotation process in the use of a ranking function for label annotation along with prediction probability. This model fuses the threshold optimization algorithm to the CNN structure. First, an optimal model trained by the CNN is used to predict the test set images, and batch normalization (BN) is added to the CNN structure to effectively accelerate the convergence speed and obtain a group of prediction probabilities. Second, threshold optimization is performed on the obtained prediction probability to derive an optimal threshold for each class of labels to form a group of optimal thresholds. When the prediction probability for this class of labels is greater than or equal to the corresponding optimal threshold, this class of labels is used as the annotation result for the image. During the annotation process, the multilabel annotation for the image to be annotated is realized by loading the optimal model and the optimal threshold. Verification experiments are performed on the MIML, COREL5K, and MSRC datasets. Compared with the MBRM, the CNN-THOP increases the average precision on MIML, COREL5K, and MSRC by 27%, 28% and 33%, respectively. Compared with the E2E-DCNN, the CNN-THOP increases the average recall rate by 3% on both COREL5K and MSRC. The most precise annotation effect for CNN-THOP is observed on the MIML dataset, with a complete matching degree reaching 64.8%.

Autofocusing technologies for whole slide imaging and automated microscopy.

Whole slide imaging (WSI) has moved digital pathology closer to diagnostic practice in recent years. Due to the inherent tissue topography variability, accurate autofocusing remains a critical challenge for WSI and automated microscopy systems. The traditional focus map surveying method is limited in its ability to acquire a high degree of focus points while still maintaining high throughput. Real-time approaches decouple image acquisition from focusing, thus allowing for rapid scanning while maintaining continuous accurate focus. This work reviews the traditional focus map approach and discusses the choice of focus measure for focal plane determination. It also discusses various real-time autofocusing approaches including reflective-based triangulation, confocal pinhole detection, low-coherence interferometry, tilted sensor approach, independent dual sensor scanning, beam splitter array, phase detection, dual-LED illumination and deep-learning approaches. The technical concepts, merits and limitations of these methods are explained and compared to those of a traditional WSI system. This review may provide new insights for the development of high-throughput automated microscopy imaging systems that can be made broadly available and utilizable without loss of capacity.