Low-power laser image enhancement via deep feature recovery for HDR 3D measurement.

Laser 3D measurement has gained widespread applications in industrial metrology . Still, it is usually limited by surfaces with high dynamic range (HDR) or the colorful surface texture of measured surfaces, such as metal and black industrial parts. Currently, conventional methods generally work with relatively strong-power laser intensities, which could potentially damage the sample or induce eye-safety concerns. For deep-learning-based methods, due to the different reflectivity of the measured surfaces, the HDR problem may require cumbersome adjustment of laser intensity in order to acquire enough training data. Even so, the problem of inaccurate ground truth may occur. To address these issues, this paper proposes the deep feature recovery (DFR) strategy to enhance low-light laser stripe images for achieving HDR 3D reconstruction with low cost, high robustness, and eye safety. To the best of our knowledge, this is the first attempt to tackle the challenge of high measurement costs associated with measuring HDR surfaces in laser 3D measurement. In learning the features of low-power laser images, the proposed strategy has a superior generalization ability and is insensitive to different low laser powers and variant surface reflectivity. To verify this point, we specially design the experiments by training the network merely using the diffusely reflective masks (DRM951) and testing the performance using diffusely reflective masks, metal surfaces, black industrial parts (contained in the constructed datasets DRO690, MO191, and BO107) and their hybrid scenes. Experimental results demonstrate that the proposed DFR strategy has good performances on robustness by testing different measurement scenes. For variously reflective surfaces, such as diffusely reflective surfaces, metal surfaces, and black parts surfaces, the reconstructed 3D shapes all have a similar quality to the reference method.

Physics-based supervised learning method for high dynamic range 3D measurement with high fidelity.

High dynamic range (HDR) 3D measurement is a meaningful but challenging problem. Recently, many deep-learning-based methods have been proposed for the HDR problem. However, due to learning redundant fringe intensity information, their networks are difficult to converge for data with complex surface reflectivity and various illumination conditions, resulting in non-robust performance. To address this problem, we propose a physics-based supervised learning method. By introducing the physical model for phase retrieval, we design a novel, to the best of our knowledge, sinusoidal-component-to-sinusoidal-component mapping paradigm. Consequently, the scale difference of fringe intensity in various illumination scenarios can be eliminated. Compared with conventional supervised-learning methods, our method can greatly promote the convergence of the network and the generalization ability, while compared with the recently proposed unsupervised-learning method, our method can recover complex surfaces with much more details. To better evaluate our method, we specially design the experiment by training the network merely using the metal objects and testing the performance using different diffuse sculptures, metal surfaces, and their hybrid scenes. Experiments for all the testing scenarios have high-quality phase recovery with an STD error of about 0.03 rad, which reveals the superior generalization ability for complex reflectivity and various illumination conditions. Furthermore, the zoom-in 3D plots of the sculpture verify its fidelity on recovering fine details.

Real-time 3D shape measurement of dynamic scenes using fringe projection profilometry: lightweight NAS-optimized dual frequency deep learning approach.

Achieving real-time and high-accuracy 3D reconstruction of dynamic scenes is a fundamental challenge in many fields, including online monitoring, augmented reality, and so on. On one hand, traditional methods, such as Fourier transform profilometry (FTP) and phase-shifting profilometry (PSP), are struggling to balance measuring efficiency and accuracy. On the other hand, deep learning-based approaches, which offer the potential for improved accuracy, are hindered by large parameter amounts and complex structures less amenable to real-time requirements. To solve this problem, we proposed a network architecture search (NAS)-based method for real-time processing and 3D measurement of dynamic scenes with rate equivalent to single-shot. A NAS-optimized lightweight neural network was designed for efficient phase demodulation, while an improved dual-frequency strategy was employed coordinately for flexible absolute phase unwrapping. The experiment results demonstrate that our method can effectively perform 3D reconstruction with a reconstruction speed of 58fps, and realize high-accuracy measurement of dynamic scenes based on deep learning for what we believe to be the first time with the average RMS error of about 0.08 mm.

Multi-dimensional information sensing of complex surfaces based on fringe projection profilometry.

Multi-dimensional and high-resolution information sensing of complex surface profiles is critical for investigating various structures and analyzing their mechanical properties. This information is currently accessed separately through different technologies and devices. Fringe projection profilometry (FPP) has been widely applied in shape measurement of complex surfaces. Since structured light information is projected instead of being attached onto the surface, it holds back accurately tracking corresponding points and fails to further analyze deformation and strain. To address this issue, we propose a multi-dimensional information sensing method based on digital image correction (DIC)-assisted FPP. Firstly, colorful fluorescent markers are introduced to produce modulated information with both high-intensity reflectivity and color difference. And then, the general information separation method is presented to simultaneously acquire speckle-free texture, fringe patterns and high-contrast speckle patterns for multi-dimensional information sensing. To the best of our knowledge, this proposed method, for the first time, simultaneously realizes accurate and high-resolution 2D texture (T), 4D shape (x, y, z, t) and analytical dimensional mechanical parameters (deformation (d), strain (s)) information sensing based on the FPP system. Experimental results demonstrate the proposed method can measure and analyze 3D geometry and mechanical state of complex surfaces, expanding the measuring dimension of the off-the-shelf FPP system without any extra hardware cost.

Learned digital lens enabled single optics achromatic imaging.

High-quality imaging with reduced optical complexity has been extensively investigated owing to its promising future in academic and industrial research. However, the practical performance of most imaging systems has encountered a bottleneck posed by optics rather than electronics. Here, we propose a digital lens (DL) to compensate for the chromatic aberration induced by physical optical elements, while the residual wavelength-independent degradation is tackled through a self-designed neural network. By transforming physical aberration correction to an algorithm-based computational imaging task, the proposed DL enables our framework to reduce optical complexity and achieve achromatic imaging in the analog domain. Real experiments have been conducted with an off-the-shelf single lens and recovered images show up to 14.62 dB higher peak signal-to-noise ratio (PSNR) than the original chromatic input. Furthermore, we run a comprehensive ablation study to highlight the contribution of embedding the proposed DL, which shows a 4.83 dB PSNR improvement compared with the methods without DL. Technically, the proposed method can be an alternative for future applications that require both simple optics and high-fidelity visualization.

High-efficiency and robust binary fringe optimization for superfast 3D shape measurement.

By utilizing 1-bit binary fringe patterns instead of conventional 8-bit sinusoidal patterns, binary defocusing techniques have been successfully applied for high-speed 3D shape measurement. However, simultaneously achieving high accuracy and high speed remains challenging. To overcome this limitation, we propose a high-efficiency and robust binary fringe optimization method for superfast 3D shape measurement, which consists of 1D optimization and 2D modulation. Specifically, for 1D optimization, the three-level OPWM technique is introduced for high-order harmonics elimination, and an optimization framework is presented for generating the 'best' three-level OPWM pattern especially for large fringe periods. For 2D modulation, a single-pattern three-level OPWM strategy is proposed by utilizing all the dimensions for intensity modulation to decrease the required projection patterns. Thus, the proposed method essentially belongs to the 2D modulation technique, yet iterative optimization is carried out along one dimension, which drastically improves the computational efficiency while ensuring high accuracy. With only one set of optimized patterns, both simulations and experiments demonstrate that high-quality phase maps can be consistently generated for a wide range of fringe periods (e.g., from 18 to 1140 pixels) and different amounts of defocusing, and it can achieve superfast and high-accuracy 3D shape measurement.

Time-overlapping structured-light projection: high performance on 3D shape measurement for complex dynamic scenes.

High-speed three-dimensional (3D) shape measurement has been continuously researched due to the demand for analyzing dynamic behavior in transient scenes. In this work, a time-overlapping structured-light 3D shape measuring technique is proposed to realize high-speed and high-performance measurement on complex dynamic scenes. Time-overlapping structured-light projection is presented to maximumly reduce the information redundancy in temporal sequences and improve the measuring efficiency; generalized tripartite phase unwrapping (Tri-PU) is used to ensure the measuring robustness; fringe period extension is achieved by improving overlapping rate to further double the encoding fringe periods for higher measuring accuracy. Based on the proposed measuring technique, one new pixel-to-pixel and unambiguous 3D reconstruction result can be updated with three newly required patterns at a reconstruction rate of 3174 fps. Three transient scenes including collapsing wood blocks struck by a flying arrow, free-falling foam snowflakes and flying water balloon towards metal grids were measured to verify the high performance of the proposed method in various complex dynamic scenes.

Large depth-of-field computational imaging with multi-spectral and dual-aperture optics.

Large DOF (depth-of-field) with high SNR (signal-noise-ratio) imaging is a crucial technique for applications from security monitoring to medical diagnostics. However, traditional optical design for large DOF requires a reduction in aperture size, and hence with a decrease in light throughput and SNR. In this paper, we report a computational imaging system integrating dual-aperture optics with a physics-informed dual-encoder neural network to realize prominent DOF extension. Boosted by human vision mechanism and optical imaging law, the dual-aperture imaging system is consisted of a small-aperture NIR camera to provide sharp edge and a large-aperture VIS camera to provide faithful color. To solve the imaging inverse problem in NIR-VIS fusion with different apertures, a specific network with parallel double encoders and the multi-scale fusion module is proposed to adaptively extract and learn the useful features, which contributes to preventing color deviation while preserving delicate scene textures. The proposed imaging framework is flexible and can be designed in different protos with varied optical elements for different applications. We provide theory for system design, demonstrate a prototype device, establish a real-scene dataset containing 3000 images, perform elaborate ablation studies and conduct peer comparative experiments. The experimental results demonstrate that our method effectively produces high-fidelity with larger DOF range than input raw images about 3 times. Without complex optical design and strict practical limitations, this novel, intelligent and integratable system is promising for variable vision applications such as smartphone photography, computational measurement, and medical imaging.

Single-shot hyperspectral imaging based on dual attention neural network with multi-modal learning.

Hyperspectral imaging is being extensively investigated owing to its promising future in critical applications such as medical diagnostics, sensing, and surveillance. However, current techniques are complex with multiple alignment-sensitive components and spatiospectral parameters predetermined by manufacturers. In this paper, we demonstrate an end-to-end snapshot hyperspectral imaging technique and build a physics-informed dual attention neural network with multimodal learning. By modeling the 3D spectral cube reconstruction procedure and solving that compressive-imaging inverse problem, the hyperspectral volume can be directly recovered from only one scene RGB image. Spectra features and camera spectral sensitivity are jointly leveraged to retrieve the multiplexed spatiospectral correlations and realize hyperspectral imaging. With the help of integrated attention mechanism, useful information supplied by disparate modal components is adaptively learned and aggregated to make our network flexible for variable imaging systems. Results show that the proposed method is ultra-faster than the traditional scanning method, and 3.4 times more precise than the existing hyperspectral imaging convolutional neural network. We provide theory for network design, demonstrate training process, and present experimental results with high accuracy. Without bulky benchtop setups and strict experimental limitations, this simple and effective method offers great potential for future spectral imaging applications such as pathological digital stain, computational imaging and virtual/augmented reality display, etc.

CVNet: confidence voting convolutional neural network for camera spectral sensitivity estimation.

Spectral sensitivity, as one of the most important parameters of a digital camera, is playing a key role in many computer vision applications. In this paper, a confidence voting convolutional neural network (CVNet) is proposed to rebuild the spectral sensitivity function, modeled as the sum of weighted basis functions. By evaluating useful information supplied by different image segments, disparate confidence is calculated to automatically learn basis functions' weights, only using one image captured by the object camera. Three types of basis functions are made up and employed in the network, including Fourier basis function (FBF), singular value decomposition basis function (SVDBF), and radial basis function (RBF). Results show that the accuracy of the proposed method with FBF, SVDBF, and RBF is 97.92%, 98.69%, and 99.01%, respectively. We provide theory for network design, build a dataset, demonstrate training process, and present experimental results with high precision. Without bulky benchtop setups and strict experimental limitations, this proposed simple and effective method could be an alternative in the future for spectral sensitivity function estimation.

Real-time motion-induced error compensation for 4-step phase-shifting profilometry.

Phase-shifting profilometry has been widely used in high-accuracy three-dimensional (3D) shape measurement. However, for dynamic scenarios, the object motion will lead to extra phase shift and then motion-induced error. Convenient and efficient motion-induced error compensation is still challenging. Therefore, we proposed a real-time motion-induced error compensation method for 4-step phase-shifting profilometry. The four phase-shifting images are divided into two groups to calculate two corresponding wrapped phases, one from the first three fringes and the other from the last three fringes. As the motion-induced error doubles the frequency of the projected fringes, the average phase can effectively compensate the motion-induced error because there is a π/2 phase shift between the adjacent frames. Furthermore, we designed a time sequence by recycling the projection fringes in a proper order, and the efficiency of 3D reconstruction could be effectively improved. This method performs pixel-wise error compensation, based on which we realized 50 fps real-time 3D measurement by GPU acceleration. Experimental results demonstrate that the proposed method can effectively reduce the motion-induced error.

Generalized phase unwrapping method that avoids jump errors for fringe projection profilometry.

Jump errors easily occur on the discontinuity of the wrapped phase because of the misalignment between wrapped phase and fringe order in fringe projection profilometry (FPP). In this paper, a phase unwrapping method that avoids jump errors is proposed for FPP. By building two other staggered wrapped phases from the original wrapped phase and dividing each period of fringe order into three parts, the proposed generalized tripartite phase unwrapping (Tri-PU) method can be used to avoid rather than compensatorily correct jump errors. It is suitable for the phase unwrapping method assisted by fringe order with a basic wrapped phase and fringe order, no matter which method is used to recover them. The experimental results demonstrate the effectiveness and generality of the proposed method, which is simple to implement and superior to measure complex objects with sharp edges.

High-speed 3D shape measurement using a rotary mechanical projector.

In this paper, a fast rotary mechanical projector (RMP) is designed and manufactured for high-speed 3D shape measurement. Compared with the common high-speed projectors, RMP has a good performance in high-speed projection, which can obtain high quality projected fringes with shorter camera exposure time by using the error diffusion binary coding method and chrome plating technology. The magnitude, acceptability of systemic projection error is analyzed and quantified in detail. For the quantified error, the probability distribution function (PDF) algorithm is introduced to correct the error. Corrected projection error is reduced to more than one third of the original error. Subsequently, a monocular measurement system composed of the RMP and a single camera is constructed. The combination of the RMP device and PDF algorithm ensure the accuracy of a corresponding 3D shape measurement system. Experiments have demonstrated that the proposed solution has a good performance for the 3D measurement of high-speed scenes.

Scan-free end-to-end new approach for snapshot camera spectral sensitivity estimation.

Spectral sensitivity is largely related to sensor imaging, which has drawn widespread attention in computer vision. Accurate estimation becomes increasingly urgent because manufacturers rarely disclose it. In this Letter, we present a novel, compact, inexpensive, and real-time computational system for snapshot spectral sensitivity estimation. A multi-scale camera based on the multi-scale convolutional neural network is first proposed, to the best of our knowledge, to automatically extract multiplexing features of an input image by multiscale deep learning, which is vital to solving the inverse problem in sensitivity estimation. Our network is flexible and can be designed with different convolutional kernel sizes for a given application. We build a dataset with 10,500 raw images and generate an excellent pre-trained model. Commercial cameras are adopted to test model validity; the results show that our system can achieve estimation accuracy as high as 91.35%. We provide a method for system design, propose a deep learning network, build a dataset, demonstrate training process, and present experimental results with high precision. This simple and effective method provides an accurate approach for precise estimation of spectral sensitivity and is suitable for computational applications such as pathological digital stain, virtual/augmented reality display, and high-quality image acquisition.

Deep-learning-based adaptive camera calibration for various defocusing degrees.

Camera calibration tends to suffer from the low-quality target image acquisition, which would yield inaccurate or inadequate extracted features, resulting in imprecise or even failed parameter estimation. To address this problem, this Letter proposes a novel deep-learning-based adaptive calibration method robust to defocus and noise, which could significantly enhance the image quality and effectively improve the calibration result. Our work provides a convenient multi-quality target dataset generation strategy and introduces a multi-scale deep learning framework that successfully recovers a sharp target image from a deteriorated one. Free from capturing additional patterns or using special calibration targets, the proposed method allows for a more reliable calibration based on the poor-quality acquired images. In this study, an initial training dataset can be easily established containing only 68 images captured by a smartphone. Based on the augmented dataset, the superior performance and flexible transferable ability of the proposed method are validated on another camera in the calibration experiments.

Passive binary defocusing for large depth 3D measurement based on deep learning.

Phase-shifting profilometry (PSP) based on the binary defocusing technique has been widely used due to its high-speed capability. However, the required adjustment in projector defocus by traditional method is inaccurate, inflexible, and associated with fringe pitch. Instead of manual defocusing adjustment, a passive defocus of the binary patterns based on deep learning is proposed in this paper. Learning the corresponding binary patterns with a specifically designed convolutional neural network, high-quality three-step sinusoidal patterns can be generated. Experimental results demonstrate that the proposed method could reduce phase error by 80%-90% for different fringe pitches without projector defocus and outperform the traditional method by providing more accurate and robust results within a large measuring depth.

Real-time 3D shape measurement with dual-frequency composite grating and motion-induced error reduction.

Phase-shifting profilometry has been increasingly sought and applied in dynamic three-dimensional (3D) shape measurement. However, the object motion will lead to extra phase shift error and thus measurement error. In this paper, a real-time 3D shape measurement method based on dual-frequency composite phase-shifting grating and motion-induced error reduction is proposed for a complex scene containing dynamic and static objects. The proposed method detects the motion region of a complex scene through the phase relations of the dual-frequency composite grating and reduces the motion-induced error with the combination of the phase calculated by a phase-shifting algorithm and the phase extracted by Fourier fringe analysis. It can correctly reconstruct the 3D shape of a complex dynamic scene and ensure high measurement accuracy of its static object as well. With the aid of the phase-shifting image ordering approach, the dynamic 3D shape of complex scenes can be reconstructed and the motion-induced error can also be suppressed in real time. Experimental results well proved that the proposed method is effective and practical.

Improve Temporal Fourier Transform Profilometry for Complex Dynamic Three-Dimensional Shape Measurement.

The high-speed three-dimensional (3-D) shape measurement technique has become more and more popular recently, because of the strong demand for dynamic scene measurement. The single-shot nature of Fourier Transform Profilometry (FTP) makes it highly suitable for the 3-D shape measurement of dynamic scenes. However, due to the band-pass filter, FTP method has limitations for measuring objects with sharp edges, abrupt change or non-uniform reflectivity. In this paper, an improved Temporal Fourier Transform Profilometry (TFTP) algorithm combined with the 3-D phase unwrapping algorithm based on a reference plane is presented, and the measurement of one deformed fringe pattern producing a new 3-D shape of an isolated abrupt objects has been achieved. Improved TFTP method avoids band-pass filter in spatial domain and unwraps 3-D phase distribution along the temporal axis based on the reference plane. The high-frequency information of the measured object can be well preserved, and each pixel is processed separately. Experiments verify that our method can be well applied to a dynamic 3-D shape measurement with isolated, sharp edges or abrupt change. A high-speed and low-cost structured light pattern sequence projection has also been presented, it is capable of projection frequencies in the kHz level. Using the proposed 3-D shape measurement algorithm with the self-made mechanical projector, we demonstrated dynamic 3-D reconstruction with a rate of 297 Hz, which is mainly limited by the speed of the camera.

High-speed three-dimensional shape measurement based on shifting Gray-code light.

The measuring technique combining a phase-shifting algorithm and Gray-code light has been widely used in three-dimensional (3D) shape measurement for static scenes because of its high robustness and anti-noise ability. However, in the high-speed measurement, phase unwrapping errors occur easily on the boundaries of adjacent Gray-code words because of the defocus of the projector, the motion of the objects and the non-uniform reflectivity of the surface. To overcome this challenge, a high-speed 3D shape measurement method based on shifting Gray-code light has been proposed in this paper. Firstly, the average intensity of three captured phase-shifting fringe images are used as a pixel-wise threshold to binarize the Gray codes and to eliminate most phase unwrapping errors caused by the non-uniform reflectivity, ambient light variations, and the defocus of projector. Then, the shifting Gray-code (SGC) coding strategy is proposed to avoid the remaining errors of phase unwrapping on the edge of the code words. In this strategy, no additional patterns are projected, and two sets of decoding words with staggered boundaries are built in the temporal sequences for one wrapped phase. Finally, the simple, efficient, and robust phase unwrapping can be achieved in the high-speed dynamic measurement. This proposed method has been applied to reconstruct 3D shape of randomly collapsing objects in a large depth range, and the experimental results demonstrate that it can reliably obtain high-quality shape and texture information at 310 frames per second.

High-speed three-dimensional shape measurement based on cyclic complementary Gray-code light.

The binary defocusing technique has been widely used in high-speed three-dimensional (3D) shape measurement because it breaks the bottlenecks in high-speed fringe projection and the projector's nonlinear response. However, it is challenging for this method to realize a two- or multi-frequency phase-shifting algorithm because it is difficult to simultaneously generate high-quality sinusoidal fringe patterns with different periods under the same defocusing degree. To bypass this challenge, we proposed a high-speed 3D shape measurement technique for dynamic scenes based on cyclic complementary Gray-code (CCGC) patterns. In this proposed method, the projected phase-shifting sinusoidal fringes kept one same frequency, which is beneficial to ensure the optimum defocusing degree for binary dithering technique. The wrapped phase can be calculated by phase-shifting algorithm and unwrapped with the aid of complementary Gray-code (CGC) patterns in a simple and robust way. Then, the cyclic coding strategy further extends the unambiguous phase measurement range and improves the measurement accuracy compared with the traditional Gray-coding strategy under the condition of the same number of projected patterns. High-quality 3D results of three complex dynamic scenes-including a cooling fan and a standard ceramic ball with a free-falling table tennis, collapsing building blocks, and impact of the Newton's cradle-were demonstrated at a frame rate of 357 fps. This verified the proposed method's feasibility and validity.