High-frequency color-encoded fringe-projection profilometry based on geometry constraint for large depth range.

In multi-view fringe projection profilometry (FPP), a limitation of geometry-constraint based approaches is the reduced measurement depth range often used to reduce the number of candidate points and increase the corresponding point selection reliability, when high-frequency fringe patterns are used. To extend the depth range, a new method of high-frequency fringe projection profilometry was developed by color encoding the projected fringe patterns to allow reliable candidate point selection even when six candidate points are in the measurement volume. The wrapped phase is directly retrieved using the intensity component of the hue-saturation-intensity (HSI) color space and complementary-hue is introduced to identify color codes for correct corresponding point selection. Mathematical analyses of the effect of color crosstalk on phase calculation and color code identification show that the phase calculation is independent of color crosstalk and that color crosstalk has little effect on color code identification. Experiments demonstrated that the new method can achieve high accuracy in 3D measurement over a large depth range and for isolated objects, using only two high-frequency color-encoded fringe patterns.

Single-shot absolute 3D shape measurement with deep-learning-based color fringe projection profilometry.

Recovering the high-resolution three-dimensional (3D) surface of an object from a single frame image has been the ultimate goal long pursued in fringe projection profilometry (FPP). The color fringe projection method is one of the technologies with the most potential towards such a goal due to its three-channel multiplexing properties. However, the associated color imbalance, crosstalk problems, and compromised coding strategy remain major obstacles to overcome. Inspired by recent successes of deep learning for FPP, we propose a single-shot absolute 3D shape measurement with deep-learning-based color FPP. Through "learning" on extensive data sets, the properly trained neural network can "predict" the high-resolution, motion-artifact-free, crosstalk-free absolute phase directly from one single color fringe image. Compared with the traditional approach, our method allows for more accurate phase retrieval and more robust phase unwrapping. Experimental results demonstrate that the proposed approach can provide high-accuracy single-frame absolute 3D shape measurement for complicated objects.

Real-time motion-induced-error compensation in 3D surface-shape measurement.

Object motion can introduce unknown phase shift and thus measurement error in multi-image phase-shifting methods of fringe projection profilometry. This paper presents a new method to estimate the unknown phase shifts and reduce the motion-induced error by using three phase maps computed over a multiple measurement sequence and calculating the difference between phase maps. The pixel-wise estimation of the motion-induced phase shifts permits phase-error compensation for non-homogeneous surface motion. Experiments demonstrated the ability of the method to reduce motion-induced error in real-time, for shape measurement of surfaces with high depth variation, and moving and deforming surfaces.

High-speed three-dimensional shape measurement based on cyclic complementary Gray-code light.

The binary defocusing technique has been widely used in high-speed three-dimensional (3D) shape measurement because it breaks the bottlenecks in high-speed fringe projection and the projector's nonlinear response. However, it is challenging for this method to realize a two- or multi-frequency phase-shifting algorithm because it is difficult to simultaneously generate high-quality sinusoidal fringe patterns with different periods under the same defocusing degree. To bypass this challenge, we proposed a high-speed 3D shape measurement technique for dynamic scenes based on cyclic complementary Gray-code (CCGC) patterns. In this proposed method, the projected phase-shifting sinusoidal fringes kept one same frequency, which is beneficial to ensure the optimum defocusing degree for binary dithering technique. The wrapped phase can be calculated by phase-shifting algorithm and unwrapped with the aid of complementary Gray-code (CGC) patterns in a simple and robust way. Then, the cyclic coding strategy further extends the unambiguous phase measurement range and improves the measurement accuracy compared with the traditional Gray-coding strategy under the condition of the same number of projected patterns. High-quality 3D results of three complex dynamic scenes-including a cooling fan and a standard ceramic ball with a free-falling table tennis, collapsing building blocks, and impact of the Newton's cradle-were demonstrated at a frame rate of 357 fps. This verified the proposed method's feasibility and validity.

Motion-artifact-free dynamic 3D shape measurement with hybrid Fourier-transform phase-shifting profilometry.

Fourier-transform profilometry (FTP) and phase-shifting profilometry (PSP) are two mainstream fringe projection techniques widely used for three-dimensional (3D) shape measurement. The former is well known for its single-shot nature and the latter for its higher measurement resolution and precision. However, when it comes to measuring the dynamic objects, neither approach is able to produce high-resolution, high-accuracy measurement results that are free from any depth ambiguities and motion-related artifacts. Furthermore, for scenes consisting of both static and dynamic objects, a trade-off between measurement precision and efficiency has to be made, suggesting that using a single approach can yield only suboptimal results. To this end, we propose a novel hybrid Fourier-transform phase-shifting profilometry method to integrate the advantages of both approaches. The motion vulnerability of multi-shot PSP can be overcome, or at least significantly alleviated, through the combination of single-shot FTP, while the high accuracy of PSP can also be preserved when the object is motionless. We design a phase-based, pixel-wise motion detection strategy that can accurately outline the moving object regions from their motionless counterparts. The final measurement result is obtained by fusing the determined regions where the PSP or FTP is applied correspondingly. To validate the proposed hybrid approach, we develop a real-time 3D shape measurement system for measuring multiple isolated moving objects. Experimental results demonstrate that our method achieves significantly higher precision and better robustness compared with conventional approaches where PSP or FTP is applied separately.

High-speed 3D shape measurement using the optimized composite fringe patterns and stereo-assisted structured light system.

In this paper, we propose a high-speed 3D shape measurement technique based on the optimized composite fringe patterns and stereo-assisted structured light system. Stereo phase unwrapping, as a new-fashioned method for absolute phase retrieval based on the multi-view geometric constraints, can eliminate the phase ambiguities and obtain a continuous phase map without projecting any additional patterns. However, in order to ensure the stability of phase unwrapping, the period of fringe is generally around 20, which limits the accuracy of 3D measurement. To solve this problem, we develop an optimized method for designing the composite pattern, in which the speckle pattern is embedded into the conventional 4-step phase-shifting fringe patterns without compromising the fringe modulation, and thus the phase measurement accuracy. We also present a simple and effective evaluation criterion for the correlation quality of the designed speckle pattern in order to improve the matching accuracy significantly. When the embedded speckle pattern is demodulated, the periodic ambiguities in the wrapped phase can be eliminated by combining the adaptive window image correlation with geometry constraint. Finally, some mismatched regions are further corrected based on the proposed regional diffusion compensation technique (RDC). These proposed techniques constitute a complete computational framework that allows to effectively recover an accurate, unambiguous, and distortion-free 3D point cloud with only 4 projected patterns. Experimental results verify that our method can achieve high-speed, high-accuracy, robust 3D shape measurement with dense (64-period) fringe patterns at 5000 frames per second.

Calibration method for panoramic 3D shape measurement with plane mirrors.

High-speed panoramic three-dimensional (3D) shape measurement can be achieved by introducing plane mirrors into the traditional fringe projection profilometry (FPP) system because such a system simultaneously captures fringe patterns from three different perspectives (i.e., by a real camera and two virtual cameras in the plane mirrors). However, calibrating such a system is nontrivial due to the complicated setup. This work introduces a flexible new technique to calibrate such a system. We first present the mathematical representation of the plane mirror, and then mathematically prove that it only requires the camera to observe a set of feature point pairs (including real points and virtual points) to generate a solution to the reflection matrix of a plane mirror. By calibrating the virtual and real camera in the same world coordinate system, 3D point cloud data obtained from real and virtual perspectives can be automatically aligned to generate a panoramic 3D model of the object. Finally, we developed a system to verify the performance of the proposed calibration technique for panoramic 3D shape measurement.

High-resolution real-time 360° 3D model reconstruction of a handheld object with fringe projection profilometry.

The digitization of the complete shape of real objects has essential applications in fields of intelligent manufacturing, industrial detection, and reverse modeling. In order to build the full geometric models of rigid objects, the object must be moved relative to the measurement system (or the scanner must be moved relative to the object) to obtain and integrate views of the object from all sides, which not only complicates the system configuration but makes the whole process time-consuming. In this Letter, we present a high-resolution real-time 360° three-dimensional (3D) model reconstruction method that allows one to rotate an object manually and see a continuously updated 3D model during the scanning process. A multi-view fringe projection profilometry system acquires high-precision depth information about a handheld object from different perspectives and, meanwhile, the multiple views are aligned and merged together in real time. Our system employs stereo phase unwrapping and an adaptive depth constraint that can unwrap the phase of dense fringe images robustly without increasing the number of captured patterns. We then develop an efficient coarse-to-fine registration strategy to match the 3D surface segments rapidly. Our experiments demonstrate that our method can reconstruct the high-precision complete 3D model of complex objects under arbitrary rotation without any instrument assist and expensive pre/post-processing.

High-speed real-time 3D shape measurement based on adaptive depth constraint.

Stereo phase unwrapping (SPU) has been increasingly applied to high-speed real-time fringe projection profilometry (FPP) because it can retrieve the absolute phase or matching points in a stereo FPP system without projecting or acquiring additional fringe patterns. Based on a pre-defined measurement volume, artificial maximum/minimum phase maps can be created solely using geometric constraints of the FPP system, permitting phase unwrapping on a pixel-by-pixel basis. However, when high-frequency fringes are used, the phase ambiguities will increase which makes SPU unreliable. Several auxiliary techniques have been proposed to enhance the robustness of SPU, but their flexibility still needs to be improved. In this paper, we proposed an adaptive depth constraint (ADC) approach for high-speed real-time 3D shape measurement, where the measurement depth volume for geometric constraints is adaptively updated according to the current reconstructed geometry. By utilizing the spatio-temporal correlation of moving objects under measurement, a customized and tighter depth constraint can be defined, which helps enhance the robustness of SPU over a large measurement volume. Besides, two complementary techniques, including simplified left-right consistency check and feedback mechanism based on valid area, are introduced to further increase the robustness and flexibility of the ADC. Experimental results demonstrate the success of our proposed SPU approach in recovering absolute 3D geometries of both simple and complicated objects with only three phase-shifted fringe images.

Active depth estimation from defocus using a camera array.

This paper introduces an active depth estimation method from defocus using a camera array. High-frequency phase-shifted sinusoidal fringe patterns are projected onto the surface of the object, making low-texture areas of the object surface easily distinguishable. Based on the light field measurement captured by a 5×5 camera array, a synthetic aperture refocusing of the fringe images can be realized after the camera array is properly calibrated and rectified. The fringe modulations at different depths are calculated based on the computationally refocused images, which are used as depth cues to reconstruct the 3D shape of the measured object. We implemented some experiments to verify the effectiveness of the proposed method.

Optimal wavelength selection strategy in temporal phase unwrapping with projection distance minimization.

Micro Fourier transform profilometry (µFTP) is a recently developed computational framework for high-speed dynamic 3D shape measurement of transient scenes based on fringe projection. It has been demonstrated that by using high-frame-rate fringe projection hardware, µFTP can achieve accurate, denser, unambiguous, and motion-artifact-free 3D reconstruction at a speed up to 10,000 Hz. µFTP utilizes a temporal phase unwrapping algorithm, so-called projection distance minimization (PDM), in which multiple wavelengths are used to solve the phase ambiguity optimally in the maximum-likelihood sense. However, it has been found that the choice of the wavelengths is essential to the unambiguous measurement range as well as the unwrapping reliability in the presence of noise. In this work, the relations between the wavelength combination and the noise resistance ability of PDM are analyzed and investigated in detail by analytical, emulational, and experimental means. This leads to a qualitative conclusion that the noise resistance ability of PDM is fundamentally determined by the value of each item in wavelength ratio: a smaller value of each item in wavelength ratio means better noise resistance ability in phase unwrapping. Our result provides a guideline for optimal wavelengths selection in order to improve the noise resistance ability of a practical fringe projection system. Simulations and experiments based on a microscopic fringe projection system are demonstrated to validate the correctness of our conclusion.

Dynamic microscopic 3D shape measurement based on marker-embedded Fourier transform profilometry.

In recent years, the fringe projection profilometry (FPP) technique has shown great prospects in the field of dynamic 3D measurement of microscopic surface shape. However, under dynamic conditions, it is desirable to use fewer projected patterns to minimize the sensitivity to motion. The commonly used phase-shifting method needs at least three fringe patterns to retrieve the wrapped phase, which depends heavily on the high-speed hardware to alleviate the effect of motion. Besides, to achieve an unambiguous measurement, at least two wrapped phase maps are required to obtain the absolute phase map, resulting in six pattern projections. In this paper, we propose the marker-embedded Fourier transform profilometry (MEFTP), which extends the modified Fourier transform profilometry with two embedded markers suited to assist the phase-unwrapping process. Combining the embedded markers with temporal phase difference information, the absolute phase can be reliably reconstructed with only two projected patterns. Furthermore, since the phase information is only encoded within a single high-frequency fringe, MEFTP is more suitable for measuring fast-moving or surface-changing objects compared with the phase-shifting method. Experiments on both static and dynamic scenes are performed, verifying that our method can achieve an accurate and robust measurement of a vibrating diaphragm at the speed of 200 frames per second.

Robust and efficient multi-frequency temporal phase unwrapping: optimal fringe frequency and pattern sequence selection.

Temporal phase unwrapping (TPU) is an essential algorithm in fringe projection profilometry (FPP), especially when measuring complex objects with discontinuities and isolated surfaces. Among others, the multi-frequency TPU has been proven to be the most reliable algorithm in the presence of noise. For a practical FPP system, in order to achieve an accurate, efficient, and reliable measurement, one needs to make wise choices about three key experimental parameters: the highest fringe frequency, the phase-shifting steps, and the fringe pattern sequence. However, there was very little research on how to optimize these parameters quantitatively, especially considering all three aspects from a theoretical and analytical perspective simultaneously. In this work, we propose a new scheme to determine simultaneously the optimal fringe frequency, phase-shifting steps and pattern sequence under multi-frequency TPU, robustly achieving high accuracy measurement by a minimum number of fringe frames. Firstly, noise models regarding phase-shifting algorithms as well as 3-D coordinates are established under a projector defocusing condition, which leads to the optimal highest fringe frequency for a FPP system. Then, a new concept termed frequency-to-frame ratio (FFR) that evaluates the magnitude of the contribution of each frame for TPU is defined, on which an optimal phase-shifting combination scheme is proposed. Finally, a judgment criterion is established, which can be used to judge whether the ratio between adjacent fringe frequencies is conducive to stably and efficiently unwrapping the phase. The proposed method provides a simple and effective theoretical framework to improve the accuracy, efficiency, and robustness of a practical FPP system in actual measurement conditions. The correctness of the derived models as well as the validity of the proposed schemes have been verified through extensive simulations and experiments. Based on a normal monocular 3-D FPP hardware system, our method enables high-precision unambiguous 3-D shape measurement with the highest fringe frequency up to 180 by using only 7 fringe patterns achieving a depth precision ∼ 38µm across a field of view of 400 × 300 × 400 mm.

Motion-oriented high speed 3-D measurements by binocular fringe projection using binary aperiodic patterns.

Fringe projection is an extensively used technique for high speed three-dimensional (3-D) measurements of dynamic objects. To precisely retrieve a moving object at pixel level, researchers prefer to project a sequence of fringe images onto its surface. However, the motion often leads to artifacts in reconstructions due to the sequential recording of the set of patterns. In order to reduce the adverse impact of the movement, we present a novel high speed 3-D scanning technique combining the fringe projection and stereo. Firstly, promising measuring speed is achieved by modifying the traditional aperiodic sinusoidal patterns so that the fringe images can be cast at kilohertz with the widely used defocusing strategy. Next, a temporal intensity tracing algorithm is developed to further alleviate the influence of motion by accurately tracing the ideal intensity for stereo matching. Then, a combined cost measure is suggested to robustly estimate the cost for each pixel and lastly a three-step framework of refinement follows not only to eliminate outliers caused by the motion but also to obtain sub-pixel disparity results for 3-D reconstructions. In comparison with the traditional method where the effect of motion is not considered, experimental results show that the reconstruction accuracy for dynamic objects can be improved by an order of magnitude with the proposed method.

High-precision real-time 3D shape measurement using a bi-frequency scheme and multi-view system.

High-speed and high-precision 3D shape measurement plays a central role in diverse applications such as automatic online inspection, robotics control, and human-computer interaction. Conventional multi-frame phase-shifting-based fringe projection profilometry techniques face inherent trade-offs between the speed and measurement precision, which are fundamentally limited by the fringe density and extra pattern projections used for de-ambiguity of fringe orders. Increasing the frequency of the projection fringes can obviously improve the measurement precision; however, it creates difficulties in the subsequent phase unwrapping. For this reason, to date, the frequency of the fringes in typical real-time 3D shape measurement techniques is generally less than 30 to guarantee a reasonable reliability of phase unwrapping. To overcome this limitation, a bi-frequency phase-shifting technique based on a multi-view fringe projection system is proposed, which significantly enhances the measurement precision without compromising the measurement speed. Based on the geometric constraints in a multi-view system, the unwrapped phase of the low-frequency (10-period) fringes can be obtained directly, which serves as a reference to unwrap the high-frequency phase map with a total number of periods of up to 160. Besides, the proposed scheme with 10-period and 160-period fringes is suitable for slightly defocusing projection, allowing a higher projection rate and measurement speed. Experiments on both static and dynamic scenes are performed, verifying that our method can achieve high-speed and high-precision 3D measurement at 300 frames per second with a precision of about 50 µm.

Real-time 3-D shape measurement with composite phase-shifting fringes and multi-view system.

In recent years, fringe projection has become an established and essential method for dynamic three-dimensional (3-D) shape measurement in different fields such as online inspection and real-time quality control. Numerous high-speed 3-D shape measurement methods have been developed by either employing high-speed hardware, minimizing the number of pattern projection, or both. However, dynamic 3-D shape measurement of arbitrarily-shaped objects with full sensor resolution without the necessity of additional pattern projections is still a big challenge. In this work, we introduce a high-speed 3-D shape measurement technique based on composite phase-shifting fringes and a multi-view system. The geometry constraint is adopted to search the corresponding points independently without additional images. Meanwhile, by analysing the 3-D position and the main wrapped phase of the corresponding point, pairs with an incorrect 3-D position or a considerable phase difference are effectively rejected. All of the qualified corresponding points are then corrected, and the unique one as well as the related period order is selected through the embedded triangular wave. Finally, considering that some points can only be captured by one of the cameras due to the occlusions, these points may have different fringe orders in the two views, so a left-right consistency check is employed to eliminate those erroneous period orders in this case. Several experiments on both static and dynamic scenes are performed, verifying that our method can achieve a speed of 120 frames per second (fps) with 25-period fringe patterns for fast, dense, and accurate 3-D measurement.

Temporal phase unwrapping using deep learning.

The multi-frequency temporal phase unwrapping (MF-TPU) method, as a classical phase unwrapping algorithm for fringe projection techniques, has the ability to eliminate the phase ambiguities even while measuring spatially isolated scenes or the objects with discontinuous surfaces. For the simplest and most efficient case in MF-TPU, two groups of phase-shifting fringe patterns with different frequencies are used: the high-frequency one is applied for 3D reconstruction of the tested object and the unit-frequency one is used to assist phase unwrapping for the wrapped phase with high frequency. The final measurement precision or sensitivity is determined by the number of fringes used within the high-frequency pattern, under the precondition that its absolute phase can be successfully recovered without any fringe order errors. However, due to the non-negligible noises and other error sources in actual measurement, the frequency of the high-frequency fringes is generally restricted to about 16, resulting in limited measurement accuracy. On the other hand, using additional intermediate sets of fringe patterns can unwrap the phase with higher frequency, but at the expense of a prolonged pattern sequence. With recent developments and advancements of machine learning for computer vision and computational imaging, it can be demonstrated in this work that deep learning techniques can automatically realize TPU through supervised learning, as called deep learning-based temporal phase unwrapping (DL-TPU), which can substantially improve the unwrapping reliability compared with MF-TPU even under different types of error sources, e.g., intensity noise, low fringe modulation, projector nonlinearity, and motion artifacts. Furthermore, as far as we know, our method was demonstrated experimentally that the high-frequency phase with 64 periods can be directly and reliably unwrapped from one unit-frequency phase using DL-TPU. These results highlight that challenging issues in optical metrology can be potentially overcome through machine learning, opening new avenues to design powerful and extremely accurate high-speed 3D imaging systems ubiquitous in nowadays science, industry, and multimedia.