Hybrid Bright-Dark-Field Microscopic Fringe Projection System for Cu Pillar Height Measurement in Wafer-Level Package.

Cu pillars serve as interconnecting structures for 3D chip stacking in heterogeneous integration, whose height uniformity directly impacts chip yield. Compared to typical methods such as white-light interferometry and confocal microscopy for measuring Cu pillars, microscopic fringe projection profilometry (MFPP) offers obvious advantages in throughput, which has great application value in on-line bump height measurement in wafer-level packages. However, Cu pillars with large curvature and smooth surfaces pose challenges for signal detection. To enable the MFPP system to measure both the top region of the Cu pillar and the substrate, which are necessary for bump height measurement, we utilized rigorous surface scattering theory to solve the bidirectional reflective distribution function of the Cu pillar surface. Subsequently, leveraging the scattering distribution properties, we propose a hybrid bright-dark-field MFPP system concept capable of detecting weakly scattered signals from the top of the Cu pillar and reflected signals from the substrate. Experimental results demonstrate that the proposed MFPP system can measure the height of Cu pillars with an effective field of view of 15.2 mm × 8.9 mm and a maximum measurement error of less than 0.65 µm.

TTFDNet: Precise Depth Estimation from Single-Frame Fringe Patterns.

This work presents TTFDNet, a transformer-based and transfer learning network for end-to-end depth estimation from single-frame fringe patterns in fringe projection profilometry. TTFDNet features a precise contour and coarse depth (PCCD) pre-processor, a global multi-dimensional fusion (GMDF) module and a progressive depth extractor (PDE). It utilizes transfer learning through fringe structure consistency evaluation (FSCE) to leverage the transformer's benefits even on a small dataset. Tested on 208 scenes, the model achieved a mean absolute error (MAE) of 0.00372 mm, outperforming Unet (0.03458 mm) models, PDE (0.01063 mm) and PCTNet (0.00518 mm). It demonstrated precise measurement capabilities with deviations of ~90 µm for a 25.4 mm radius ball and ~6 µm for a 20 mm thick metal part. Additionally, TTFDNet showed excellent generalization and robustness in dynamic reconstruction and varied imaging conditions, making it appropriate for practical applications in manufacturing, automation and computer vision.

Single-Shot 3D Reconstruction via Nonlinear Fringe Transformation: Supervised and Unsupervised Learning Approaches.

The field of computer vision has been focusing on achieving accurate three-dimensional (3D) object representations from a single two-dimensional (2D) image through deep artificial neural networks. Recent advancements in 3D shape reconstruction techniques that combine structured light and deep learning show promise in acquiring high-quality geometric information about object surfaces. This paper introduces a new single-shot 3D shape reconstruction method that uses a nonlinear fringe transformation approach through both supervised and unsupervised learning networks. In this method, a deep learning network learns to convert a grayscale fringe input into multiple phase-shifted fringe outputs with different frequencies, which act as an intermediate result for the subsequent 3D reconstruction process using the structured-light fringe projection profilometry technique. Experiments have been conducted to validate the practicality and robustness of the proposed technique. The experimental results demonstrate that the unsupervised learning approach using a deep convolutional generative adversarial network (DCGAN) is superior to the supervised learning approach using UNet in image-to-image generation. The proposed technique's ability to accurately reconstruct 3D shapes of objects using only a single fringe image opens up vast opportunities for its application across diverse real-world scenarios.

3D Shape Measurement of Aeroengine Blade Based on Fringe Projection Profilometer Improved by Multi-Layer Concentric Ring Calibration.

The precision requirements for aeroengine blade machining are exceedingly stringent. This study aims to improve the accuracy of existing aeroengine blade measurement methods while achieving comprehensive measurement. Therefore, this study proposes a new concentric ring calibration method and designs a multi-layer concentric ring calibration plate. The effectiveness of this calibration method was verified through actual testing of standard ball gauges. Compared with the checkerboard-grid calibration method, the average deviation of the multilayer concentric ring calibration method for measuring the center distance of the standard sphere is 0.02352, which improves the measurement accuracy by 3-4 times. On the basis of multi-layer concentric ring calibration, this study builds a fringe projection profiler based on the three-frequency twelve-step phase shift method. Compared with the CMM, the average deviation of the blade chord length measured by this solution is 0.064, which meets the measurement index requirements of aeroengine fan blades.

Weakly Supervised Depth Estimation for 3D Imaging with Single Camera Fringe Projection Profilometry.

Fringe projection profilometry (FPP) is widely used for high-accuracy 3D imaging. However, employing multiple sets of fringe patterns ensures 3D reconstruction accuracy while inevitably constraining the measurement speed. Conventional dual-frequency FPP reduces the number of fringe patterns for one reconstruction to six or fewer, but the highest period-number of fringe patterns generally is limited because of phase errors. Deep learning makes depth estimation from fringe images possible. Inspired by unsupervised monocular depth estimation, this paper proposes a novel, weakly supervised method of depth estimation for single-camera FPP. The trained network can estimate the depth from three frames of 64-period fringe images. The proposed method is more efficient in terms of fringe pattern efficiency by at least 50% compared to conventional FPP. The experimental results show that the method achieves competitive accuracy compared to the supervised method and is significantly superior to the conventional dual-frequency methods.

Applications of the new Aeva-HE™ imaging system: Its link with the visual evaluation of facial wrinkles and its potential in screening tensile products.

OBJECTIVES: To evaluate, in vivo, a recently developed imaging system (Aeva-HE™), based on fringe projection methodology: (i) its correlation with expert's assessments and real age of 85 French Caucasian women and (ii) its potential, as a screening tool, in rapidly selecting prototypes of tensile cosmetic products. MATERIALS AND METHODS: First, the bare faces of 85 differently aged French/Caucasian women were photographed under standard conditions and further analyzed by the Aeva-HE™ device. An expert aesthetician graded the severity of wrinkling on a fixed scale, helped by the use of a dedicated Skin Aging Atlas. A panel of 15 judges performed the same task on full-face standard photographs. The Aeva-HE™ software yielded various wrinkle's parameters (density, volume, mean depth, etc.) on different facial locations, according to age-groups. Second, seven women, balanced in age and wrinkling severity, were recruited in a separate study. These women applied at Day 1 a prototype of the tensile product and at Day 2, they applied another prototype of the tensile product. The whole faces (before and after products application) of the seven studied women were captured and analyzed by the Aeva-HE™ system. RESULTS: The density of wrinkles was significantly highly correlated with the aesthetician scores and, unsurprisingly, with age. Some parameters (volume, depths) of different wrinkles (glabellar, crow's feet, cheeks) were quantified, showing different absolute values and of statistically different progressions with age. The amplitudes of the rapid effects brought by the two prototypes of tensile products were clearly differentiated. CONCLUSION: The recently developed device Aeva-HE™ is an efficient system for rapidly establishing a faithful and precise status of facial wrinkles, in vivo and seems like a precious tool in the rapid screening of tensile products, possibly performed on a rather limited number of subjects.

An Error Estimation System for Close-Range Photogrammetric Systems and Algorithms.

Close-range photogrammetry methods are widely used for non-contact and accurate measurements of surface shapes. These methods are based on calculating the three-dimensional coordinates of an object from two-dimensional images using special digital processing algorithms. Due to the relatively complex measurement principle, the accurate estimation of the photogrammetric measurement error is a non-trivial task. Typically, theoretical estimations or computer modelling are used to solve this problem. However, these approaches cannot provide an accurate estimate because it is impossible to consider all factors that influence the measurement results. To solve this problem, we propose the use of physical modelling. The measurement results from the photogrammetric system under test were compared with the results of a more accurate reference measurement method. This comparison allowed the error to be estimated under controlled conditions. The test object was a flexible surface whose shape could vary smoothly over a wide range. The estimation of the measurement accuracy for a large number of different surface shapes allows us to obtain new results that are difficult to obtain using standard approaches. To implement the proposed approach, a laboratory system for the error estimation of close-range photogrammetric measurements was developed. The paper contains a detailed description of the developed system and the proposed technique for a comparison of the measurement results. The error in the reference method, which was chosen to be phasogrammetry, was evaluated experimentally. Experimental testing of the stereo photogrammetric system was performed according to the proposed technique. The obtained results show that the proposed technique can reveal dependencies that may not be detected by standard approaches.

A Phase Retrieval Method for 3D Shape Measurement of High-Reflectivity Surface Based on π Phase-Shifting Fringes.

Fringe projection profilometry (FPP) has been widely used for 3D reconstruction, surface measurement, and reverse engineering. However, if the surface of an object has a high reflectivity, overexposure can easily occur. Image saturation caused by overexposure can lead to an incorrect intensity of the captured pattern images, resulting in phase and measurement errors of FPP. To address this issue, we propose a phase retrieval method for the 3D shape measurement of high-reflectivity surfaces based on π phase-shifting fringes. Our method only requires eight images to be projected, including three single-frequency three-step phase-shifting patterns and one pattern used to provide phase unwrapping constraints, called conventional patterns, as well as the π phase-shifting patterns corresponding to the four conventional patterns, called supplemental patterns. Saturated pixels of conventional fringes are replaced by unsaturated pixels in supplemental fringes to suppress phase retrieval errors. We analyzed all 16 replacement cases of fringe patterns and provided calculation methods for unwrapped phases. The main advantages of our method are as follows: (1) By combining the advantages of the stereo phase unwrapping (SPU) algorithm, the number of projected fringes is reduced. (2) By utilizing the phase unwrapping constraint provided by the fourth fringe pattern, the accuracy of SPU is improved. For highly reflective surfaces, the experimental results demonstrate the performance of the proposed method.

Human Respiration Rate Measurement with High-Speed Digital Fringe Projection Technique.

This paper proposes a non-contact continuous respiration monitoring method based on Fringe Projection Profilometry (FPP). This method aims to overcome the limitations of traditional intrusive techniques by providing continuous monitoring without interfering with normal breathing. The FPP sensor captures three-dimensional (3D) respiratory motion from the chest wall and abdomen, and the analysis algorithms extract respiratory parameters. The system achieved a high Signal-to-Noise Ratio (SNR) of 37 dB with an ideal sinusoidal respiration signal. Experimental results demonstrated that a mean correlation of 0.95 and a mean Root-Mean-Square Error (RMSE) of 0.11 breaths per minute (bpm) were achieved when comparing to a reference signal obtained from a spirometer.

PSNet: A Deep Learning Model-Based Single-Shot Digital Phase-Shifting Algorithm.

In contrast to traditional phase-shifting (PS) algorithms, which rely on capturing multiple fringe patterns with different phase shifts, digital PS algorithms provide a competitive alternative to relative phase retrieval, which achieves improved efficiency since only one pattern is required for multiple PS pattern generation. Recent deep learning-based algorithms further enhance the retrieved phase quality of complex surfaces with discontinuity, achieving state-of-the-art performance. However, since much attention has been paid to understanding image intensity mapping, such as supervision via fringe intensity loss, global temporal dependency between patterns is often ignored, which leaves room for further improvement. In this paper, we propose a deep learning model-based digital PS algorithm, termed PSNet. A loss combining both local and global temporal information among the generated fringe patterns has been constructed, which forces the model to learn inter-frame dependency between adjacent patterns, and hence leads to the improved accuracy of PS pattern generation and the associated phase retrieval. Both simulation and real-world experimental results have demonstrated the efficacy and improvement of the proposed algorithm against the state of the art.

Three-Dimensional Shape and Deformation Measurements Based on Fringe Projection Profilometry and Fluorescent Digital Image Correlation via a 3 Charge Coupled Device Camera.

We propose a novel hybrid FPP-DIC technique to measure an object's shape and deformation in 3D simultaneously by using a single 3CCD color camera, which captures the blue fringe patterns and red fluorescent speckles within the same image. Firstly, red fluorescent speckles were painted on the surface of the specimen. Subsequently, 12 computer-generated blue fringe patterns with a black background were projected onto the surface of the specimen using a DLP projector. Finally, both the reference and deformed images with three different frequencies and four shifted phases were captured using a 3CCD camera. This technique employed a three-chip configuration in which red-green-blue chips were discretely integrated in the 3CCD color camera sensor, rendering independent capture of RGB information possible. Measurement of out-of-plane displacement was carried out through the implementation of Fringe Projection Profilometry (FPP), whereas the in-plane displacement was evaluated using a 2D Digital Image Correlation (DIC) method by leveraging a telecentric-lens-based optical system. In comparison to the traditional FPP-DIC hybrid methodology, the present approach showed a lower incidence of crosstalk between the fringe patterns and speckle patterns while also offering a corrective for the coupling of the in-plane displacement and out-of-plane displacement. Experimental results for the in-plane cantilever beam and out-of-plane disk comparisons with the traditional 3D-DIC method indicated that the maximum discrepancy obtained between FPP-DIC and 3D-DIC was 0.7 µm and 0.034 mm with different magnifications, respectively, validating the effectiveness and precision of the novel proposed FPP-DIC method.

Efficient Evaluation of Concrete Fracture Surface Roughness Using Fringe Projection Technology.

The evaluation of concrete surface roughness is crucial in the field of civil engineering. The purpose of this study is to propose a no-contact and efficient method for the measurement of the roughness of concrete fracture surfaces based on fringe-projection technology. A simple phase-correction method using one additional strip image is presented for the phase unwrapping to improve the measurement efficiency and accuracy. The experimental results indicate that the measuring error for plane height is less than 0.1mm, and the relative accuracy for measuring a cylindrical object is about 0.1%, meeting the requirements for concrete fracture-surface measurement. On this basis, three-dimensional reconstructions were carried out on various concrete fracture surfaces to evaluate the roughness. The results reveal that the surface roughness (R) and fractal dimension (D) decrease as the concrete strength increases or the water-to-cement ratio decreases, consistent with previous studies. In addition, compared with the surface roughness, the fractal dimension is more sensitive to the change in concrete surface shape. The proposed method is effective for detecting concrete fracture-surface features.

Saturation-Induced Phase Error Compensation Method Using Complementary Phase.

Intensity saturation can induce phase error and, thus, measurement error in fringe projection profilometry. To reduce saturation-induced phase errors, a compensation method is developed. The mathematical model of saturation-induced phase errors is analyzed for N-step phase-shifting profilometry, and the phase error is approximately N-folder of the frequency of the projected fringe. Additional N-step phase-shifting fringe patterns with initial phase-shift π/N are projected for generating a complementary phase map. The final phase map is obtained by averaging the original phase map extracted from the original fringe patterns and the complementary phase map, and then the phase error can be canceled out. Both simulations and experiments demonstrated that the proposed method can substantially reduce the saturation-induced phase error and realize accurate measurements for a highly dynamic range of scenarios.

Phase Unwrapping Error Correction Based on Multiple Linear Regression Analysis.

Fringe projection profilometry (FPP) is prone to phase unwrapping error (PUE) due to phase noise and measurement conditions. Most of the existing PUE-correction methods detect and correct PUE on a pixel-by-pixel or partitioned block basis and do not make full use of the correlation of all information in the unwrapped phase map. In this study, a new method for detecting and correcting PUE is proposed. First, according to the low rank of the unwrapped phase map, multiple linear regression analysis is used to obtain the regression plane of the unwrapped phase, and thick PUE positions are marked on the basis of the tolerance set according to the regression plane. Then, an improved median filter is used to mark random PUE positions and finally correct marked PUE. Experimental results show that the proposed method is effective and robust. In addition, this method is progressive in the treatment of highly abrupt or discontinuous regions.

Fringe Projection Profilometry Based on Saturated Fringe Restoration in High Dynamic Range Scenes.

In high dynamic scenes, fringe projection profilometry (FPP) may encounter fringe saturation, and the phase calculated will also be affected to produce errors. This paper proposes a saturated fringe restoration method to solve this problem, taking the four-step phase shift as an example. Firstly, according to the saturation of the fringe group, the concepts of reliable area, shallow saturated area, and deep saturated area are proposed. Then, the parameter A related to the reflectivity of the object in the reliable area is calculated to interpolate A in the shallow and deep saturated areas. The theoretically shallow and deep saturated areas are not known in actual experiments. However, morphological operations can be used to dilate and erode reliable areas to produce cubic spline interpolation areas (CSI) and biharmonic spline interpolation (BSI) areas, which roughly correspond to shallow and deep saturated areas. After A is restored, it can be used as a known quantity to restore the saturated fringe using the unsaturated fringe in the same position, the remaining unrecoverable part of the fringe can be completed using CSI, and then the same part of the symmetrical fringe can be further restored. To further reduce the influence of nonlinear error, the Hilbert transform is also used in the phase calculation process of the actual experiment. The simulation and experimental results validate that the proposed method can still obtain correct results without adding additional equipment or increasing projection number, which proves the feasibility and robustness of the method.

Single-Shot Multi-Frequency 3D Shape Measurement for Discontinuous Surface Object Based on Deep Learning.

It is challenging to stably and rapidly achieve accurate absolute phase retrieval for isolated objects with a single-shot pattern in fringe projection profilometry (FPP). In this context, a single-shot multi-frequency absolute phase retrieval (SAPR-DL) method based on deep learning is proposed, which only needs to capture one fringe image to obtain the full-field precise absolute phase. Specifically, a low-frequency deformed fringe image is loaded into the trained one-to-two deep learning framework (DLFT) to predict unit-frequency and high-frequency deformed fringe images. Then, three fringe images with different frequencies are loaded into the trained deep learning phase retrieval framework (DLPR) to calculate the corresponding absolute phase. The experimental results prove that the proposed SAPR-DL method can obtain the three-dimensional (3D) shape measurement of multiple complex objects by collecting a single-shot fringe image, showing great prospects in advancing scientific and engineering applications.

An Improved Projector Calibration Method by Phase Mapping Based on Fringe Projection Profilometry.

Aiming at the problem of the low accuracy of projector calibration in a structured light system, an improved projector calibration method is proposed in this paper. One of the key ideas is to estimate the sub-pixel coordinates in the projector image plane using local random sample consensus (RANSAC). A bundle adjustment (BA) algorithm is adopted to optimize the calibration parameters to further improve the accuracy and robustness of the projector calibration. After system calibration and epipolar rectification, the mapping relationship between the pixel coordinates and the absolute phase in the projector image plane is established by using cubic polynomial fitting, and the disparity is rapidly solved by using the mapping relationship, which not only ensures the measurement accuracy, but also improves the measurement efficiency. The experimental results demonstrated that the average re-projection error after optimization is reduced to 0.03 pixels, and the proposed method is suitable for high-speed 3D reconstruction without the time-consuming homogenous point searching.

In vitro and in vivo validation of a new three-dimensional fringe projection-based device (AEVA-HE) dedicated to skin surface mapping.

BACKGROUND: Aging signs are much visible on the surface of the skin that presents different changes: cheeks start to sag, more and deeper wrinkles appear, and pigmentation spots increase. Face diagnostic to recommend products includes assessing cutaneous micro-relief or the micro-depressive network on the face. Furthermore, there is an increasing demand for clinical and instrumental methods to prove the efficacy of anti-aging treatments. As a result, very accurate and sensitive three-dimensional (3D) devices are developed and validated to measure and quantify aging skin and to catch fine anti-aging products acting on wrinkles and fine lines. METHODS: AEVA-HE, a non-invasive 3D method based on fringe projection technology, is used to robustly characterize the skin micro-relief from a full-face acquisition and from multiple extracted zones of interest. In vitro and in vivo studies are conducted to assess the reproducibility of this system and its precision toward a standard fringe projection system, DermaTOP. RESULTS: The AEVA-HE successfully measured micro-relief and wrinkles and demonstrated the reproducibility of measurements.  AEVA-HE parameters were found highly correlated to DermaTOP. CONCLUSION: The present work illustrates the performance of the AEVA-HE device and its dedicated software kit as a precious tool for quantifying the major characteristics of wrinkles appearing with age and thus offers a high potential for assessing the effect of anti-wrinkling products.

High-Accuracy Three-Dimensional Deformation Measurement System Based on Fringe Projection and Speckle Correlation.

Fringe projection profilometry (FPP) and digital image correlation (DIC) are widely applied in three-dimensional (3D) measurements. The combination of DIC and FPP can effectively overcome their respective shortcomings. However, the speckle on the surface of an object seriously affects the quality and modulation of fringe images captured by cameras, which will lead to non-negligible errors in the measurement results. In this paper, we propose a fringe image extraction method based on deep learning technology, which transforms speckle-embedded fringe images into speckle-free fringe images. The principle of the proposed method, 3D coordinate calculation, and deformation measurements are introduced. Compared with the traditional 3D-DIC method, the experimental results show that this method is effective and precise.

An Improved Synthesis Phase Unwrapping Method Based on Three-Frequency Heterodyne.

An improved three-frequency heterodyne synthesis phase unwrapping method is proposed to improve the measurement accuracy through phase difference and phase sum operations. This method can reduce the effect of noise and increase the equivalent phase frequency. According to the distribution found in the phase difference calculation process, the Otsu segmentation is introduced to judge the phase threshold. The equivalent frequency obtained from the phase sum is more than those of all projected fringe patterns. In addition, the appropriate period combinations are also studied. The simulations and related experiments demonstrate the feasibility of the proposed method and the ability to improve the accuracy of the measurement results further.