NMSCANet: stereo matching network for speckle variations in single-shot speckle projection profilometry.

In single-shot speckle projection profilometry (SSPP), the projected speckle inevitably undergoes changes in shape and size due to variations such as viewing angles, complex surface modulations of the test object and different projection ratios. These variations introduce randomness and unpredictability to the speckle features, resulting in erroneous or missing feature extraction and subsequently degrading 3D reconstruction accuracy across the tested surface. This work strives to explore the relationship between speckle size variations and feature extraction, and address the issue solely from the perspective of network design by leveraging specific variations in speckle size without expanding the training set. Based on the analysis of the relationship between speckle size variations and feature extraction, we introduce the NMSCANet, enabling the extraction of multi-scale speckle features. Multi-scale spatial attention is employed to enhance the perception of complex and varying speckle features in space, allowing comprehensive feature extraction across different scales. Channel attention is also employed to selectively highlight the most important and representative feature channels in each image, which is able to enhance the detection capability of high-frequency 3D surface profiles. Especially, a real binocular 3D measurement system and its digital twin with the same calibration parameters are established. Experimental results imply that NMSCANet can also exhibit more than 8 times the point cloud reconstruction stability (Std) on the testing set, and the smallest change range in terms of Mean~dis (0.0614 mm - 0.4066 mm) and Std (0.0768 mm - 0.7367 mm) when measuring a standard sphere and plane compared to other methods, faced with the speckle size changes, meanwhile NMSCANet boosts the disparity matching accuracy (EPE) by over 35% while reducing the matching error (N-PER) by over 62%. Ablation studies and validity experiments collectively substantiate that our proposed modules and constructed network have made significant advancements in enhancing network accuracy and robustness against speckle variations.

Flexible and accurate system calibration method in microscopic fringe projection profilometry.

The three-dimensional (3D) measurement task of complex microstructures holds paramount significance in the domains of precision manufacturing and inspection. The calibration of the 3D system heavily determines the final reconstruction accuracy. The widely adopted system calibration method is phase-height mapping (PHM) and stereo vision (SV) based. The former can be applied directly to the calculation without considering the imaging model of the system, but it relies on highly precise and expensive translation stages or standard blocks. The latter's accuracy cannot be guaranteed because it is difficult to accurately calibrate the projector. In this paper, we establish an optically coupled microscopic fringe projection profilometry system that consists of a Scheimpflug pinhole projector and a super-low distortion bi-telecentric camera. We introduce a simplified 3D system calibration approach that combines phase modulation transfer and ray propagation. Our method enables the simultaneous calibration of the system, including the calibration of the projector, camera, and the phase to a 3D coordinates relationship, using only a 2D target. The calibrated projector's external parameters are used to obtain the target's complete poses, and then the direct mapping coefficients of the phase to the 3D coordinates can be obtained through the optical geometry structure and phase labels. Comparable experiments verify the feasibility of the proposed method.

Parameter selection on a multi-exposure fusion method for measuring surfaces with varying reflectivity in microscope fringe projection profilometry.

As industrial and scientific advancements continue, the demand for precise measurement of three-dimensional (3D) shapes and surfaces is steadily increasing. However, accurate 3D measurement of certain surfaces, especially those with varying reflectivities, has always been a challenging issue. Multi-exposure fusion methods have shown stable, high-quality measurement results, but the selection of parameters for these methods has largely been based on experience. To address this issue, this paper has improved the multi-exposure fusion method and introduced a guided approach for parameter selection, significantly enhancing the completeness of measurement results. Additionally, a comparative model is developed to experimentally validate the specific impacts of Gaussian window variance, optimal grayscale range, and attenuation factor variance on the integrity of 3D reconstruction. The experimental results demonstrate that under the guidance of the parameter adjustment method proposed in this paper, the multi-exposure fusion for measuring the 3D topography of high-dynamic surfaces improves the restoration coverage from the original 86% (bright areas) and 50% (dark areas) to over 99%. This provides a selection strategy for parameter adjustment guidance in precise measurements based on the multi-exposure method.

Accurate 3D reconstruction of single-frame speckle-encoded textureless surfaces based on densely connected stereo matching network.

Speckle projection profilometry (SPP) determines the global correspondence between stereo images by speckle pattern(s) projection in three-dimensional (3D) vision. However, it is extremely challenging for traditional algorithms to achieve a satisfactory 3D reconstruction accuracy generally via single-frame speckle pattern, which heavily constraints the application in dynamic 3D imaging. Recently some deep learning (DL) based methods have made process in this issue but there exist deficiencies in feature extraction, leading to a limited improvement in accuracy. In this paper, we propose a stereo matching network called Densely Connected Stereo Matching (DCSM) Network that requires only single-frame speckle pattern as input, adopts densely connected feature extraction and incorporates attention weight volume construction. The densely connected multi-scale feature extraction module we constructed in DCSM Network has a positive effect on the combination of global and local information and inhibition of information loss. We also establish a real measurement system and its digital twin through Blender to obtain rich speckle data under SPP framework. Meanwhile, we introduce Fringe Projection Profilometry (FPP) to obtain phase information to assist in generating high-precision disparity as Ground Truth (GT). Experiments with different types of models and models with various perspectives are implemented to prove the effectiveness and generalization of the proposed network compared with classic and the latest DL-based algorithms. Finally, the 0.5-Pixel-Error of our method in the disparity maps is as low as 4.81%, and the accuracy is verified to be improved by up to 33.4%. As for the cloud point, our method has a reduction of 18%∼30% compared with other network-based methods.

Deep learning-based end-to-end 3D depth recovery from a single-frame fringe pattern with the MSUNet++ network.

Deep learning (DL)-based single-frame fringe pattern to 3D depth reconstruction methods have aroused extensive research interest. The goal is to estimate high-precision 3D shape from a single frame of fringe pattern with limited information. Therefore, the purpose of this work attempts to propose an end-to-end DL-based 3D reconstruction method from the single fringe pattern with excellent capability of achieving high accuracy depth recovery and geometry details preservation of tested objects. We construct a multi-scale feature fusion convolutional neural network (CNN) called MSUNet++, which incorporates discrete wavelet transform (DWT) in data preprocessing for extracting high-frequency signals of fringe patterns as input of the network. Additionally, a loss function that combines structural similarity with edge perception is established. Through these measures, high-frequency geometry details of the reconstruction results can be obviously enhanced, while the geometric shape can be effectively maintained. Ablation experiments are involved in validating the effectiveness of our proposed solution. 3D reconstructed results and analysis of generalization experiments on different tested samples imply that the proposed method in this research enjoys capabilities of higher accuracy, better detail preservation, and robustness in comparison with the compared methods.

High dynamic reflection surface 3D reconstruction with sharing phase demodulation mechanism and multi-indicators guided phase domain fusion.

Accurate and complete 3D measurement of complex high dynamic range (HDR) surfaces has been challenging for structured light projection technique. The behavior of spraying a layer of diffuse reflection material, which will inevitably incur additional thickness. Existing methods based on additional facilities will increase the cost of hardware system. The algorithms-based methods are cost-effective and nondestructive, but they generally require redundant patterns for image fusion and model training, which fail to be suitable for practicing automated 3D measurement for complex HDR surfaces. In this paper, a HDR surface 3D reconstruction method based on sharing demodulation phase unwrapping mechanism and multi-indicators guided phase fusion strategy is proposed. The division of the exposure interval is optimized via the image entropy to generate an optimal exposure sequence. The combination of temporal-spatial binary (TSB) encoding fringe patterns with time-integration strategy and the variable exposure mode of digital mirror device (DMD)-based projector with a minimum projection exposure time of 233µs enables the proposed approach to broadly adapt complex HDR surfaces. We propose an efficient phase analysis solution called sharing mechanism that wrapped phase sequences from captured different intensity fringe images are unwrapped through sharing the same group of misaligned Gray code (MGC) decoding result. Finally, a phase sequences fusion model guided by multi-indicators, including exposure quality, phase gradient smoothness and pixel effectiveness, is established to obtain an optimum phase map for final 3D reconstruction. Comparative experiments indicate that the proposed method can completely restore the 3D topography of HDR surfaces with the images reduction of at least 65% and the measurement integrity is maintained at over 98% while preserving the measurement accuracy and excluding the outliers.

Temporal-spatial binary encoding method based on dynamic threshold optimization for 3D shape measurement.

The binary encoding method has been widely used for three-dimensional (3D) shape measurement due to the high-speed projection characteristics of its digital mirror device (DMD)-based projector. However, traditional binary encoding methods require a larger defocus to achieve a good sinusoidality, leading to a reduction in the measurement depth of field and signal-to-noise ratio (SNR) of captured images, which can adversely affect the accuracy of phase extraction, particularly high-frequency fringes for 3D reconstruction. This paper proposes a spatial-temporal binary encoding method based on dynamic threshold optimization for 3D shape measurement. The proposed method decomposes an 8-bit sinusoidal fringe pattern into multiple(K) binary patterns, which can be outlined into two steps: determining the dynamic threshold and then performing temporal-spatial error diffusion encoding. By using an integral imaging strategy, approximate sinusoidal patterns can be obtained under nearly focused projection, which can then be subjected to absolute phase unwrapping and 3D reconstruction. The experiments show that compared to the three comparative algorithms under the same experimental conditions, this proposed method improves the reconstruction error of measuring a plane and an object by at least 13.66% and 12.57% when K=2. The dynamic experimental result on the palm confirms that the proposed method can reliably reconstruct the 3D shape of the moving object.

Nonlinear optimization considering target feature points for bitelecentric camera calibration: experimental study.

The telecentric camera has found extensive application in microscopy imaging due to its remarkable attributes of maintaining constant magnification and minimal distortion within its depth of field. In telecentric imaging technology, the accuracy of measurements frequently hinges upon the calibration precision of the telecentric camera. In real-world scenarios, the shallow depth of field characteristic of telecentric cameras often leads to out-of-focus targets during the capturing process, which in turn results in the inability to accurately extract pixel coordinates of feature points, making it difficult for optimization algorithms to converge to the optimal value. We propose a nonlinear optimization algorithm based on pixel coordinates of optimized feature points for bitelecentric cameras. Incorporating pixel coordinates into the optimization process yields the theoretically optimal solution based on bitelecentric camera model. The obtained pixel coordinates are used for second initial value estimation, followed by the optimization of all parameters. Compared to existing methods, the proposed approach significantly reduces reprojection errors under both blurry and clear target conditions. Experimental results demonstrate superior performance in processing blurry defocused images.

Theoretical analysis and experimental investigation of the Floyd-Steinberg-based fringe binary method with offset compensation for accurate 3D measurement.

Digital fringe projection (DFP) with defocused binary fringe patterns has the ability to overcome the projector nonlinearity and achieve a high-speed 3D measurement. The Floyd-Steinberg (FS) dithering technique is one of the most commonly adopted binary fringe coding methods due to its relatively high measurement accuracy. Nevertheless, we found that the FS binary fringe would cause a fixed error in the recovered phase, which is proven to be invariable for various defocusing levels and various phase-shift steps according to the analysis of the phase error based on noise model of phase-shifting profilometry. It means that FS binary fringe would have a certain offset in space, compared with standard sinusoidal fringe, which is verified to be essentially constant for different fringe pitches through simulation and experiment. This offset would distort the 3D geometry of the tested target for monocular systems relying on triangulation, which needs to be compensated to improve 3D measurement accuracy. Experiments are presented to demonstrate the enhanced 3D result after compensation.

Constrained nonlinear optimization method for accurate calibration of a bi-telecentric camera in a three-dimensional microtopography system.

Telecentric cameras are widely used in the field of microscopic imaging because of their constant magnification and tiny distortion in the depth of field. Camera calibration has always been a key step in the field of computer vision. Usually, the precise parameters of the telecentric camera are obtained by nonlinear optimization; however, the randomness of the optimization algorithm without proper constraints will cause the results to be inconsistent with reality. Existing studies paid little attention to this issue; therefore, we show a reliable optimization approach for the bi-telecentric camera in a structured illumination three-dimensional microtopography measurement system. In this method, the distortion-free camera parameters are solved through a closed-form solution. Then a nonlinear optimization algorithm with constraining the world coordinates of the precise calibration target is proposed to refine the global parameters, leading to the calibration results being more accurate and authentic. The real experiments are conducted to verify the feasibility of the proposed method. The comparative experiments with the exiting approach are then carried out, manifesting that the proposed method enjoys advantages in terms of both reprojection error and operating efficiency. Additionally, the average offset of the world coordinates on the calibration target derived from the proposed method verifies its effectiveness and reasonability.

Optimal frequency selection for accuracy improvement in binary defocusing fringe projection profilometry.

The binary defocusing fringe projection profilometry (FPP) technique has demonstrated various advantages for high-speed and high-accuracy three-dimensional (3D) surface measurement. However, higher fringe frequency does not necessarily give better measurements in binary defocusing FPP. To improve the 3D geometry measurement accuracy, this paper proposes an optimal frequency selection approach by analyzing the phase error distribution under different defocusing degrees. The phase error is analyzed theoretically based on the multi-frequency temporal phase unwrapping process, and the associated relationship with fringe frequency, system defocusing degree, noise, and other influencing factors is established. Meanwhile, optimal fringe frequency in a specific system is selected by the theoretical model combined with the validation of simulation experiments. Finally, the measurement accuracy could be effectively enhanced by the generated binary fringe patterns of optimal frequency. Both simulations and experiments verify the effectiveness and robustness of the proposed method.

Efficient 3D measurement of a HDR surface based on adaptive fringe projection.

3D measurement methods of a high-dynamic-range (HDR) surface based on adaptive fringe projection have aroused extensive research interest. They tend to pixel-wise adjust the fringe projection intensity to ensure full-field phase quality in light or dark regions, which has two problems: (1) traditional image intensity-based temporal phase unwrapping (TPU) is susceptible to noise in dark regions, and (2) it is time-consuming to project orthogonal fringe patterns for coordinate mapping and 3D reconstruction. Aiming to address these issues, we present an efficient adaptive fringe projection method where misaligned Gray code patterns are adopted to remove the phase error induced by low-frequency fringe patterns. Compared with traditional image intensity-based TPU, misaligned Gray-code-based TPU provides a better noise-suppression effect in dark regions, as Gray codes are generally better preserved than image intensity. Moreover, the images captured in the coordinate mapping process are shared for optimal projection intensity calculation and 3D reconstruction to reduce the number of total projection patterns, thus improving measurement efficiency. Extensive contrast experiments are conducted to demonstrate that the proposed method retrieves the 3D shapes of micro-scale HDR surfaces with high accuracy and a minimum number of projection patterns on the premise of high measurement integrity.

A Wireless Real-Time Continuous Monitoring System for the Internal Movements of Mountain Glaciers Using Sensor Networks.

With the escalation of global warming, the shrinkage of mountain glaciers has accelerated globally, the water volume from glaciers has changed, and relative disasters have increased in intensity and frequency (for example, ice avalanches, surging glaciers, and glacial lake outburst floods). However, the wireless monitoring of glacial movements cannot currently achieve omnidirectional, high-precision, real-time results, since there are some technical bottlenecks. Based on wireless networks and sensor application technologies, this study designed a wireless monitoring system for measuring the internal parameters of mountain glaciers, such as temperature, pressure, humidity, and power voltage, and for wirelessly transmitting real-time measurement data. The system consists of two parts, with a glacier internal monitoring unit as one part and a glacier surface base station as the second part. The former wirelessly transmits the monitoring data to the latter, and the latter processes the received data and then uploads the data to a cloud data platform via 4G or satellite signals. The wireless system can avoid cable constraints and transmission failures due to breaking cables. The system can provide more accurate field-monitoring data for simulating glacier movements and further offers an early warning system for glacial disasters.

3D face imaging with the spatial-temporal correlation method using a rotary speckle projector.

In this paper, a compact, cost-effective, and fast rotary speckle projector (RSP) is designed and manufactured for high-precision three-dimensional (3D) face data acquisition. Compared with the common speckle projectors, RSP uses a simple speckle pattern design method and has a good performance in high-speed projection and compact structure, which allows a flexible balance between measurement accuracy and time cost in a real acquisition task. Using a carefully designed rotation angle of the speckle mask, temporally and spatially non-correlative speckle patterns in the measurement volume can be generated. The rotation angle of the speckle mask is carefully checked and optimally selected via detailed theoretical analysis, simulation, and experiments to ensure 3D reconstruction accuracy across the reconstruction area. Subsequently, a binocular 3D face imaging system composed of the RSP and two cameras is constructed. With captured stereo speckle image pairs, we adopted our previously well-established spatial-temporal correlation method to determine the disparity. The accuracy of the 3D face imaging system was verified by using a real face mask, which is standardized by a certified, high-precision industrial 3D scanner. The real face data collection under various expressions has demonstrated that the proposed system also has a good performance for 3D face imaging in dynamic scenes.

Accurate stereo vision system calibration with chromatic concentric fringe patterns.

Camera calibration is used to determine the intrinsic and extrinsic parameters of a 3D imaging system based on structured light. Traditional methods like chessboard and circular dots usually employ an intensity-based feature point detection procedure, and are susceptible to noise, image contrast, and image blur. To address these issues, we proposed an active calibration method to accurately detect the centers of chromatic concentric fringe patterns (CCFP). Specifically, we first acquired the circular phase using a phase analysis algorithm, then extracted nine phase contours from the circular phase for the corresponding subpixel center coordinates using an ellipse fitting algorithm, and precisely calculated the final center with their weighted sum. We ran a simulation and evaluated the impacts of different degrees of Gaussian blur and noise on the calibrated parameters. The simulation demonstrates that our approach is more robust to noise and blur than previous ones, and our approach yields a higher calibration accuracy. Moreover, we carried out a comparison experiment to evaluate the performance of our method. It showed that the reprojection error can be reduced by at least 10% in the out-of-focus condition (i.e., the target is beyond the working distance of the camera) and the 3D reconstruction accuracy can be improved by nearly 10%.

3-D face registration solution with speckle encoding based spatial-temporal logical correlation algorithm.

3-D information acquisition (registration) of whole face plays a significant role in 3-D human face recognition application. In this paper, we develop a prototype of 3-D system consisting of two binocular measurement units that allows a full 3-D reconstruction by utilizing the advantages of a novel correlation algorithm. In this system, we use optical modulation to produce temporally and spatially varying high-density binary speckle patterns to encode the tested face, then propose a spatial-temporal logical correlation (STLC) stereo matching algorithm to fast determine the accurate disparity with a coarse and refined strategy. Finally the 3-D information of whole face from left- and right ear (~180°) can be obtainable by fusing the data from two measurement units. Comparative researches are performed to test a plastic model and a real human face by simulating real application situations. The results verify the feasibility and good performances of our computational frameworks and experimental configuration in terms of accuracy and time cost, which show a good application prospect in our future 3-D human face recognition research.

Optical 3-D surface reconstruction with color binary speckle pattern encoding.

This paper proposes a novel 3-D surface profile measurement scheme by only a single-shot color binary speckle pattern (CBSP) and a temporal-spatial correlation matching algorithm, which can be applied to measurements of dynamic and static objects. R/G/B channels of CBSP are coupled with three carefully designed black and white binary speckle patterns (BWBSPs), whose physical features are associated with the system configuration parameters. We mathematically deduce the concrete details of how to design such a pattern and its relationship with the system parameters selected in the experiment. During 3-D reconstruction, we develop an extended temporal-spatial correlation framework to determine the correspondence between two stereo images sequences that are composed of R/G/B images separated from a captured color stereo image pair. Comparative experiments and analysis are implemented to assess the measurement accuracy using standard workpieces (dumbbell and optical flat). The results indicate that the proposed approach enjoys better performance than the conventional BWBSP-based method in terms of spatial resolution, accuracy, and efficient reconstructed points. An experiment of applying CBSP to measuring a moving A4 paper is also presented, demonstrating the success of our computational framework. Finally discussions concerning the limitations of this method are implemented.

Integrated metagenomic data analysis demonstrates that a loss of diversity in oral microbiota is associated with periodontitis.

BACKGROUND: Periodontitis is an inflammatory disease affecting the tissues supporting teeth (periodontium). Integrative analysis of metagenomic samples from multiple periodontitis studies is a powerful way to examine microbiota diversity and interactions within host oral cavity. METHODS: A total of 43 subjects were recruited to participate in two previous studies profiling the microbial community of human subgingival plaque samples using shotgun metagenomic sequencing. We integrated metagenomic sequence data from those two studies, including six healthy controls, 14 sites representative of stable periodontitis, 16 sites representative of progressing periodontitis, and seven periodontal sites of unknown status. We applied phylogenetic diversity, differential abundance, and network analyses, as well as clustering, to the integrated dataset to compare microbiological community profiles among the different disease states. RESULTS: We found alpha-diversity, i.e., mean species diversity in sites or habitats at a local scale, to be the single strongest predictor of subjects' periodontitis status (P < 0.011). More specifically, healthy subjects had the highest alpha-diversity, while subjects with stable sites had the lowest alpha-diversity. From these results, we developed an alpha-diversity logistic model-based naive classifier able to perfectly predict the disease status of the seven subjects with unknown periodontal status (not used in training). Phylogenetic profiling resulted in the discovery of nine marker microbes, and these species are able to differentiate between stable and progressing periodontitis, achieving an accuracy of 94.4%. Finally, we found that the reduction of negatively correlated species is a notable signature of disease progression. CONCLUSIONS: Our results consistently show a strong association between the loss of oral microbiota diversity and the progression of periodontitis, suggesting that metagenomics sequencing and phylogenetic profiling are predictive of early periodontitis, leading to potential therapeutic intervention. Our results also support a keystone pathogen-mediated polymicrobial synergy and dysbiosis (PSD) model to explain the etiology of periodontitis. Apart from P. gingivalis, we identified three additional keystone species potentially mediating the progression of periodontitis progression based on pathogenic characteristics similar to those of known keystone pathogens.

Experimental study of temporal-spatial binary pattern projection for 3D shape acquisition.

Three-dimensional (3D) acquisition of an object with modest accuracy and speed is of particular concern in practice. The performance of digital sinusoidal fringe pattern projection using an off-the-shelf digital video projector is generally discounted by the nonlinearity and low switch rate. In this paper, a binary encoding method to encode one computer-generated standard sinusoidal fringe pattern is presented for circumventing such deficiencies. In previous work [Opt. Eng.54, 054108 (2015)OPEGAR0091-328610.1117/1.OE.54.5.054108], we have developed a 3D system based on this encoding tactic and showed its prospective application. Here, we first build a physical model to explain the mechanism of how to generate good sinusoidality. The phase accuracy with respect to the conventional spatial binary encoding method and sinusoidal fringe pattern is also comparatively evaluated through simulation and experiments. We also adopt two phase-height mapping relationships to experimentally compare the measurement accuracy among them. The results indicate that the proposed binary encoding strategy has a comparable performance to that of sinusoidal fringe pattern projection and enjoys advantages over the spatial binary method under the same conditions.

Characterization of micro structure through hybrid interference and phase determination in broadband light interferometry.

Broadband light interferometry, which is a well-developed method for surface profiling, has been applied with great success in the past years. Conventional multi-wavelength interferometric surface profilers mostly utilize the light irradiance to locate the zero fringe order, but the accuracy and stability can be negatively influenced by intensity fluctuations and external light disturbance, which is a serious problem. In this paper we discuss a hybrid technique combining light intensity and spectral modulation to determine zero optical path difference in which the light instability can be effectively suppressed. Additionally, the phase evaluation at each pixel will provide a high vertical resolution to obtain the characterization of the micro structure. The hybrid-interference method will not only improve the sensitivity of the measurement system but also level up the robustness and stability. Both simulation and experiment on a micro-dome structure have been presented to verify the effectiveness. Furthermore, the proposed method may be promising to replace the previously intensity-based method, especially in a complex application environment.