Fusion of Multimodal Imaging and 3D Digitization Using Photogrammetry.

Multimodal sensors capture and integrate diverse characteristics of a scene to maximize information gain. In optics, this may involve capturing intensity in specific spectra or polarization states to determine factors such as material properties or an individual's health conditions. Combining multimodal camera data with shape data from 3D sensors is a challenging issue. Multimodal cameras, e.g., hyperspectral cameras, or cameras outside the visible light spectrum, e.g., thermal cameras, lack strongly in terms of resolution and image quality compared with state-of-the-art photo cameras. In this article, a new method is demonstrated to superimpose multimodal image data onto a 3D model created by multi-view photogrammetry. While a high-resolution photo camera captures a set of images from varying view angles to reconstruct a detailed 3D model of the scene, low-resolution multimodal camera(s) simultaneously record the scene. All cameras are pre-calibrated and rigidly mounted on a rig, i.e., their imaging properties and relative positions are known. The method was realized in a laboratory setup consisting of a professional photo camera, a thermal camera, and a 12-channel multispectral camera. In our experiments, an accuracy better than one pixel was achieved for the data fusion using multimodal superimposition. Finally, application examples of multimodal 3D digitization are demonstrated, and further steps to system realization are discussed.

Efficient freeform-based pattern projection system for 3D measurements.

For three-dimensional (3D) measurement of object surface and shape by pattern projection systems, we used a hybrid projection system, i.e., a combination of a projection lens and a transmissive freeform to generate an aperiodic sinusoidal fringe pattern. Such a freeform effects a light redistribution, thus leading to an effective and low-loss pattern projection, as it increases the total transmission intensity of the system and has less power dissipation than classical projection systems. In this paper, we present the conception and realization of the measurement setup of a transmissive fringe projection system. We compare the characteristics of the generated intensity distribution with the classical system based on GOBO (GOes Before Optics) projection and show measurement results of different surface shapes, recorded with the new system.

High-resolution 3D shape measurement with extended depth of field using fast chromatic focus stacking.

Close-range 3D sensors based on the structured light principle have a constrained measuring range due to their depth of field (DOF). Focus stacking is a method to extend the DOF. The additional time to change the focus is a drawback in high-speed measurements. In our research, the method of chromatic focus stacking was applied to a high-speed 3D sensor with 180 fps frame rate. The extended DOF was evaluated by the distance-dependent 3D resolution derived from the 3D-MTF of a tilted edge. The conventional DOF of 14 mm was extended to 21 mm by stacking two foci at 455 and 520 nm wavelength. The 3D sensor allowed shape measurements with extended DOF within 44 ms.

The Duality of Ray-Based and Pinhole-Camera Modeling and 3D Measurement Improvements Using the Ray-Based Model.

Geometrical camera modeling is the precondition for 3D-reconstruction tasks using photogrammetric sensor systems. The purpose of this study is to describe an approach for possible accuracy improvements by using the ray-based-camera model. The relations between the common pinhole and the generally valid ray-based-camera model are shown. A new approach to the implementation and calibration of the ray-based-camera model is introduced. Using a simple laboratory setup consisting of two cameras and a projector, experimental measurements were performed. The experiments and results showed the possibility of easily transforming the common pinhole model into a ray-based model and of performing calibration using the ray-based model. These initial results show the model's potential for considerable accuracy improvements, especially for sensor systems using wide-angle lenses or with deep 3D measurements. This study presents several approaches for further improvements to and the practical usage of high-precision optical 3D measurements.

High-resolution sequential thermal fringe projection technique for fast and accurate 3D shape measurement of transparent objects.

Three-dimensional (3D) shape measurement systems based on diffuse reflection of projected structured light do not deliver reliable data when measuring glossy, transparent, absorbent, or translucent objects. In recent years, we have developed a method based on stereo recording with infrared cameras and projection of areal aperiodic sinusoidal thermal patterns to detect such objects. However, the measurements took longer than 10 s, up to minutes; moreover, the measurement accuracy was improvable. Now, we have succeeded in both drastically reducing measurement time and significantly increasing measurement quality. This finally provides a technique for reliably measuring transparent objects, e.g., in series production. We demonstrate measurement examples achieved within 1 s and with 3D standard deviations less than 10 µm.

5D hyperspectral imaging: fast and accurate measurement of surface shape and spectral characteristics using structured light.

Measuring the shape (coordinates x, y, z ) and spectral characteristics (wavelength-dependent reflectance R (λi)) of macroscopic objects as a function of time (t) is of great interest in areas such as medical imaging, precision agriculture, or optical sorting. Here, we present an approach that allows to determine all these quantities with high resolution and accuracy, enabling measurement in five dimensions. We call this approach 5D hyperspectral imaging. We describe the design and implementation of a 5D sensor operating in the visible to near-infrared spectral range, which provides excellent spatial and spectral resolution, great depth accuracy, and high frame rates. The results of various experiments strongly indicate the great benefit of the new technology.

BRDF-dependent accuracy of array-projection-based 3D sensors.

In order to perform high-speed three-dimensional (3D) shape measurements with structured light systems, high-speed projectors are required. One possibility is an array projector, which allows pattern projection at several tens of kilohertz by switching on and off the LEDs of various slide projectors. The different projection centers require a separate analysis, as the intensity received by the cameras depends on the projection direction and the object's bidirectional reflectance distribution function (BRDF). In this contribution, we investigate the BRDF-dependent errors of array-projection-based 3D sensors and propose an error compensation process.

Theoretical considerations on aperiodic sinusoidal fringes in comparison to phase-shifted sinusoidal fringes for high-speed three-dimensional shape measurement.

The demand for optically reconstructing the three-dimensional (3D) surface shape of moving objects or deformation processes makes the development of high-speed projectors necessary. Our 3D sensor containing an array projector can achieve frame rates of several tens of kilohertz and is based on the projection of aperiodic sinusoidal fringes. This approach is compared with phase-shifting fringe projection as probably the most widely used technique. Theoretical considerations as well as extensive simulations are conducted to derive criteria for the design of optimal sequences of aperiodic sinusoidal fringes and to compare the number of patterns of both approaches necessary for comparable accuracies.

Underwater 3D Surface Measurement Using Fringe Projection Based Scanning Devices.

In this work we show the principle of optical 3D surface measurements based on the fringe projection technique for underwater applications. The challenges of underwater use of this technique are shown and discussed in comparison with the classical application. We describe an extended camera model which takes refraction effects into account as well as a proposal of an effective, low-effort calibration procedure for underwater optical stereo scanners. This calibration technique combines a classical air calibration based on the pinhole model with ray-based modeling and requires only a few underwater recordings of an object of known length and a planar surface. We demonstrate a new underwater 3D scanning device based on the fringe projection technique. It has a weight of about 10 kg and the maximal water depth for application of the scanner is 40 m. It covers an underwater measurement volume of 250 mm × 200 mm × 120 mm. The surface of the measurement objects is captured with a lateral resolution of 150 µm in a third of a second. Calibration evaluation results are presented and examples of first underwater measurements are given.

3D phase-shifting fringe projection system on the basis of a tailored free-form mirror.

Phase-shifting fringe projection is an effective method to perform 3D shape measurements. Conventionally, fringe projection systems utilize a digital projector that images fringes into the measurement plane. The performance of such systems is limited to the visible spectral range, as most projectors experience technical limitations in UV or IR spectral ranges. However, for certain applications these spectral ranges are of special interest. We present a wideband fringe projector that has been developed on the basis of a picture generating beamshaping mirror. This mirror generates a sinusoidal fringe pattern in the measurement plane without any additional optical elements. Phase shifting is realized without any mechanical movement by a multichip LED. As the system is based on a single mirror, it is wavelength-independent in a wide spectral range and therefore applicable in UV and IR spectral ranges. We present the design and a realized setup of this fringe projection system and the characterization of the generated intensity distribution. Experimental results of 3D shape measurements are presented.

Enhanced Contactless Vital Sign Estimation from Real-Time Multimodal 3D Image Data.

The contactless estimation of vital signs using conventional color cameras and ambient light can be affected by motion artifacts and changes in ambient light. On both these problems, a multimodal 3D imaging system with an irritation-free controlled illumination was developed in this work. In this system, real-time 3D imaging was combined with multispectral and thermal imaging. Based on 3D image data, an efficient method was developed for the compensation of head motions, and novel approaches based on the use of 3D regions of interest were proposed for the estimation of various vital signs from multispectral and thermal video data. The developed imaging system and algorithms were demonstrated with test subjects, delivering a proof-of-concept.

GOBO projection for 3D measurements at highest frame rates: a performance analysis.

Aperiodic sinusoidal patterns that are cast by a GOBO (GOes Before Optics) projector are a powerful tool for optically measuring the surface topography of moving or deforming objects with very high speed and accuracy. We optimised the first experimental setup that we were able to measure inflating car airbags at frame rates of more than 50 kHz while achieving a 3D point standard deviation of ~500 µm. Here, we theoretically investigate the method of GOBO projection of aperiodic sinusoidal fringes. In a simulation-based performance analysis, we examine the parameters that influence the accuracy of the measurement result and identify an optimal pattern design that yields the highest measurement accuracy. We compare the results with those that were obtained via GOBO projection of phase-shifted sinusoidal fringes. Finally, we experimentally verify the theoretical findings. We show that the proposed technique has several advantages over conventional fringe projection techniques, as the easy-to-build and cost-effective GOBO projector can provide a high radiant flux, allows high frame rates, and can be used over a wide spectral range.

A new method for the computer-aided evaluation of three-dimensional changes in gypsum materials.

OBJECTIVES: The quantitative evaluation of the time- or process-dependent three-dimensional stability of dental materials is a common question in dentistry. An investigation procedure has been developed based on a CAD-surface model of a prepared upper canine, as well as a high-precision physical copy (metal master die). The specific aim of this study was to test this method's reliability. Additionally, the ability of the developed procedure to determine the three-dimensional stability of resin-reinforced gypsum master casts over time was investigated. METHODS: Ten duplicate dies of improved dental stone (esthetic-rock, dentona, Germany) were manufactured, and digitized 1, 3, 7, 28 and 56 days after pouring. A three-coordinate optical measuring device was used for the data acquisition. The three-dimensional accuracy of stone dies was determined by comparing the digitized data of the stone dies made from the metal master die to its CAD-surface model (Surfacer) Version 9.0. Imageware Inc., Ann Arbor Michigan, USA). To assess the procedure, test surfaces were created from the digitized data and compared with a reference. RESULTS: The mean deviation between the digitized point cloud and the test surface was less than 3 microm. During the 56 day examination period no significant three-dimensional changes in dimensional stability were found. SIGNIFICANCE: The procedure for the quantitative three-dimensional evaluation was shown to be suitable. Best-fit registration enabled a reliable alignment of the point cloud to the CAD-surface model. Alteration of three-dimensional accuracy over 6 weeks was insignificant and without clinical relevance.