Fusion of Multimodal Imaging and 3D Digitization Using Photogrammetry.

Multimodal sensors capture and integrate diverse characteristics of a scene to maximize information gain. In optics, this may involve capturing intensity in specific spectra or polarization states to determine factors such as material properties or an individual's health conditions. Combining multimodal camera data with shape data from 3D sensors is a challenging issue. Multimodal cameras, e.g., hyperspectral cameras, or cameras outside the visible light spectrum, e.g., thermal cameras, lack strongly in terms of resolution and image quality compared with state-of-the-art photo cameras. In this article, a new method is demonstrated to superimpose multimodal image data onto a 3D model created by multi-view photogrammetry. While a high-resolution photo camera captures a set of images from varying view angles to reconstruct a detailed 3D model of the scene, low-resolution multimodal camera(s) simultaneously record the scene. All cameras are pre-calibrated and rigidly mounted on a rig, i.e., their imaging properties and relative positions are known. The method was realized in a laboratory setup consisting of a professional photo camera, a thermal camera, and a 12-channel multispectral camera. In our experiments, an accuracy better than one pixel was achieved for the data fusion using multimodal superimposition. Finally, application examples of multimodal 3D digitization are demonstrated, and further steps to system realization are discussed.

Efficient freeform-based pattern projection system for 3D measurements.

For three-dimensional (3D) measurement of object surface and shape by pattern projection systems, we used a hybrid projection system, i.e., a combination of a projection lens and a transmissive freeform to generate an aperiodic sinusoidal fringe pattern. Such a freeform effects a light redistribution, thus leading to an effective and low-loss pattern projection, as it increases the total transmission intensity of the system and has less power dissipation than classical projection systems. In this paper, we present the conception and realization of the measurement setup of a transmissive fringe projection system. We compare the characteristics of the generated intensity distribution with the classical system based on GOBO (GOes Before Optics) projection and show measurement results of different surface shapes, recorded with the new system.

High-resolution 3D shape measurement with extended depth of field using fast chromatic focus stacking.

Close-range 3D sensors based on the structured light principle have a constrained measuring range due to their depth of field (DOF). Focus stacking is a method to extend the DOF. The additional time to change the focus is a drawback in high-speed measurements. In our research, the method of chromatic focus stacking was applied to a high-speed 3D sensor with 180 fps frame rate. The extended DOF was evaluated by the distance-dependent 3D resolution derived from the 3D-MTF of a tilted edge. The conventional DOF of 14 mm was extended to 21 mm by stacking two foci at 455 and 520 nm wavelength. The 3D sensor allowed shape measurements with extended DOF within 44 ms.

High-resolution sequential thermal fringe projection technique for fast and accurate 3D shape measurement of transparent objects.

Three-dimensional (3D) shape measurement systems based on diffuse reflection of projected structured light do not deliver reliable data when measuring glossy, transparent, absorbent, or translucent objects. In recent years, we have developed a method based on stereo recording with infrared cameras and projection of areal aperiodic sinusoidal thermal patterns to detect such objects. However, the measurements took longer than 10 s, up to minutes; moreover, the measurement accuracy was improvable. Now, we have succeeded in both drastically reducing measurement time and significantly increasing measurement quality. This finally provides a technique for reliably measuring transparent objects, e.g., in series production. We demonstrate measurement examples achieved within 1 s and with 3D standard deviations less than 10 µm.

5D hyperspectral imaging: fast and accurate measurement of surface shape and spectral characteristics using structured light.

Measuring the shape (coordinates x, y, z ) and spectral characteristics (wavelength-dependent reflectance R (λi)) of macroscopic objects as a function of time (t) is of great interest in areas such as medical imaging, precision agriculture, or optical sorting. Here, we present an approach that allows to determine all these quantities with high resolution and accuracy, enabling measurement in five dimensions. We call this approach 5D hyperspectral imaging. We describe the design and implementation of a 5D sensor operating in the visible to near-infrared spectral range, which provides excellent spatial and spectral resolution, great depth accuracy, and high frame rates. The results of various experiments strongly indicate the great benefit of the new technology.

Calibration of an array projector used for high-speed three-dimensional shape measurements using a single camera.

Geometric calibration of digital light processing projectors in single-camera, fringe-projecting 3D measurement systems have been studied assuming the projector is inverse pinhole modeled. Conversely, a high-speed multi-aperture array projector (MAAP) projecting aperiodic fringes is not dependent on a digital mirror device and cannot be pinhole modeled. With MAAP projection, a stereo camera setup is required. This paper presents a model-less method to calibrate a MAAP by direct measurement of its illumination field and re-enables 3D measurements with a single camera even with surface discontinuities present. Experimental proof of principle and preliminary measurement performance are shown.

BRDF-dependent accuracy of array-projection-based 3D sensors.

In order to perform high-speed three-dimensional (3D) shape measurements with structured light systems, high-speed projectors are required. One possibility is an array projector, which allows pattern projection at several tens of kilohertz by switching on and off the LEDs of various slide projectors. The different projection centers require a separate analysis, as the intensity received by the cameras depends on the projection direction and the object's bidirectional reflectance distribution function (BRDF). In this contribution, we investigate the BRDF-dependent errors of array-projection-based 3D sensors and propose an error compensation process.

Theoretical considerations on aperiodic sinusoidal fringes in comparison to phase-shifted sinusoidal fringes for high-speed three-dimensional shape measurement.

The demand for optically reconstructing the three-dimensional (3D) surface shape of moving objects or deformation processes makes the development of high-speed projectors necessary. Our 3D sensor containing an array projector can achieve frame rates of several tens of kilohertz and is based on the projection of aperiodic sinusoidal fringes. This approach is compared with phase-shifting fringe projection as probably the most widely used technique. Theoretical considerations as well as extensive simulations are conducted to derive criteria for the design of optimal sequences of aperiodic sinusoidal fringes and to compare the number of patterns of both approaches necessary for comparable accuracies.

3D phase-shifting fringe projection system on the basis of a tailored free-form mirror.

Phase-shifting fringe projection is an effective method to perform 3D shape measurements. Conventionally, fringe projection systems utilize a digital projector that images fringes into the measurement plane. The performance of such systems is limited to the visible spectral range, as most projectors experience technical limitations in UV or IR spectral ranges. However, for certain applications these spectral ranges are of special interest. We present a wideband fringe projector that has been developed on the basis of a picture generating beamshaping mirror. This mirror generates a sinusoidal fringe pattern in the measurement plane without any additional optical elements. Phase shifting is realized without any mechanical movement by a multichip LED. As the system is based on a single mirror, it is wavelength-independent in a wide spectral range and therefore applicable in UV and IR spectral ranges. We present the design and a realized setup of this fringe projection system and the characterization of the generated intensity distribution. Experimental results of 3D shape measurements are presented.

GOBO projection for 3D measurements at highest frame rates: a performance analysis.

Aperiodic sinusoidal patterns that are cast by a GOBO (GOes Before Optics) projector are a powerful tool for optically measuring the surface topography of moving or deforming objects with very high speed and accuracy. We optimised the first experimental setup that we were able to measure inflating car airbags at frame rates of more than 50 kHz while achieving a 3D point standard deviation of ~500 µm. Here, we theoretically investigate the method of GOBO projection of aperiodic sinusoidal fringes. In a simulation-based performance analysis, we examine the parameters that influence the accuracy of the measurement result and identify an optimal pattern design that yields the highest measurement accuracy. We compare the results with those that were obtained via GOBO projection of phase-shifted sinusoidal fringes. Finally, we experimentally verify the theoretical findings. We show that the proposed technique has several advantages over conventional fringe projection techniques, as the easy-to-build and cost-effective GOBO projector can provide a high radiant flux, allows high frame rates, and can be used over a wide spectral range.