Multi-View Fusion-Based 3D Object Detection for Robot Indoor Scene Perception.

To autonomously move and operate objects in cluttered indoor environments, a service robot requires the ability of 3D scene perception. Though 3D object detection can provide an object-level environmental description to fill this gap, a robot always encounters incomplete object observation, recurring detections of the same object, error in detection, or intersection between objects when conducting detection continuously in a cluttered room. To solve these problems, we propose a two-stage 3D object detection algorithm which is to fuse multiple views of 3D object point clouds in the first stage and to eliminate unreasonable and intersection detections in the second stage. For each view, the robot performs a 2D object semantic segmentation and obtains 3D object point clouds. Then, an unsupervised segmentation method called Locally Convex Connected Patches (LCCP) is utilized to segment the object accurately from the background. Subsequently, the Manhattan Frame estimation is implemented to calculate the main orientation of the object and subsequently, the 3D object bounding box can be obtained. To deal with the detected objects in multiple views, we construct an object database and propose an object fusion criterion to maintain it automatically. Thus, the same object observed in multi-view is fused together and a more accurate bounding box can be calculated. Finally, we propose an object filtering approach based on prior knowledge to remove incorrect and intersecting objects in the object dataset. Experiments are carried out on both SceneNN dataset and a real indoor environment to verify the stability and accuracy of 3D semantic segmentation and bounding box detection of the object with multi-view fusion.

Multi-Channel Convolutional Neural Network Based 3D Object Detection for Indoor Robot Environmental Perception.

Environmental perception is a vital feature for service robots when working in an indoor environment for a long time. The general 3D reconstruction is a low-level geometric information description that cannot convey semantics. In contrast, higher level perception similar to humans requires more abstract concepts, such as objects and scenes. Moreover, the 2D object detection based on images always fails to provide the actual position and size of an object, which is quite important for a robot's operation. In this paper, we focus on the 3D object detection to regress the object's category, 3D size, and spatial position through a convolutional neural network (CNN). We propose a multi-channel CNN for 3D object detection, which fuses three input channels including RGB, depth, and bird's eye view (BEV) images. We also propose a method to generate 3D proposals based on 2D ones in the RGB image and semantic prior. Training and test are conducted on the modified NYU V2 dataset and SUN RGB-D dataset in order to verify the effectiveness of the algorithm. We also carry out the actual experiments in a service robot to utilize the proposed 3D object detection method to enhance the environmental perception of the robot.

Phase-shifting profilometry combined with Gray-code patterns projection: unwrapping error removal by an adaptive median filter.

Phase-shifting profilometry combined with Gray-code patterns projection has been widely used for 3D measurement. In this technique, a phase-shifting algorithm is used to calculate the wrapped phase, and a set of Gray-code binary patterns is used to determine the unwrapped phase. In the real measurement, the captured Gray-code patterns are no longer binary, resulting in phase unwrapping errors at a large number of erroneous pixels. Although this problem has been attended and well resolved by a few methods, it remains challenging when a measured object has step-heights and the captured patterns contain invalid pixels. To effectively remove unwrapping errors and simultaneously preserve step-heights, in this paper, an effective method using an adaptive median filter is proposed. Both simulations and experiments can demonstrate its effectiveness.

Ternary Gray code-based phase unwrapping for 3D measurement using binary patterns with projector defocusing.

The three-dimensional measurement technique using binary pattern projection with projector defocusing has become increasingly important due to its high speed and high accuracy. To obtain even faster speed without sacrificing accuracy, a ternary Gray code-based phase-unwrapping method is proposed by using even fewer binary patterns, which makes it possible to efficiently and accurately unwrap the phase. Theoretical analysis, simulations, and experiments are presented to validate the proposed method's efficiency and robustness.

Phase error analysis and compensation for phase shifting profilometry with projector defocusing.

Phase shifting profilometry (PSP) using binary fringe patterns with projector defocusing is promising for high-speed 3D shape measurement. To obtain a high-quality phase, the projector usually requires a high defocusing level, which leads to a drastic fall in fringe contrast. Due to its convenience and high speed, PSP using squared binary patterns with small phase shifting algorithms and slight defocusing is highly desirable. In this paper, the phase accuracies of the classical phase shifting algorithms are analyzed theoretically, and then compared using both simulation and experiment. We also adapt two algorithms for PSP using squared binary patterns, which include a Hilbert three-step PSP and a double three-step PSP. Both algorithms can increase phase accuracy, with the latter featuring additional invalid point detection. The adapted algorithms are also compared with the classical algorithms. Based on our analysis and comparison results, proper algorithm selection can be easily made according to the practical requirement.

Skeleton-Based Parametric 2-D Region Representation: Disk B-Spline Curves.

The skeleton, or medial axis, is an important attribute of 2-D shapes. The disk B-spline curve (DBSC) is a skeleton-based parametric freeform 2-D region representation, which is defined in the B-spline form. The DBSC describes not only a 2-D region, which is suitable for describing heterogeneous materials in the region, but also the center curve (skeleton) of the region explicitly, which is suitable for animation, simulation, and recognition. In addition to being useful for error estimation of the B-spline curve, the DBSC can be used in designing and animating freeform 2-D regions. Despite increasing DBSC applications, its theory and fundamentals have not been thoroughly investigated. In this article, we discuss several fundamental properties and algorithms, such as the de Boor algorithm for DBSCs. We first derive the explicit evaluation and derivatives formulas at arbitrary points of a 2-D region (interior and boundary) represented by a DBSC and then provide heterogeneous object representation. We also introduce modeling and interactive heterogeneous object design methods for a DBSC, which consolidates DBSC theory and supports its further applications.

Registration of in vivo fluorescence endomicroscopy images based on feature detection.

The confocal fluorescence endomicroscopy is an emerging technology for imaging the living subjects inside the animals and human bodies. However, the acquired images vary, due to two degrees of freedom-tissue movement and tissue expansion/contraction. This makes the 3D reconstruction of them difficult and thus limits the clinic applications. In this chapter, we propose a feature-based registration algorithm to correct the distortions between these fluorescence images. The good alignment enables us to reconstruct and visualize the 3D structure of the living cells and tissues in real time, which provides the opportunity for the clinicians to diagnose various diseases, including the early-stage cancers. Experimental results on a collection of more than 300 confocal fluorescence images of the gerbil brain microvasculature clearly demonstrate the effectiveness and accuracy of our method.

Assessing the suitability of virtual reality for psychological testing.

Virtual reality (VR) is rapidly becoming an inexpensive, mainstream technology. VR technology is superambulatory as it allows participants to be examined under standardized environments and tests anywhere. In addition, it can test participants in different virtual spaces, including environments that are unsafe, inaccessible, costly or difficult to set up, or even nonexistent. We summarize the benefits and potential problems of VR technology, but we also move beyond theoretical approaches and present a customizable, open-source VR system (PSY-VR) that allows scalable psychological testing in modifiable VR environments. This system allows users to modify the environment using a simple graphical interface, without programming expertise. Moreover, as a proof-of-concept, we compare responses in a typical Flanker task between a real laboratory and a painstakingly matched virtual laboratory. Results indicate that the VR responses are comparable to real life testing, demonstrating the utility of VR for psychological assessment studies. The predicted rapid advancement of VR immersive technologies, as well the ease of their integration with physiological metrics ensures that VR-based assessment will be the modus operandi of psychological assessment in the future. This will allow controllable, low-cost assessment on a global scale. (PsycINFO Database Record (c) 2019 APA, all rights reserved).

An effective illustrative visualization framework based on photic extremum lines (PELs).

Conveying shape using feature lines is an important visualization tool in visual computing. The existing feature lines (e.g., ridges, valleys, silhouettes, suggestive contours, etc.) are solely determined by local geometry properties (e.g., normals and curvatures) as well as the view position. This paper is strongly inspired by the observation in human vision and perception that a sudden change in the luminance plays a critical role to faithfully represent and recover the 3D information. In particular, we adopt the edge detection techniques in image processing for 3D shape visualization and present Photic Extremum Lines (PELs) which emphasize significant variations of illumination over 3D surfaces. Comparing with the existing feature lines, PELs are more flexible and offer users more freedom to achieve desirable visualization effects. In addition, the user can easily control the shape visualization by changing the light position, the number of light sources, and choosing various light models. We compare PELs with the existing approaches and demonstrate that PEL is a flexible and effective tool to illustrate 3D surface and volume for visual computing.

Demons registration for in vivo and deformable laser scanning confocal endomicroscopy.

A critical effect found in noninvasive in vivo endomicroscopic imaging modalities is image distortions due to sporadic movement exhibited by living organisms. In three-dimensional confocal imaging, this effect results in a dataset that is tilted across deeper slices. Apart from that, the sequential flow of the imaging-processing pipeline restricts real-time adjustments due to the unavailability of information obtainable only from subsequent stages. To solve these problems, we propose an approach to render Demons-registered datasets as they are being captured, focusing on the coupling between registration and visualization. To improve the acquisition process, we also propose a real-time visual analytics tool, which complements the imaging pipeline and the Demons registration pipeline with useful visual indicators to provide real-time feedback for immediate adjustments. We highlight the problem of deformation within the visualization pipeline for object-ordered and image-ordered rendering. Visualizations of critical information including registration forces and partial renderings of the captured data are also presented in the analytics system. We demonstrate the advantages of the algorithmic design through experimental results with both synthetically deformed datasets and actual in vivo, time-lapse tissue datasets expressing natural deformations. Remarkably, this algorithm design is for embedded implementation in intelligent biomedical imaging instrumentation with customizable circuitry.

On combining multiple features for cartoon character retrieval and clip synthesis.

How do we retrieve cartoon characters accurately? Or how to synthesize new cartoon clips smoothly and efficiently from the cartoon library? Both questions are important for animators and cartoon enthusiasts to design and create new cartoons by utilizing existing cartoon materials. The first key issue to answer those questions is to find a proper representation that describes the cartoon character effectively. In this paper, we consider multiple features from different views, i.e., color histogram, Hausdorff edge feature, and skeleton feature, to represent cartoon characters with different colors, shapes, and gestures. Each visual feature reflects a unique characteristic of a cartoon character, and they are complementary to each other for retrieval and synthesis. However, how to combine the three visual features is the second key issue of our application. By simply concatenating them into a long vector, it will end up with the so-called "curse of dimensionality," let alone their heterogeneity embedded in different visual feature spaces. Here, we introduce a semisupervised multiview subspace learning (semi-MSL) algorithm, to encode different features in a unified space. Specifically, under the patch alignment framework, semi-MSL uses the discriminative information from labeled cartoon characters in the construction of local patches where the manifold structure revealed by unlabeled cartoon characters is utilized to capture the geometric distribution. The experimental evaluations based on both cartoon character retrieval and clip synthesis demonstrate the effectiveness of the proposed method for cartoon application. Moreover, additional results of content-based image retrieval on benchmark data suggest the generality of semi-MSL for other applications.

Toward real-time virtual biopsy of oral lesions using confocal laser endomicroscopy interfaced with embedded computing.

Oral lesions are conventionally diagnosed using white light endoscopy and histopathology. This can pose a challenge because the lesions may be difficult to visualise under white light illumination. Confocal laser endomicroscopy can be used for confocal fluorescence imaging of surface and subsurface cellular and tissue structures. To move toward real-time "virtual" biopsy of oral lesions, we interfaced an embedded computing system to a confocal laser endomicroscope to achieve a prototype three-dimensional (3-D) fluorescence imaging system. A field-programmable gated array computing platform was programmed to enable synchronization of cross-sectional image grabbing and Z-depth scanning, automate the acquisition of confocal image stacks and perform volume rendering. Fluorescence imaging of the human and murine oral cavities was carried out using the fluorescent dyes fluorescein sodium and hypericin. Volume rendering of cellular and tissue structures from the oral cavity demonstrate the potential of the system for 3-D fluorescence visualization of the oral cavity in real-time. We aim toward achieving a real-time virtual biopsy technique that can complement current diagnostic techniques and aid in targeted biopsy for better clinical outcomes.

Complex object correspondence construction in two-dimensional animation.

Correspondence construction of objects in key frames is the precondition for inbetweening and coloring in 2-D computer-assisted animation production. Since each frame of an animation consists of multiple layers, objects are complex in terms of shape and structure. Therefore, existing shape-matching algorithms specifically designed for simple structures such as a single closed contour cannot perform well on objects constructed by multiple contours with an open shape. This paper introduces a semisupervised patch alignment framework for complex object correspondence construction. In particular, the new framework constructs local patches for each point on an object and aligns these patches in a new feature space, in which correspondences between objects can be detected by the subsequent clustering. For local patch construction, pairwise constraints, which indicate the corresponding points (must link) or unfitting points (cannot link), are introduced by users to improve the performance of correspondence construction. This kind of input is convenient for animation software users via user-friendly interfaces. A dozen of experimental results on our cartoon data set that is built on industrial production suggest the effectiveness of the proposed framework for constructing correspondences of complex objects. As an extension of our framework, additional shape retrieval experiments on MPEG-7 data set show that its performance is comparable with that of a prominent algorithm published in T-PAMI 2009.

Online volume rendering of incrementally accumulated LSCEM images for superficial oral cancer detection.

Laser scanning confocal endomicroscope (LSCEM) has emerged as an imaging modality which provides non-invasive, in vivo imaging of biological tissue on a microscopic scale. Scientific visualizations for LSCEM datasets captured by current imaging systems require these datasets to be fully acquired and brought to a separate rendering machine. To extend the features and capabilities of this modality, we propose a system which is capable of performing realtime visualization of LSCEM datasets. Using field-programmable gate arrays, our system performs three tasks in parallel: (1) automated control of dataset acquisition; (2) imaging-rendering system synchronization; and (3) realtime volume rendering of dynamic datasets. Through fusion of LSCEM imaging and volume rendering processes, acquired datasets can be visualized in realtime to provide an immediate perception of the image quality and biological conditions of the subject, further assisting in realtime cancer diagnosis. Subsequently, the imaging procedure can be improved for more accurate diagnosis and reduce the need for repeating the process due to unsatisfactory datasets.

Fringe pattern denoising using coherence-enhancing diffusion.

Electronic speckle pattern interferometry is one of the methods measuring the displacement on object surfaces in which fringe patterns need to be evaluated. Noise is one of the key problems affecting further processing and reducing measurement quality. We propose an application of coherence-enhancing diffusion to fringe-pattern denoising. It smoothes a fringe pattern along directions both parallel and perpendicular to fringe orientation with suitable diffusion speeds to more effectively reduce noise and improve fringe-pattern quality. It is a generalized work of Tang's et al.'s [Opt. Lett.33, 2179 (2008)] model that only smoothes a fringe pattern along fringe orientation. Since our model diffuses a fringe pattern with an additional direction, it is able to denoise low-density fringes as well as improve denoising effectiveness for high-density fringes. Theoretical analysis as well as simulation and experimental verifications are addressed.

Generalized three-dimensional windowed Fourier transform for fringe analysis.

A 3D windowed Fourier transform is proposed for fringe sequence analysis, which processes the joint spatial and temporal information of the fringe sequence simultaneously. The 2D windowed Fourier transform in the spatial domain and the 1D windowed Fourier transform in the temporal domain are two special cases of the proposed method. The principles of windowed Fourier filtering and windowed Fourier ridges are developed. Experimental verification shows encouraging results despite a longer processing time.