The Effect of Training on Localizing HoloLens-Generated 3D Sound Sources.

Sound localization is a crucial aspect of human auditory perception. VR (virtual reality) technologies provide immersive audio platforms that allow human listeners to experience natural sounds based on their ability to localize sound. However, the simulations of sound generated by these platforms, which are based on the general head-related transfer function (HRTF), often lack accuracy in terms of individual sound perception and localization due to significant individual differences in this function. In this study, we aimed to investigate the disparities between the perceived locations of sound sources by users and the locations generated by the platform. Our goal was to determine if it is possible to train users to adapt to the platform-generated sound sources. We utilized the Microsoft HoloLens 2 virtual platform and collected data from 12 subjects based on six separate training sessions arranged in 2 weeks. We employed three modes of training to assess their effects on sound localization, in particular for studying the impacts of multimodal error, visual, and sound guidance in combination with kinesthetic/postural guidance, on the effectiveness of the training. We analyzed the collected data in terms of the training effect between pre- and post-sessions as well as the retention effect between two separate sessions based on subject-wise paired statistics. Our findings indicate that, as far as the training effect between pre- and post-sessions is concerned, the effect is proven to be statistically significant, in particular in the case wherein kinesthetic/postural guidance is mixed with visual and sound guidance. Conversely, visual error guidance alone was found to be largely ineffective. On the other hand, as far as the retention effect between two separate sessions is concerned, we could not find any meaningful statistical implication on the effect for all three error guidance modes out of the 2-week session of training. These findings can contribute to the improvement of VR technologies by ensuring they are designed to optimize human sound localization abilities.

Real-Time Deep Recognition of Standardized Liver Ultrasound Scan Locations.

Liver ultrasound (US) plays a critical role in diagnosing liver diseases. However, it is often difficult for examiners to accurately identify the liver segments captured in US images due to patient variability and the complexity of US images. Our study aim is automatic, real-time recognition of standardized US scans coordinated with reference liver segments to guide examiners. We propose a novel deep hierarchical architecture for classifying liver US images into 11 standardized US scans, which has yet to be properly established due to excessive variability and complexity. We address this problem based on a hierarchical classification of 11 US scans with different features applied to individual hierarchies as well as a novel feature space proximity analysis for handling ambiguous US images. Experiments were performed using US image datasets obtained from a hospital setting. To evaluate the performance under patient variability, we separated the training and testing datasets into distinct patient groups. The experimental results show that the proposed method achieved an F1-score of more than 93%, which is more than sufficient for a tool to guide examiners. The superior performance of the proposed hierarchical architecture was demonstrated by comparing its performance with that of non-hierarchical architecture.

Effect of Perceptual Training with Sound-Guided and Kinesthetic Feedback on Human 3D Sound Localization Capabilities.

In this paper, we experimentally investigate how the 3D sound localization capabilities of the blind can improve through perceptual training. To this end, we develop a novel perceptual training method with sound-guided feedback and kinesthetic assistance to evaluate its effectiveness compared to conventional training methods. In perceptual training, we exclude visual perception by blindfolding the subjects to apply the proposed method to the visually impaired. Subjects used a specially designed pointing stick to generate a sound at the tip, indicating localization error and tip position. The proposed perceptual training aims to evaluate the training effect on 3D sound localization, including variations in azimuth, elevation, and distance. The six days of training based on six subjects resulted in the following outcomes: (1) In general, accuracy in full 3D sound localization can be improved based on training. (2) Training based on relative error feedback is more effective than absolute error feedback. (3) Subjects tend to underestimate distance when the sound source is near, less than 1000 mm, or larger than 15° to the left, and overestimate the elevation when the sound source is near or in the center, and within ±15° in azimuth estimations.

Progressive Deep Learning Framework for Recognizing 3D Orientations and Object Class Based on Point Cloud Representation.

Deep learning approaches to estimating full 3D orientations of objects, in addition to object classes, are limited in their accuracies, due to the difficulty in learning the continuous nature of three-axis orientation variations by regression or classification with sufficient generalization. This paper presents a novel progressive deep learning framework, herein referred to as 3D POCO Net, that offers high accuracy in estimating orientations about three rotational axes yet with efficiency in network complexity. The proposed 3D POCO Net is configured, using four PointNet-based networks for independently representing the object class and three individual axes of rotations. The four independent networks are linked by in-between association subnetworks that are trained to progressively map the global features learned by individual networks one after another for fine-tuning the independent networks. In 3D POCO Net, high accuracy is achieved by combining a high precision classification based on a large number of orientation classes with a regression based on a weighted sum of classification outputs, while high efficiency is maintained by a progressive framework by which a large number of orientation classes are grouped into independent networks linked by association subnetworks. We implemented 3D POCO Net for full three-axis orientation variations and trained it with about 146 million orientation variations augmented from the ModelNet10 dataset. The testing results show that we can achieve an orientation regression error of about 2.5° with about 90% accuracy in object classification for general three-axis orientation estimation and object classification. Furthermore, we demonstrate that a pre-trained 3D POCO Net can serve as an orientation representation platform based on which orientations as well as object classes of partial point clouds from occluded objects are learned in the form of transfer learning.

Micro star tracker with a curved vane for a short baffle length and sharp stray light attenuation.

We present a micro star tracker with curved vanes that offers a short length of the baffle and a sharp cutoff of stray light. The curved vanes are derived mathematically by ray-tracing in such a way that all the stray light from outside of the desired field of view (FOV) is reflected out. The proposed curved vane design allows a smaller number of vanes to completely cut off stray light, leading to a shorter length in baffle design. Furthermore, the capability of a sharp cutoff of stray light eases the sensitivity requirement of image sensors. For the experiment, we fabricated three micro star tracker baffles with curved vanes for 22° FOV, which are required to handle a maximum star magnitude of 5.35 for 100% sky coverage. The sizes of the baffles fabricated are 16mmΦ×16.5mm L with double curved vanes, 24mmΦ×12.1mm L with a single curved vane, and 27mmΦ×14.4mm L with double curved vanes. In comparison, the straight vane baffle designed for 22° FOV requires seven vanes with 18 mm length but results only in mild stray light attenuation with the cutoff at 32°. The proposed star tracker utilizes a 5-megapixel image sensor, 16mm×16mm×39mm in size and weighing 9.2 g with an accuracy of 1.288 arcsecond, a 20.6% improvement over when no baffle is used.

Gaze and Eye Tracking: Techniques and Applications in ADAS.

Tracking drivers' eyes and gazes is a topic of great interest in the research of advanced driving assistance systems (ADAS). It is especially a matter of serious discussion among the road safety researchers' community, as visual distraction is considered among the major causes of road accidents. In this paper, techniques for eye and gaze tracking are first comprehensively reviewed while discussing their major categories. The advantages and limitations of each category are explained with respect to their requirements and practical uses. In another section of the paper, the applications of eyes and gaze tracking systems in ADAS are discussed. The process of acquisition of driver's eyes and gaze data and the algorithms used to process this data are explained. It is explained how the data related to a driver's eyes and gaze can be used in ADAS to reduce the losses associated with road accidents occurring due to visual distraction of the driver. A discussion on the required features of current and future eye and gaze trackers is also presented.

A Comprehensive Survey of Driving Monitoring and Assistance Systems.

Improving a vehicle driver's performance decreases the damage caused by, and chances of, road accidents. In recent decades, engineers and researchers have proposed several strategies to model and improve driving monitoring and assistance systems (DMAS). This work presents a comprehensive survey of the literature related to driving processes, the main reasons for road accidents, the methods of their early detection, and state-of-the-art strategies developed to assist drivers for a safe and comfortable driving experience. The studies focused on the three main elements of the driving process, viz. driver, vehicle, and driving environment are analytically reviewed in this work, and a comprehensive framework of DMAS, major research areas, and their interaction is explored. A well-designed DMAS improves the driving experience by continuously monitoring the critical parameters associated with the driver, vehicle, and surroundings by acquiring and processing the data obtained from multiple sensors. A discussion on the challenges associated with the current and future DMAS and their potential solutions is also presented.

Accurate and Cost-Effective Micro Sun Sensor based on CMOS Black Sun Effect.

An accurate and cost-effective micro sun sensor based on the extraction of the sun vector using a phenomenon called the "black sun" is presented. Unlike conventional image-based sun sensors where there is difficulty in accurately detecting the sun center, the black sun effect allows the sun center to be accurately extracted even with the sun image appearing irregular and noisy due to glare. This allows the proposed micro sun sensor to achieve high accuracy even when a 1 mm × 1 mm CMOS image sensor with a resolution of 250 × 250 pixels is used. The proposed micro sun sensor is implemented in two application modes: (1) a stationary mode targeted at tracking the sun for heliostats or solar panels with a fixed pose of single image sensor of 1 mm × 1 mm × 1.74 mm in size and (2) a non-stationary mode targeted at determining the orientation of moving platforms with six sensors on the platform, which is configured in an icosahedron geometry of 23 mm × 23 mm × 12 mm in size. For the stationary mode, we obtained an accuracy of 0.013° by applying Kalman filter to the sun sensor measurement for a particular sensor orientation. For the non-stationary mode, we obtained an improved accuracy of 0.05° by fusing the measurements from three sun sensors available at any instant of time. Furthermore, experiments indicate that the black sun effect makes the precision of sun vector extraction independent of the sun location captured on the image plane.

A Framework of Covariance Projection on Constraint Manifold for Data Fusion.

A general framework of data fusion is presented based on projecting the probability distribution of true states and measurements around the predicted states and actual measurements onto the constraint manifold. The constraint manifold represents the constraints to be satisfied among true states and measurements, which is defined in the extended space with all the redundant sources of data such as state predictions and measurements considered as independent variables. By the general framework, we mean that it is able to fuse any correlated data sources while directly incorporating constraints and identifying inconsistent data without any prior information. The proposed method, referred to here as the Covariance Projection (CP) method, provides an unbiased and optimal solution in the sense of minimum mean square error (MMSE), if the projection is based on the minimum weighted distance on the constraint manifold. The proposed method not only offers a generalization of the conventional formula for handling constraints and data inconsistency, but also provides a new insight into data fusion in terms of a geometric-algebraic point of view. Simulation results are provided to show the effectiveness of the proposed method in handling constraints and data inconsistency.

FPGA Based Adaptive Rate and Manifold Pattern Projection for Structured Light 3D Camera System.

The quality of the captured point cloud and the scanning speed of a structured light 3D camera system depend upon their capability of handling the object surface of a large reflectance variation in the trade-off of the required number of patterns to be projected. In this paper, we propose and implement a flexible embedded framework that is capable of triggering the camera single or multiple times for capturing single or multiple projections within a single camera exposure setting. This allows the 3D camera system to synchronize the camera and projector even for miss-matched frame rates such that the system is capable of projecting different types of patterns for different scan speed applications. This makes the system capturing a high quality of 3D point cloud even for the surface of a large reflectance variation while achieving a high scan speed. The proposed framework is implemented on the Field Programmable Gate Array (FPGA), where the camera trigger is adaptively generated in such a way that the position and the number of triggers are automatically determined according to camera exposure settings. In other words, the projection frequency is adaptive to different scanning applications without altering the architecture. In addition, the proposed framework is unique as it does not require any external memory for storage because pattern pixels are generated in real-time, which minimizes the complexity and size of the application-specific integrated circuit (ASIC) design and implementation.

Distributed Multisensor Data Fusion under Unknown Correlation and Data Inconsistency.

The paradigm of multisensor data fusion has been evolved from a centralized architecture to a decentralized or distributed architecture along with the advancement in sensor and communication technologies. These days, distributed state estimation and data fusion has been widely explored in diverse fields of engineering and control due to its superior performance over the centralized one in terms of flexibility, robustness to failure and cost effectiveness in infrastructure and communication. However, distributed multisensor data fusion is not without technical challenges to overcome: namely, dealing with cross-correlation and inconsistency among state estimates and sensor data. In this paper, we review the key theories and methodologies of distributed multisensor data fusion available to date with a specific focus on handling unknown correlation and data inconsistency. We aim at providing readers with a unifying view out of individual theories and methodologies by presenting a formal analysis of their implications. Finally, several directions of future research are highlighted.

Indoor Location Sensing with Invariant Wi-Fi Received Signal Strength Fingerprinting.

A method of location fingerprinting based on the Wi-Fi received signal strength (RSS) in an indoor environment is presented. The method aims to overcome the RSS instability due to varying channel disturbances in time by introducing the concept of invariant RSS statistics. The invariant RSS statistics represent here the RSS distributions collected at individual calibration locations under minimal random spatiotemporal disturbances in time. The invariant RSS statistics thus collected serve as the reference pattern classes for fingerprinting. Fingerprinting is carried out at an unknown location by identifying the reference pattern class that maximally supports the spontaneous RSS sensed from individual Wi-Fi sources. A design guideline is also presented as a rule of thumb for estimating the number of Wi-Fi signal sources required to be available for any given number of calibration locations under a certain level of random spatiotemporal disturbances. Experimental results show that the proposed method not only provides 17% higher success rate than conventional ones but also removes the need for recalibration. Furthermore, the resolution is shown finer by 40% with the execution time more than an order of magnitude faster than the conventional methods. These results are also backed up by theoretical analysis.

Wide-baseline stereo matching based on the line intersection context for real-time workspace modeling.

Line matching in widely separated views is challenging because of large perspective distortion and violation of the planarity assumption in local regions. We introduce a novel method of wide-baseline image matching based on the coplanar line intersections for poorly textured and/or nonplanar structured scenes. The local areas of the coplanar line pairs are normalized into canonical frames by rectifying the coplanar line pairs to be orthogonal. Then, the 3D interpretation of the intersection context of the coplanar line pairs helps to match the nonplanar local regions. Furthermore, for calibrated stereo cameras, we propose a matching criterion based on 3D planar homography to improve the matching accuracy while reconstructing most likely physically existing planar patches. Experimental results demonstrate the effectiveness of the proposed method for real-world scenes.

Boundary Inheritance Codec for high-accuracy structured light three-dimensional reconstruction with comparative performance evaluation.

This paper presents a new method of structured light-based 3D reconstruction, referred to here as Boundary Inheritance Codec, that provides high accuracy and low noise in projector-camera correspondence. The proposed method features (1) real-boundary recovery: the exact locations of region boundaries, defined by a coded pattern, are identified in terms of their real coordinates on the image plane. To this end, a radiance independent recovery of accurate boundaries and a disambiguation of true and false boundaries are presented. (2) Boundary inheritance: the consistency among the same boundaries of different layers in pattern hierarchy is exploited to further enhance the accuracy of region correspondence and boundary estimation. Extensive experimentations are carried out to verify the performance of the proposed Boundary Inheritance Codec, especially, in comparison with a number of well-known methods currently available, including Gray-code (GC) plus line/phase shift (LS/PS). The results indicate that the proposed method of recovering real boundaries with boundary inheritance is superior in accuracy and robustness to Gray-code inverse (GCI), GC+LS/PS. For instance, the error standard deviation and the percentile of outliers of the proposed method were 0.152 mm and 0.089%, respectively, while those of GCI were 0.312 mm and 3.937%, respectively, and those of GC+LS/PS were 0.280/0.321 mm and 0.159/7.074%, respectively.

Structured-light-based highly dense and robust 3D reconstruction.

In this paper, a structured-light-based highly dense and robust 3D reconstruction method is proposed by combining a Gray code and region-shifting pattern. The region-shifting pattern is transformed to the trapezoidal and triangle wave shifting pattern by combining all frames of the region-shifting pattern, and then the boundary of the trapezoidal wave shifting pattern and the peak and phase of the triangle wave shifting pattern are estimated. Through this technique, the spatial resolution is increased about three times. Consequently, the 3D points are reconstructed with a resolution much higher than a camera image resolution. Moreover, as the proposed method measures the boundary and the peak with all frames, it increases the signal-to-noise ratio and is more robust than the conventional methods that use only one or two frames to detect them.

Contributions of syntactic awareness to reading in Chinese-speaking adolescent readers with and without dyslexia.

This study investigated the relative contribution of syntactic awareness to Chinese reading among Chinese-speaking adolescent readers with and without dyslexia. A total of 78 junior high school students in Hong Kong, 26 dyslexic adolescent readers, 26 average adolescent readers of the same age (chronological age control group) and 26 younger readers matched with the same reading level (reading-level group) participated and were administered measures of IQ, syntactic awareness, morphological awareness, vocabulary knowledge, working memory, word reading, and reading comprehension. Results showed that dyslexic readers scored significantly lower than chronological age but similarly to reading level control groups in most measures, especially in the areas of syntactic skills. Analyses of individual data also revealed that over half of the dyslexic readers exhibited certain aspects of deficits in syntactic skills. In regression analyses, syntactic skills were the strongest predictors of ability in word reading and reading comprehension measures. This study highlights the uniquely important correlates of syntactic skills in Chinese reading acquisition and impairment.

Energy-efficient SVM learning control system for biped walking robots.

An energy-efficient support vector machine (EE-SVM) learning control system considering the energy cost of each training sample of biped dynamic is proposed to realize energy-efficient biped walking. Energy costs of the biped walking samples are calculated. Then the samples are weighed with the inverses of the energy costs. An EE-SVM objective function with energy-related slack variables is proposed, which follows the principle that the sample with the lowest energy consumption is treated as the most important one in the training. That means the samples with lower energy consumption contribute more to the EE-SVM regression function learning, which highly increases the energy efficiency of the biped walking. Simulation results demonstrate the effectiveness of the proposed method.

Probabilistic 3D object recognition and pose estimation using multiple interpretations generation.

This paper presents a probabilistic object recognition and pose estimation method using multiple interpretation generation in cluttered indoor environments. How to handle pose ambiguity and uncertainty is the main challenge in most recognition systems. In order to solve this problem, we approach it in a probabilistic manner. First, given a three-dimensional (3D) polyhedral object model, the parallel and perpendicular line pairs, which are detected from stereo images and 3D point clouds, generate pose hypotheses as multiple interpretations, with ambiguity from partial occlusion and fragmentation of 3D lines especially taken into account. Different from the previous methods, each pose interpretation is represented as a region instead of a point in pose space reflecting the measurement uncertainty. Then, for each pose interpretation, more features around the estimated pose are further utilized as additional evidence for computing the probability using the Bayesian principle in terms of likelihood and unlikelihood. Finally, fusion strategy is applied to the top ranked interpretations with high probabilities, which are further verified and refined to give a more accurate pose estimation in real time. The experimental results show the performance and potential of the proposed approach in real cluttered domestic environments.

Cognitive skills and literacy performance of Chinese adolescents with and without dyslexia.

The present study sought to identify cognitive abilities that might distinguish Hong Kong Chinese adolescents with dyslexia and to assess how these abilities were associated with Chinese word reading, word dictation, and reading comprehension. The cognitive skills of interest were morphological awareness, visual-orthographic knowledge, rapid naming, and verbal working memory. A total of 90 junior secondary school students, 30 dyslexic, 30 chronological age controls, and 30 reading level controls was tested on a range of cognitive and literacy tasks. Dyslexic students were less competent than the control students in all cognitive and literacy measures. The regression analyses also showed that verbal working memory, rapid naming, morphological awareness, and visual-orthographic knowledge were significantly associated with literacy performance. Findings underscore the importance of these cognitive skills for Chinese literacy acquisition. Overall, this study highlights the persistent difficulties of Chinese dyslexic adolescents who seem to have multiple causes for reading and spelling difficulties.

Accurate estimation of the boundaries of a structured light pattern.

Depth recovery based on structured light using stripe patterns, especially for a region-based codec, demands accurate estimation of the true boundary of a light pattern captured on a camera image. This is because the accuracy of the estimated boundary has a direct impact on the accuracy of the depth recovery. However, recovering the true boundary of a light pattern is considered difficult due to the deformation incurred primarily by the texture-induced variation of the light reflectance at surface locales. Especially for heavily textured surfaces, the deformation of pattern boundaries becomes rather severe. We present here a novel (to the best of our knowledge) method to estimate the true boundaries of a light pattern that are severely deformed due to the heavy textures involved. First, a general formula that models the deformation of the projected light pattern at the imaging end is presented, taking into account not only the light reflectance variation but also the blurring along the optical passages. The local reflectance indices are then estimated by applying the model to two specially chosen reference projections, all-bright and all-dark. The estimated reflectance indices are to transform the edge-deformed, captured pattern signal into the edge-corrected, canonical pattern signal. A canonical pattern implies the virtual pattern that would have resulted if there were neither the reflectance variation nor the blurring in imaging optics. Finally, we estimate the boundaries of a light pattern by intersecting the canonical form of a light pattern with that of its inverse pattern. The experimental results show that the proposed method results in significant improvements in the accuracy of the estimated boundaries under various adverse conditions.