Perception-Driven Soft-Edge Occlusion for Optical See-Through Head-Mounted Displays.

Systems with occlusion capabilities, such as those used in vision augmentation, image processing, and optical see-through head-mounted display (OST-HMD), have gained popularity. Achieving precise (hard-edge) occlusion in these systems is challenging, often requiring complex optical designs and bulky volumes. On the other hand, utilizing a single transparent liquid crystal display (LCD) is a simple approach to create occlusion masks. However, the generated mask will appear defocused (soft-edge) resulting in insufficient blocking or occlusion leakage. In our work, we delve into the perception of soft-edge occlusion by the human visual system and present a preference-based optimal expansion method that minimizes perceived occlusion leakage. In a user study involving 20 participants, we made a noteworthy observation that the human eye perceives a sharper edge blur of the occlusion mask when individuals see through it and gaze at a far distance, in contrast to the camera system's observation. Moreover, our study revealed significant individual differences in the perception of soft-edge masks in human vision when focusing. These differences may lead to varying degrees of demand for mask size among individuals. Our evaluation demonstrates that our method successfully accounts for individual differences and achieves optimal masking effects at arbitrary distances and pupil sizes.

Unobtrusive Refractive Power Monitoring: Using EOG to Detect Blurred Vision.

The rise in population and aging has led to a significant increase in the number of individuals affected by common causes of vision loss. Early diagnosis and treatment are crucial to avoid the consequences of visual impairment. However, in early stages, many visual problems are making it difficult to detect. Visual adaptation can compensate for several visual deficits with adaptive eye movements. These adaptive eye movements may serve as indicators of vision loss. In this work, we investigate the association between eye movement and blurred vision. By using Electrooculography (EOG) to record eye movements, we propose a new tracking model to identify the deterioration of refractive power. We verify the technical feasibility of this method by designing a blurred vision simulation experiment. Six sets of prescription lenses and a pair of flat lenses were used to create different levels of blurring effects. We analyzed binocular movements through EOG signals and performed a seven-class classification using the ResNet18 architecture. The results revealed an average classification accuracy of 94.7% in the subject-dependent model. However, the subject-independent model presented poor performance, with the highest accuracy reaching only 34.5%. Therefore, the potential of an EOG-based visual quality monitoring system is proven. Furthermore, our experimental design provides a novel approach to assessing blurred vision.

HazARdSnap: Gazed-based Augmentation Delivery for Safe Information Access while Cycling.

During cycling activities, cyclists often monitor a variety of information such as heart rate, distance, and navigation using a bike-mounted phone or cyclocomputer. In many cases, cyclists also ride on sidewalks or paths that contain pedestrians and other obstructions such as potholes, so monitoring information on a bike-mounted interface can slow the cyclist down or cause accidents and injury. In this paper, we present HazARdSnap, an augmented reality-based information delivery approach that improves the ease of access to cycling information and at the same time preserves the user's awareness of hazards. To do so, we implemented real-time outdoor hazard detection using a combination of computer vision and motion and position data from a head mounted display (HMD). We then developed an algorithm that snaps information to detected hazards when they are also viewed so that users can simultaneously view both rendered virtual cycling information and the real-world cues such as depth, position, time to hazard, and speed that are needed to assess and avoid hazards. Results from a study with 24 participants that made use of real-world cycling and virtual hazards showed that both HazARdSnap and forward-fixed augmented reality (AR) user interfaces (UIs) can effectively help cyclists access virtual information without having to look down, which resulted in fewer collisions (51% and 43% reduction compared to baseline, respectively) with virtual hazards.

Integration of Independent Heat Transfer Mechanisms for Non-Contact Cold Sensation Presentation With Low Residual Heat.

Thermal sensation is crucial to enhancing our comprehension of the world and enhancing our ability to interact with it. Therefore, the development of thermal sensation presentation technologies holds significant potential, providing a novel method of interaction. Traditional technologies often leave residual heat in the system or the skin, affecting subsequent presentations. Our study focuses on presenting thermal sensations with low residual heat, especially cold sensations. To mitigate the impact of residual heat in the presentation system, we opted for a non-contact method, and to address the influence of residual heat on the skin, we present thermal sensations without significantly altering skin temperature. Specifically, we integrated two highly responsive and independent heat transfer mechanisms: convection via cold air and radiation via visible light, providing non-contact thermal stimuli. By rapidly alternating between perceptible decreases and imperceptible increases in temperature on the same skin area, we maintained near-constant skin temperature while presenting continuous cold sensations. In our experiments involving 15 participants, we observed that when the cooling rate was -0.2 to -0.24 °C/s and the cooling time ratio was 30 to 50%, more than 86.67% of the participants perceived only persistent cold without any warmth.

Identification of Language-Induced Mental Load from Eye Behaviors in Virtual Reality.

Experiences of virtual reality (VR) can easily break if the method of evaluating subjective user states is intrusive. Behavioral measures are increasingly used to avoid this problem. One such measure is eye tracking, which recently became more standard in VR and is often used for content-dependent analyses. This research is an endeavor to utilize content-independent eye metrics, such as pupil size and blinks, for identifying mental load in VR users. We generated mental load independently from visuals through auditory stimuli. We also defined and measured a new eye metric, focus offset, which seeks to measure the phenomenon of "staring into the distance" without focusing on a specific surface. In the experiment, VR-experienced participants listened to two native and two foreign language stimuli inside a virtual phone booth. The results show that with increasing mental load, relative pupil size on average increased 0.512 SDs (0.118 mm), with 57% reduced variance. To a lesser extent, mental load led to fewer fixations, less voluntary gazing at distracting content, and a larger focus offset as if looking through surfaces (about 0.343 SDs, 5.10 cm). These results are in agreement with previous studies. Overall, we encourage further research on content-independent eye metrics, and we hope that hardware and algorithms will be developed in the future to further increase tracking stability.

Add-on Occlusion: Turning Off-the-Shelf Optical See-through Head-mounted Displays Occlusion-capable.

The occlusion-capable optical see-through head-mounted display (OC-OSTHMD) is actively developed in recent years since it allows mutual occlusion between virtual objects and the physical world to be correctly presented in augmented reality (AR). However, implementing occlusion with the special type of OSTHMDs prevents the appealing feature from the wide application. In this paper, a novel approach for realizing mutual occlusion for common OSTHMDs is proposed. A wearable device with per-pixel occlusion capability is designed. OSTHMD devices are upgraded to be occlusion-capable by attaching the device before optical combiners. A prototype with HoloLens 1 is built. The virtual display with mutual occlusion is demonstrated in real-time. A color correction algorithm is proposed to mitigate the color aberration caused by the occlusion device. Potential applications, including the texture replacement of real objects and the more realistic semi-transparent objects display, are demonstrated. The proposed system is expected to realize a universal implementation of mutual occlusion in AR.

Effects of AR-Based Home Appliance Agents on User's Perception and Maintenance Behavior.

Maintenance of home appliances can be tedious. Maintenance work can be physically demanding and it is not always easy to know the cause of a malfunctioning appliance. Many users need to motivate themselves to perform maintenance work and consider it ideal for home appliances to be maintenance-free. On the other hand, pets and other living creatures can be taken care of with joy and without much pain, even if they are difficult to take care of. To alleviate the hassle associated with the maintenance of home appliances, we propose an augmented reality (AR) system to superimpose an agent over the home appliance of concern who changes their behavior according to the internal state of the appliance. Taking a refrigerator as an example, we verify whether such AR agent visualization motivates users to perform maintenance work and reduces the associated discomfort. We designed a cartoon-like agent and implemented a prototype system using a HoloLens 2, which can switch between several animations depending on the internal state of the refrigerator. Using the prototype system, a Wizard of Oz user study comparing three conditions was conducted. We compared the proposed method (Animacy condition), an additional behavior method (Intelligence condition), and a text-based method as a baseline for presenting the refrigerator state. In the Intelligence condition, the agent looked at the participants from time to time as if it was aware of them and exhibited help-seeking behavior only when it was considered that they could take a short break. The results show that both the Animacy and Intelligence conditions induced animacy perception and a sense of intimacy. It was also evident that the agent visualization made the participants feel more pleasant. On the other hand, the sense of discomfort was not reduced by the agent visualization and the Intelligence condition did not improve the perceived intelligence or the sense of coercion further compared to the Animacy condition.

The Effects of Speed-Modulated Visual Stimuli Seen through Smart Glasses on Work Efficiency after Viewing.

It is known that subjective time and work efficiency are affected by visual stimuli. However, existing studies only consider the effects of visual information on the user during viewing and ignore the after effects. Using smart glasses lets users see visual information while moving until just before arriving at the office or school. We hypothesize that the user's effects from the visual information they were looking at just before working or studying affects the subsequent work. Through two user studies, we investigated whether information presented on smart glasses affected subsequent work efficiency. In the first experiment, participants were presented with avatars running at two levels of speed, or no avatars, through simulated smart glasses in a virtual environment. They then solved a dot-clicking task on a desktop monitor. In the second experiment, we investigated whether the same effect could be shown while walking in the real environment, with a running and a fast-walking avatar both at the same speed in order to see the difference in the effects of the different movements. In the first experiment, we confirmed that the speed of later work tended to improve when presenting the running human-shaped avatar. From the results of the second experiment, which was conducted in the real environment, we did not confirm that the subsequent work speed varied depending on the type of avatar being displayed. As a reason for the trend of improvement in the task efficiency in the first experiment, observation of fast human motion may have unconsciously accelerated the observers' body movement speed due to the mirror neuron mechanism. As a reason for why the work speed did not improve in the second experiment, the participants may be affected by other pedestrians and running cars. Additionally, it was difficult to see the images on the smart glasses while walking in the real environment.

Multisensory Proximity and Transition Cues for Improving Target Awareness in Narrow Field of View Augmented Reality Displays.

Augmented reality applications allow users to enrich their real surroundings with additional digital content. However, due to the limited field of view of augmented reality devices, it can sometimes be difficult to become aware of newly emerging information inside or outside the field of view. Typical visual conflicts like clutter and occlusion of augmentations occur and can be further aggravated especially in the context of dense information spaces. In this article, we evaluate how multisensory cue combinations can improve the awareness for moving out-of-view objects in narrow field of view augmented reality displays. We distinguish between proximity and transition cues in either visual, auditory or tactile manner. Proximity cues are intended to enhance spatial awareness of approaching out-of-view objects while transition cues inform the user that the object just entered the field of view. In study 1, user preference was determined for 6 different cue combinations via forced-choice decisions. In study 2, the 3 most preferred modes were then evaluated with respect to performance and awareness measures in a divided attention reaction task. Both studies were conducted under varying noise levels. We show that on average the Visual-Tactile combination leads to 63% and Audio-Tactile to 65% faster reactions to incoming out-of-view augmentations than their Visual-Audio counterpart, indicating a high usefulness of tactile transition cues. We further show a detrimental effect of visual and audio noise on performance when feedback included visual proximity cues. Based on these results, we make recommendations to determine which cue combination is appropriate for which application.

Optical see-through augmented reality displays with wide field of view and hard-edge occlusion by using paired conical reflectors.

Optical see-through head-mounted displays are actively developed in recent years. An appropriate method for mutual occlusion is essential to provide a decent user experience in many application scenarios of augmented reality. However, existing mutual occlusion methods fail to work well with a large field of view (FOV). In this Letter, we propose a double-parabolic-mirror structure that renders hard-edge occlusion within a wide FOV. The parabolic mirror increases the numerical aperture of the system significantly, and the usage of paired parabolic mirrors eliminates most optical aberrations. A liquid crystal on silicon device is introduced as the spatial light modulator for imaging a bright see-through view and rendering sharp occlusion patterns. A loop structure is built to eliminate vertical parallax. The system is designed to obtain a maximum monocular FOV of H114∘×V95∘ with hard-edge occlusion, and a FOV of H83.5∘×V53.1∘ is demonstrated with our bench-top prototype.

Head-Mounted Display with Increased Downward Field of View Improves Presence and Sense of Self-Location.

Common existing head-mounted displays (HMDs) for virtual reality (VR) provide users with a high presence and embodiment. However, the field of view (FoV) of a typical HMD for VR is about 90 to 110 [deg] in the diagonal direction and about 70 to 90 [deg] in the vertical direction, which is narrower than that of humans. Specifically, the downward FoV of conventional HMDs is too narrow to present the user avatar's body and feet. To address this problem, we have developed a novel HMD with a pair of additional display units to increase the downward FoV by approximately 60 ( 10+50) [deg]. We comprehensively investigated the effects of the increased downward FoV on the sense of immersion that includes presence, sense of self-location (SoSL), sense of agency (SoA), and sense of body ownership (SoBO) during VR experience and on patterns of head movements and cybersickness as its secondary effects. As a result, it was clarified that the HMD with an increased FoV improved presence and SoSL. Also, it was confirmed that the user could see the object below with a head movement pattern close to the real behavior, and did not suffer from cybersickness. Moreover, the effect of the increased downward FoV on SoBO and SoA was limited since it was easier to perceive the misalignment between the real and virtual bodies.

Recurrent Neural Network for Inertial Gait User Recognition in Smartphones.

In this article, a gait recognition algorithm is presented based on the information obtained from inertial sensors embedded in a smartphone, in particular, the accelerometers and gyroscopes typically embedded on them. The algorithm processes the signal by extracting gait cycles, which are then fed into a Recurrent Neural Network (RNN) to generate feature vectors. To optimize the accuracy of this algorithm, we apply a random grid hyperparameter selection process followed by a hand-tuning method to reach the final hyperparameter configuration. The different configurations are tested on a public database with 744 users and compared with other algorithms that were previously tested on the same database. After reaching the best-performing configuration for our algorithm, we obtain an equal error rate (EER) of 11.48% when training with only 20% of the users. Even better, when using 70% of the users for training, that value drops to 7.55%. The system manages to improve on state-of-the-art methods, but we believe the algorithm could reach a significantly better performance if it was trained with more visits per user. With a large enough database with several visits per user, the algorithm could improve substantially.

Light Attenuation Display: Subtractive See-Through Near-Eye Display via Spatial Color Filtering.

We present a display for optical see-through near-eye displays based on light attenuation, a new paradigm that forms images by spatially subtracting colors of light. Existing optical see-through head-mounted displays (OST-HMDs) form virtual images in an additive manner-they optically combine the light from an embedded light source such as a microdisplay into the users' field of view (FoV). Instead, our light attenuation display filters the color of the real background light pixel-wise in the users' see-through view, resulting in an image as a spatial color filter. Our image formation is complementary to existing light-additive OST-HMDs. The core optical component in our system is a phase-only spatial light modulator (PSLM), a liquid crystal module that can control the phase of the light in each pixel. By combining PSLMs with polarization optics, our system realizes a spatially programmable color filter. In this paper, we introduce our optics design, evaluate the spatial color filter, consider applications including image rendering and FoV color control, and discuss the limitations of the current prototype.

The Influence of Label Design on Search Performance and Noticeability in Wide Field of View Augmented Reality Displays.

In Augmented Reality (AR), search performance for outdoor tasks is an important metric for evaluating the success of a large number of AR applications. Users must be able to find content quickly, labels and indicators must not be invasive but still clearly noticeable, and the user interface should maximize search performance in a variety of conditions. To address these issues, we have set up a series of experiments to test the influence of virtual characteristics such as color, size, and leader lines on the performance of search tasks and noticeability in both real and simulated environments. We evaluate two primary areas, including 1) the effects of peripheral field of view (FOV) limitations and labeling techniques on target acquisition during outdoor mobile search, and 2) the influence of local characteristics such as color, size, and motion on text labels over dynamic backgrounds. The first experiment showed that limited FOV will severely limit search performance, but that appropriate placement of labels and leaders within the periphery can alleviate this problem without interfering with walking or decreasing user comfort. In the second experiment, we found that different types of motion are more noticeable in optical versus video see-through displays, but that blue coloration is most noticeable in both. Results can aid in designing more effective view management techniques, especially for wider field of view displays.

Force Rendering and its Evaluation of a Friction-Based Walking Sensation Display for a Seated User.

Most existing locomotion devices that represent the sensation of walking target a user who is actually performing a walking motion. Here, we attempted to represent the walking sensation, especially a kinesthetic sensation and advancing feeling (the sense of moving forward) while the user remains seated. To represent the walking sensation using a relatively simple device, we focused on the force rendering and its evaluation of the longitudinal friction force applied on the sole during walking. Based on the measurement of the friction force applied on the sole during actual walking, we developed a novel friction force display that can present the friction force without the influence of body weight. Using performance evaluation testing, we found that the proposed method can stably and rapidly display friction force. Also, we developed a virtual reality (VR) walk-through system that is able to present the friction force through the proposed device according to the avatar's walking motion in a virtual world. By evaluating the realism, we found that the proposed device can represent a more realistic advancing feeling than vibration feedback.

Emulation of Physician Tasks in Eye-Tracked Virtual Reality for Remote Diagnosis of Neurodegenerative Disease.

For neurodegenerative conditions like Parkinson's disease, early and accurate diagnosis is still a difficult task. Evaluations can be time consuming, patients must often travel to metropolitan areas or different cities to see experts, and misdiagnosis can result in improper treatment. To date, only a handful of assistive or remote methods exist to help physicians evaluate patients with suspected neurological disease in a convenient and consistent way. In this paper, we present a low-cost VR interface designed to support evaluation and diagnosis of neurodegenerative disease and test its use in a clinical setting. Using a commercially available VR display with an infrared camera integrated into the lens, we have constructed a 3D virtual environment designed to emulate common tasks used to evaluate patients, such as fixating on a point, conducting smooth pursuit of an object, or executing saccades. These virtual tasks are designed to elicit eye movements commonly associated with neurodegenerative disease, such as abnormal saccades, square wave jerks, and ocular tremor. Next, we conducted experiments with 9 patients with a diagnosis of Parkinson's disease and 7 healthy controls to test the system's potential to emulate tasks for clinical diagnosis. We then applied eye tracking algorithms and image enhancement to the eye recordings taken during the experiment and conducted a short follow-up study with two physicians for evaluation. Results showed that our VR interface was able to elicit five common types of movements usable for evaluation, physicians were able to confirm three out of four abnormalities, and visualizations were rated as potentially useful for diagnosis.

Design and Psychophysical Evaluation of the HapSticks: A Novel Non-Grounded Mechanism for Presenting Tool-Mediated Vertical Forces.

Force feedback in tool-mediated interactions with the environment is important for the successful performance of complex tasks in daily life as well as in specialized fields such as medicine. Most stylus-based haptic devices require either grounding or attachment to the user's body. Recently, non-grounded haptic devices have attracted a growing interest. In this study, we propose a non-grounded rotation mechanism to represent the vertical forces applied on the tip of a tool by mimicking the cutaneous sensations that are caused by such forces. As an example of an application of our method, we developed a non-grounded haptic device called HapSticks, which mimicked the sensation of manipulating objects using chopsticks. First, using an adjustment paradigm, we directly compared a virtual weight rendered by our device and a real weight to investigate the relation of real weight and virtual weight. Next, we used a forced choice constant stimuli paradigm in a virtual and a real weight discrimination task. We conclude that our novel device renders a reliable illusion of sensed weight that leads to a discrimination ability that is typical of virtual-reality applications but worse than the discrimination between real weights.

Feature detection in biological tissues using multi-band and narrow-band imaging.

PURPOSE: In the past decade, augmented reality systems have been expected to support surgical operations by making it possible to view invisible objects that are inside or occluded by the skull, hands, or organs. However, the properties of biological tissues that are non-rigid and featureless require a large number of distributed features to track the movement of tissues in detail. METHODS: With the goal of increasing the number of feature points in organ tracking, we propose a feature detection using multi-band and narrow-band imaging and a new band selection method. The depth of light penetration into an object depends on the wavelength of light based on optical characteristics. We applied typical feature detectors to detect feature points using three selected bands in a human hand. To consider surgical situations, we applied our method to a chicken liver with a variety of light conditions. RESULTS: Our experimental results revealed that the image of each band exhibited a different distribution of feature points. In addition, the total number of feature points determined by the proposed method exceeded that of the R, G, and B images obtained using a normal camera. The results using a chicken liver with various light sources and intensities also show different distributions with each selected band. CONCLUSIONS: We have proposed a feature detection method using multi-band and narrow-band imaging and a band selection method. The results of our experiments confirmed that the proposed method increased the number of distributed feature points. The proposed method was also effective for different light conditions.

ModulAR: eye-controlled vision augmentations for head mounted displays.

In the last few years, the advancement of head mounted display technology and optics has opened up many new possibilities for the field of Augmented Reality. However, many commercial and prototype systems often have a single display modality, fixed field of view, or inflexible form factor. In this paper, we introduce Modular Augmented Reality (ModulAR), a hardware and software framework designed to improve flexibility and hands-free control of video see-through augmented reality displays and augmentative functionality. To accomplish this goal, we introduce the use of integrated eye tracking for on-demand control of vision augmentations such as optical zoom or field of view expansion. Physical modification of the device's configuration can be accomplished on the fly using interchangeable camera-lens modules that provide different types of vision enhancements. We implement and test functionality for several primary configurations using telescopic and fisheye camera-lens systems, though many other customizations are possible. We also implement a number of eye-based interactions in order to engage and control the vision augmentations in real time, and explore different methods for merging streams of augmented vision into the user's normal field of view. In a series of experiments, we conduct an in depth analysis of visual acuity and head and eye movement during search and recognition tasks. Results show that methods with larger field of view that utilize binary on/off and gradual zoom mechanisms outperform snapshot and sub-windowed methods and that type of eye engagement has little effect on performance.

Corneal-Imaging Calibration for Optical See-Through Head-Mounted Displays.

In recent years optical see-through head-mounted displays (OST-HMDs) have moved from conceptual research to a market of mass-produced devices with new models and applications being released continuously. It remains challenging to deploy augmented reality (AR) applications that require consistent spatial visualization. Examples include maintenance, training and medical tasks, as the view of the attached scene camera is shifted from the user's view. A calibration step can compute the relationship between the HMD-screen and the user's eye to align the digital content. However, this alignment is only viable as long as the display does not move, an assumption that rarely holds for an extended period of time. As a consequence, continuous recalibration is necessary. Manual calibration methods are tedious and rarely support practical applications. Existing automated methods do not account for user-specific parameters and are error prone. We propose the combination of a pre-calibrated display with a per-frame estimation of the user's cornea position to estimate the individual eye center and continuously recalibrate the system. With this, we also obtain the gaze direction, which allows for instantaneous uncalibrated eye gaze tracking, without the need for additional hardware and complex illumination. Contrary to existing methods, we use simple image processing and do not rely on iris tracking, which is typically noisy and can be ambiguous. Evaluation with simulated and real data shows that our approach achieves a more accurate and stable eye pose estimation, which results in an improved and practical calibration with a largely improved distribution of projection error.