Integrating Expert Knowledge with Deep Learning Improves QSAR Models for CADD Modeling.

In recent years several applications of graph neural networks (GNNs) to molecular tasks have emerged. Whether GNNs outperform the traditional descriptor-based methods in the quantitative structure activity relationship (QSAR) modeling in early computer-aided drug discovery (CADD) remains an open question. This paper introduces a simple yet effective strategy to boost the predictive power of QSAR deep learning models. The strategy proposes to train GNNs together with traditional descriptors, combining the strengths of both methods. The enhanced model consistently outperforms vanilla descriptors or GNN methods on nine well-curated high throughput screening datasets over diverse therapeutic targets.

Evaluating Augmented Reality Landmark Cues and Frame of Reference Displays with Virtual Reality.

Daily travel usually demands navigation on foot across a variety of different application domains, including tasks like search and rescue or commuting. Head-mounted augmented reality (AR) displays provide a preview of future navigation systems on foot, but designing them is still an open problem. In this paper, we look at two choices that such AR systems can make for navigation: 1) whether to denote landmarks with AR cues and 2) how to convey navigation instructions. Specifically, instructions can be given via a head-referenced display (screen-fixed frame of reference) or by giving directions fixed to global positions in the world (world-fixed frame of reference). Given limitations with the tracking stability, field of view, and brightness of most currently available head-mounted AR displays for lengthy routes outdoors, we decided to simulate these conditions in virtual reality. In the current study, participants navigated an urban virtual environment and their spatial knowledge acquisition was assessed. We experimented with whether or not landmarks in the environment were cued, as well as how navigation instructions were displayed (i.e., via screen-fixed or world-fixed directions). We found that the world-fixed frame of reference resulted in better spatial learning when there were no landmarks cued; adding AR landmark cues marginally improved spatial learning in the screen-fixed condition. These benefits in learning were also correlated with participants' reported sense of direction. Our findings have implications for the design of future cognition-driven navigation systems.

Perceiving distance in virtual reality: theoretical insights from contemporary technologies.

Decades of research have shown that absolute egocentric distance is underestimated in virtual environments (VEs) when compared with the real world. This finding has implications on the use of VEs for applications that require an accurate sense of absolute scale. Fortunately, this underperception of scale can be attenuated by several factors, making perception more similar to (but still not the same as) that of the real world. Here, we examine these factors as two categories: (i) experience inherent to the observer, and (ii) characteristics inherent to the display technology. We analyse how these factors influence the sources of information for absolute distance perception with the goal of understanding how the scale of virtual spaces is calibrated. We identify six types of cues that change with these approaches, contributing both to a theoretical understanding of depth perception in VEs and a call for future research that can benefit from changing technologies. This article is part of the theme issue 'New approaches to 3D vision'.

The Impact of Embodiment and Avatar Sizing on Personal Space in Immersive Virtual Environments.

In this paper, we examine how embodiment and manipulation of a self-avatar's dimensions - specifically the arm length - affect users' judgments of the personal space around them in an immersive virtual environment. In the real world, personal space is the immediate space around the body in which physical interactions are possible. Personal space is increasingly studied in virtual environments because of its importance to social interactions. Here, we specifically look at two components of personal space, interpersonal and peripersonal space, and how they are affected by embodiment and the sizing of a self-avatar. We manipulated embodiment, hypothesizing that higher levels of embodiment will result in larger measures of interpersonal space and smaller measures of peripersonal space. Likewise, we manipulated the arm length of a self-avatar, hypothesizing that while interpersonal space would change with changing arm length, peripersonal space would not. We found that the representation of both interpersonal and peripersonal space change when the user experiences differing levels of embodiment in accordance with our hypotheses, and that only interpersonal space was sensitive to changes in the dimensions of a self-avatar's arms. These findings provide increased understanding of the role of embodiment and self-avatars in the regulation of personal space, and provide foundations for improved design of social interaction in virtual environments.

Shedding Light on Cast Shadows: An Investigation of Perceived Ground Contact in AR and VR.

Virtual objects in augmented reality (AR) often appear to float atop real world surfaces, which makes it difficult to determine where they are positioned in space. This is problematic as many applications for AR require accurate spatial perception. In the current study, we examine how the way we render cast shadows-which act as an important monocular depth cue for creating a sense of contact between an object and the surface beneath it-impacts spatial perception. Over two experiments, we evaluate people's sense of surface contact given both traditional and non-traditional shadow shading methods in optical see-through augmented reality (OST AR), video see-through augmented reality (VST AR), and virtual reality (VR) head-mounted displays. Our results provide evidence that nontraditional shading techniques for rendering shadows in AR displays may enhance the accuracy of one's perception of surface contact. This finding implies a possible tradeoff between photorealism and accuracy of depth perception, especially in OST AR displays. However, it also supports the use of more stylized graphics like non-traditional cast shadows to improve perception and interaction in AR applications.

Altered Peripersonal Space and the Bodily Self in Schizophrenia: A Virtual Reality Study.

Self-disturbances such as an anomalous perception of one's own body boundary are central to the phenomenology of schizophrenia (SZ), but measuring the spatial parameters of the hypothesized self-other boundary has proved to be challenging. Peripersonal space (PPS) refers to the immediate zone surrounding the body where the self interacts physically with the environment; the space that corresponds to hypothesized self-other boundary. PPS is represented by enhanced multisensory integration and faster reaction time (RT) for objects near the body. Thus, multisensory RT tasks can be used to estimate self-other boundary. We aimed to quantify PPS in SZ using an immersive virtual reality visuotactile RT paradigm. Twenty-four participants with SZ and 24 demographically matched controls (CO) were asked to detect tactile vibration while watching a ball approaching them, thrown by either a machine (nonsocial condition) or an avatar (social condition). Parameters of PPS were estimated from the midpoint of the spatial range where the tactile RT decreased most rapidly (size) and the gradient of the RT change at this midpoint (slope). Overall, PPS was smaller in participants with SZ compared with CO. PPS slope for participants with SZ was shallower than CO in the social but not in nonsocial condition, indicating an increased uncertainty of self-other boundary across an extended zone in SZ. Social condition also increased false alarms for tactile detection in SZ. Clinical symptoms were not clearly associated with PPS parameters. These findings suggest the context-dependent nature of weakened body boundary in SZ and underscore the importance of reconciliating objective and subjective aspects of self-disturbances.

Emergency Clinical Procedure Detection With Deep Learning.

Information about a patient's state is critical for hospitals to provide timely care and treatment. Prior work on improving the information flow from emergency medical services (EMS) to hospitals demonstrated the potential of using automated algorithms to detect clinical procedures. However, prior work has not made effective use of video sources that might be available during patient care. In this paper we explore the use convolutional neural networks (CNNs) on raw video data to determine how well video data alone can automatically identify clinical procedures. We apply multiple deep learning models to this problem, with significant variation in results. Our findings indicate performance improvements compared to prior work, but also indicate a need for more training data to reach clinically deployable levels of success.

Interpersonal Affordances and Social Dynamics in Collaborative Immersive Virtual Environments: Passing Together Through Apertures.

An essential question in understanding how to develop and build collaborative immersive virtual environments (IVEs) is recognizing how people perform actions together. Many actions in the real world require that people act without prior planning, and these actions are executed quite successfully. In this paper, we study the common action of two people passing through an aperture together in both the real world (Experiment 1) and in a distributed, collaborative IVE (Experiment 2). The aperture's width is varied from too narrow to be passable to so wide as to be easily passable by both participants together simultaneously. We do this in the real world for all possible gender-based pairings. In virtual reality, however, there is potential for the gender of the participant and the gender of the self-avatar to be different. We also investigate the joint action for all possible gender-based pairings in the distributed IVE. Results indicated that, in the real world, social dynamics between gendered pairings emerged; male-male pairings refused to concede to one another until absolutely necessary while other pairings did not. Male-female pairings were most likely to provide ample space to one another during passage. These behaviors seemed not to appear in the IVE, and avatar gender across all pairings generated no significant behavioral differences. In addition, participants tended to require wider gaps to allow for passage in the IVE. These findings establish base knowledge of social dynamics and affordance behaviors within multi-user IVEs.

Automatic Clinical Procedure Detection for Emergency Services.

Understanding a patient's state is critical to providing optimal care. However, information loss occurs during patient hand-offs (e.g., emergency services (EMS) transferring patient care to a receiving hospital), which hinders care quality. Augmenting the information flow from an EMS vehicle to a receiving hospital may reduce information loss and improve patient outcomes. Such augmentation requires a noninvasive system that can automatically recognize clinical procedures being performed and send near real-time information to a receiving hospital. An automatic clinical procedure detection system that uses wearable sensors, video, and machine-learning to recognize clinical procedures within a controlled environment is presented. The system demonstrated how contextual information and a majority vote method can substantially improve procedure recognition accuracy. Future work concerning computer vision techniques and deep learning are discussed.

Mind the Gap: Gap Affordance Judgments of Children, Teens, and Adults in an Immersive Virtual Environment.

Affordances are possibilities for action that depend on both an observer's capabilities and the properties of the environment. Immersive Virtual Environments (IVEs) have been used to examine affordances in adults, demonstrating that judgments about action capabilities are made similarly to the real world. However, less is known about affordance judgments in middle-aged children and adolescents in IVEs. Differences in rate of growth, decision criteria, and perceived risk could influence affordance judgments for children. In Experiment 1, children, teens, and adults stood in an IVE at ground level or at a height of 15 m, and were asked to view gaps of different widths. Across all age groups, estimates of gap crossing were underestimated at the higher height compared to the ground, consistent with reports of fear and risk of falling. Children, compared to adults, underestimated their maximum crossable gap compared to their actual crossable gap. To test whether this difference was specific to IVEs or a more generalized age effect, children and adults were tested on gap estimates in the real world in Experiment 2. This real world study showed no difference between children and adults, suggesting a unique contribution of the IVE to children's affordance judgments. We discuss the implications for using IVEs to study children's affordances.

Acquisition and transfer of spatial knowledge during wayfinding.

In the current study, we investigated the ways in which the acquisition and transfer of spatial knowledge were affected by (a) the type of spatial relations predominately experienced during learning (routes determined by walkways vs. straight-line paths between locations); (b) environmental complexity; and (c) the availability of rotational body-based information. Participants learned the layout of a virtual shopping mall by repeatedly searching for target storefronts located in 1 of the buildings. We created 2 novel learning conditions to encourage participants to use either route knowledge (paths on walkways between buildings) or survey knowledge (straight-line distances and directions from storefront to storefront) to find the target, and measured the development of route and survey knowledge in both learning conditions. Environmental complexity was manipulated by varying the alignment of the buildings with the enclosure, and the visibility within space. Body-based information was manipulated by having participants perform the experiment in front of a computer monitor or using a head-mounted display. After navigation, participants pointed to various storefronts from a fixed position and orientation. Results showed that the frequently used spatial knowledge could be developed similarly across environments with different complexities, but the infrequently used spatial knowledge was less developed in the complex environment. Furthermore, rotational body-based information facilitated spatial learning under certain conditions. Our results suggest that path integration may play an important role in spatial knowledge transfer, both from route to survey knowledge (cognitive map construction), and from survey to route knowledge (using cognitive map to guide wayfinding). (PsycINFO Database Record (c) 2019 APA, all rights reserved).

Locomotive Recalibration and Prism Adaptation of Children and Teens in Immersive Virtual Environments.

As virtual reality expands in popularity, an increasingly diverse audience is gaining exposure to immersive virtual environments (IVEs). A significant body of research has demonstrated how perception and action work in such environments, but most of this work has been done studying adults. Less is known about how physical and cognitive development affect perception and action in IVEs, particularly as applied to preteen and teenage children. Accordingly, in the current study we assess how preteens (children aged 8-12 years) and teenagers (children aged 15-18 years) respond to mismatches between their motor behavior and the visual information presented by an IVE. Over two experiments, we evaluate how these individuals recalibrate their actions across functionally distinct systems of movement. The first experiment analyzed forward walking recalibration after exposure to an IVE with either increased or decreased visual flow. Visual flow during normal bipedal locomotion was manipulated to be either twice or half as fast as the physical gait. The second experiment leveraged a prism throwing adaptation paradigm to test the effect of recalibration on throwing movement. In the first experiment, our results show no differences across age groups, although subjects generally experienced a post-exposure effect of shortened distance estimation after experiencing visually faster flow and longer distance estimation after experiencing visually slower flow. In the second experiment, subjects generally showed the typical prism adaptation behavior of a throwing after-effect error. The error lasted longer for preteens than older children. Our results have implications for the design of virtual systems with children as a target audience.

Does direction of walking impact binocular rivalry between competing patterns of optic flow?

When dissimilar monocular images are viewed simultaneously by the two eyes, stable binocular vision gives way to unstable vision characterized by alternations in dominance between the two images in a phenomenon called binocular rivalry. These alternations in perception reveal the existence of inhibitory interactions between neural representations associated with conflicting visual inputs. Binocular rivalry has been studied since the days of Wheatstone, but one recent strategy is to investigate its susceptibility to influences caused by one's own motor activity. This paper focused on the activity of walking, which produces an expected, characteristic direction of optic flow dependent upon the direction of one's walking. In a set of experiments, we employed virtual reality technology to present dichoptic stimuli to observers who walked forward, backward, or were sitting. Optic flow was presented to a given eye, and was sometimes congruent with the direction of walking, sometimes incongruent, and sometimes random, except when the participant was sitting. Our results indicate that, while walking had a reliable influence on rivalry dynamics, the predominance of congruent or incongruent motion did not.

Individual differences in using geometric and featural cues to maintain spatial orientation: cue quantity and cue ambiguity are more important than cue type.

Two experiments explored the role of environmental cues in maintaining spatial orientation (sense of self-location and direction) during locomotion. Of particular interest was the importance of geometric cues (provided by environmental surfaces) and featural cues (nongeometric properties provided by striped walls) in maintaining spatial orientation. Participants performed a spatial updating task within virtual environments containing geometric or featural cues that were ambiguous or unambiguous indicators of self-location and direction. Cue type (geometric or featural) did not affect performance, but the number and ambiguity of environmental cues did. Gender differences, interpreted as a proxy for individual differences in spatial ability and/or experience, highlight the interaction between cue quantity and ambiguity. When environmental cues were ambiguous, men stayed oriented with either one or two cues, whereas women stayed oriented only with two. When environmental cues were unambiguous, women stayed oriented with one cue.

The shape of human navigation: how environmental geometry is used in maintenance of spatial orientation.

The role of environmental geometry in maintaining spatial orientation was measured in immersive virtual reality using a spatial updating task (requiring maintenance of orientation during locomotion) within rooms varying in rotational symmetry (the number of room orientations providing the same perspective). Spatial updating was equally good in trapezoidal, rectangular and square rooms (one-fold, two-fold and four-fold rotationally symmetric, respectively) but worse in a circular room (infinity-fold rotationally symmetric). This contrasts with reorientation performance, which was incrementally impaired by increasing rotational symmetry. Spatial updating performance in a shape-changing room (containing visible corners and flat surfaces, but changing its shape over time) was no better than performance in a circular room, indicating that superior spatial updating performance in angular environments was due to remembered room shape, rather than improved self-motion perception in the presence of visible corners and flat surfaces.

Intersurgeon variability in the selection of anterior and posterior commissures and its potential effects on target localization.

BACKGROUND: This study reports the intersurgeon variability in manual selection of the anterior and posterior commissures (AC and PC). The study also investigates the effect of this variability on the localization of targets like the subthalamic nucleus, ventralis intermedius nucleus and globus pallidus internus. The additional effect of variation in the selection of the mid-plane on target localization is also evaluated. METHODS: 43 neurosurgeons (38 attendings, 5 residents/ fellows) were asked to select the AC and the PC points (as routinely used for stereotactic neurosurgical planning) on two MRI scans. The corresponding mid-commissural points (MCPs) and target coordinates were calculated. RESULTS: The collected data show that the MCP is more reliable than either the AC or the PC points. These data also show that, even for experienced neurosurgeons, variations in selecting the AC and the PC point result in substantial variations at the target points: 1.15 +/- 0.89 mm, 1.45 +/- 1.25 mm, 1.21 +/- 0.83 for the subthalamic nucleus, ventralis intermedius nucleus, and globus pallidus internus, respectively, for the first MRI volumeand 1.08 +/- 1.37 mm, 1.35 +/- 1.71 mm, 1.12 +/- 1.17 mm for the same structures for the second volume. These variations are larger when residents/fellows are included in the data set. CONCLUSIONS: The data collected in this study highlight the difficulty in establishing a common reference system that can be used to communicate target location across sites. It indicates the need for the development and evaluation of alternative normalization methods that would permit specifying targets directly in image coordinates or the development of improved imaging techniques that would permit direct targeting.

Tabletop computed lighting for practical digital photography.

We apply simplified image-based lighting methods to reduce the equipment, cost, time, and specialized skills required for high-quality photographic lighting of desktop-sized static objects such as museum artifacts. We place the object and a computer-steered moving-head spotlight inside a simple foam-core enclosure and use a camera to record photos as the light scans the box interior. Optimization, guided by interactive user sketching, selects a small set of these photos whose weighted sum best matches the user-defined target sketch. Unlike previous image-based relighting efforts, our method requires only a single area light source, yet it can achieve high-resolution light positioning to avoid multiple sharp shadows. A reduced version uses only a handheld light and may be suitable for battery-powered field photography equipment that fits into a backpack.

NAV: a tool for producing presentation quality animations of graphical cognitive model dynamics.

Computational models of cognition often exhibit complex dynamics that are difficult to discern without the use of visualization tools. Current tools often provide insight only to the modeling expert, however, and they provide limited functionality for communicating model dynamics to the nonexpert, as is needed during scientific presentations and in educational settings. We present NAV, the Node Activity Visualizer, an easy-to-use and portable software tool that interactively transforms the output of cognitive modeling simulators into presentation quality animations of model performance.

Three-dimensional visualization of protein expression in mouse brain structures using imaging mass spectrometry.

We have developed a method to visualize matrix-assisted laser desorption ionization imaging mass spectrometry (MALDI IMS) data aligned with optically determinable tissue structures in three dimensions. Details of the methodology are exemplified using the 3-D reconstruction of myelin basic protein (MBP) in the corpus callosum of a mouse brain. In this procedure, optical images obtained from serial coronal sections are first aligned to each other to reconstruct a surface of the corpus callosum from segmented contours of the aligned images. The MALDI IMS data are then coregistered to the optical images and superimposed into the surface to create the final 3-D visualization. Correlating proteomic data with anatomical structures provides a more comprehensive understanding of healthy and pathological brain functions, and holds promise to be utilized in more complex anatomical arrangements.