C.DOT - Convolutional Deep Object Tracker for Augmented Reality Based Purely on Synthetic Data.

Augmented reality applications use object tracking to estimate the pose of a camera and to superimpose virtual content onto the observed object. Today, a number of tracking systems are available, ready to be used in industrial applications. However, such systems are hard to handle for a service maintenance engineer, due to obscure configuration procedures. In this article, we investigate options towards replacing the manual configuration process with a machine learning approach based on automatically synthesized data. We present an automated process of creating object tracker facilities exclusively from synthetic data. The data is highly enhanced to train a convolutional neural network, while still being able to receive reliable and robust results during real world applications only from simple RGB cameras. Comparison against related work using the LINEMOD dataset showed that we are able to outperform similar approaches. For our intended industrial applications with high accuracy demands, its performance is still lower than common object tracking methods with manual configuration. Yet, it can greatly support those as an add-on during initialization, due to its higher reliability.

Ethical and Social Aspects of Neurorobotics.

The interdisciplinary field of neurorobotics looks to neuroscience to overcome the limitations of modern robotics technology, to robotics to advance our understanding of the neural system's inner workings, and to information technology to develop tools that support those complementary endeavours. The development of these technologies is still at an early stage, which makes them an ideal candidate for proactive and anticipatory ethical reflection. This article explains the current state of neurorobotics development within the Human Brain Project, originating from a close collaboration between the scientific and technical experts who drive neurorobotics innovation, and the humanities and social sciences scholars who provide contextualising and reflective capabilities. This article discusses some of the ethical issues which can reasonably be expected. On this basis, the article explores possible gaps identified within this collaborative, ethical reflection that calls for attention to ensure that the development of neurorobotics is ethically sound and socially acceptable and desirable.

Serious Games for Nutritional Education: Online Survey on Preferences, Motives, and Behaviors Among Young Adults at University.

BACKGROUND: Data on nutritional information and digital gameplay are limited among young adults in Germany. OBJECTIVE: This survey aimed to gather data on nutritional information sources and digital games for nutritional education (preferences, motives, and behaviors) among young adults at both Munich universities in Germany. METHODS: An online survey was developed by an multidisciplinary research group using EvaSys, an in-house survey software. The questionnaire (47 items) covered questions about baseline characteristics (eg, housing situation and weight), nutrition (eg, nutritional information sources), and digital (nutritional) gameplay (eg, preferences, motives, and behaviors). A feedback field was also provided. This publication is based on a selection of 20 questions (7 baseline characteristics, 2 nutrition, 11 gameplay). Young adults, primarily Munich university students aged from 18 to 24 years, were invited to participate by digital and nondigital communication channels between 2016 and 2017. Statistical analyses were performed using Excel 2013 (Microsoft Corp) and R version 3.1.3 (R Foundation for Statistical Computing). RESULTS: In total, 468 young adults (342/468, 73.1% women; 379/468, 81.0% university students) participated. Most of the participants (269/468, 57.5%) were aged 18 to 24 years with a BMI in the normal weight range (346/447, 77.4%). Mean body weight was 65.5 [SD 14.0] kg. Most participants reported getting nutritional information from the internet (372/467, 79.7%) and printed media (298/467, 63.8%), less than 1.0% (2/467, 0.4%) named digital games. Apps (100/461, 21.7%) and university/workplace (146/461, 31.7%) were the most desired sources for additional information about nutrition, while 10.0% (46/461, 10.0%) of participants stated wanting digital games. Almost two-thirds (293/468, 62.6%) of participants played digital games, while one-fifth (97/456, 21.3%) played digital games daily using smartphones or tablets. Finally, most respondents (343/468, 73.3%), mainly women, expressed interest in obtaining nutritional information during digital gameplay. However, significant gender differences were shown for nutritional acquisition behaviors and digital gameplay preferences, motives, and behaviors. CONCLUSIONS: Our survey population reported playing digital games (especially men) and wanting nutritional information during digital gameplay (especially women). Furthermore, university or workplace are named as preferred settings for nutritional information. Therefore, a digital game app might have the potential to be a tool for nutritional education among young adults within the university or workplace environment.

Short-Term Effects of the Serious Game "Fit, Food, Fun" on Nutritional Knowledge: A Pilot Study among Children and Adolescents.

"Serious games" are a novel and entertaining approach for nutritional education. The aim of this pilot study was to evaluate the short-term effectiveness of "Fit, Food, Fun" (FFF), a serious game to impart nutritional knowledge among children and adolescents. Data collection was conducted at two secondary schools in Bavaria, Germany. The gameplay intervention (gameplay group; GG) consisted of a 15-minute FFF gameplay session during each of three consecutive days. The teaching intervention (teaching group; TG) was performed in a classic lecture format. Nutritional knowledge was evaluated via questionnaires at baseline and post-intervention. Statistical analyses were performed using R (R Core Team, 2018). In total, baseline data were available for 39 participants in the GG and 44 participants in the TG. The mean age was 13.5 ± 0.7 years in the GG and 12.8 ± 0.9 years in the TG. There was a significant (p-value < 0.001) improvement in nutritional knowledge in both intervention groups. Moreover, a between-group difference with a significantly (p-value = 0.01) higher increase in nutritional knowledge was detected for the TG. This pilot study provides evidence for the short-term effectiveness of both educational interventions on the improvement in nutritional knowledge. Finally, the FFF game might be an adequate educational tool for the transfer of nutritional knowledge among children and adolescents.

Connecting Artificial Brains to Robots in a Comprehensive Simulation Framework: The Neurorobotics Platform.

Combined efforts in the fields of neuroscience, computer science, and biology allowed to design biologically realistic models of the brain based on spiking neural networks. For a proper validation of these models, an embodiment in a dynamic and rich sensory environment, where the model is exposed to a realistic sensory-motor task, is needed. Due to the complexity of these brain models that, at the current stage, cannot deal with real-time constraints, it is not possible to embed them into a real-world task. Rather, the embodiment has to be simulated as well. While adequate tools exist to simulate either complex neural networks or robots and their environments, there is so far no tool that allows to easily establish a communication between brain and body models. The Neurorobotics Platform is a new web-based environment that aims to fill this gap by offering scientists and technology developers a software infrastructure allowing them to connect brain models to detailed simulations of robot bodies and environments and to use the resulting neurorobotic systems for in silico experimentation. In order to simplify the workflow and reduce the level of the required programming skills, the platform provides editors for the specification of experimental sequences and conditions, environments, robots, and brain-body connectors. In addition to that, a variety of existing robots and environments are provided. This work presents the architecture of the first release of the Neurorobotics Platform developed in subproject 10 "Neurorobotics" of the Human Brain Project (HBP). At the current state, the Neurorobotics Platform allows researchers to design and run basic experiments in neurorobotics using simulated robots and simulated environments linked to simplified versions of brain models. We illustrate the capabilities of the platform with three example experiments: a Braitenberg task implemented on a mobile robot, a sensory-motor learning task based on a robotic controller, and a visual tracking embedding a retina model on the iCub humanoid robot. These use-cases allow to assess the applicability of the Neurorobotics Platform for robotic tasks as well as in neuroscientific experiments.

Gaussian Light Field: Estimation of Viewpoint-Dependent Blur for Optical See-Through Head-Mounted Displays.

We propose a method to calibrate viewpoint-dependent, channel-wise image blur of near-eye displays, especially of Optical See-Through Head-Mounted Displays (OST-HMDs). Imperfections in HMD optics cause channel-wise image shift and blur that degrade the image quality of the display at a user's viewpoint. If we can estimate such characteristics perfectly, we could mitigate the effect by applying correction techniques from the computational photography in computer vision as analogous to cameras. Unfortunately, directly applying existing calibration techniques of cameras to OST-HMDs is not a straightforward task. Unlike ordinary imaging systems, image blur in OST-HMDs is viewpoint-dependent, i.e., the optical characteristic of a display dynamically changes depending on the current viewpoint of the user. This constraint makes the problem challenging since we must measure image blur of an HMD, ideally, over the entire 3D eyebox in which a user can see an image. To overcome this problem, we model the viewpoint-dependent blur as a Gaussian Light Field (GLF) that stores spatial information of the display screen as a (4D) light field with depth information and the blur as point-spread functions in the form of Gaussian kernels, respectively. We first describe both our GLF model and a calibration procedure to learn a GLF for a given OST-HMD. We then apply our calibration method to two HMDs that use different optics: a cubic prism or holographic gratings. The results show that our method achieves significantly better accuracy in Point-Spread Function (PSF) estimations with an accuracy about 2 to 7 dB in Peak SNR.

Sticky Projections-A Model-Based Approach to Interactive Shader Lamps Tracking.

Shader lamps can augment physical objects with projected virtual replications using a camera-projector system, provided that the physical and virtual object are well registered to each other. Precise registration and tracking has been a cumbersome and intrusive process in the past. In this paper, we present a new method for tracking complex-shaped physical objects interactively. In contrast to previous approaches our system is mobile and makes solely use of the projection of the virtual replication to track the physical object and "stick" the projection to it. Our method consists of two stages, a fast pose initialization based on structured light patterns and a non-intrusive frame-by-frame tracking based on features detected in the projection. During the tracking phase, a radiometrically corrected virtual camera view based on the current pose prediction is rendered and compared to the captured image. Matched features are triangulated providing a sparse set of surface points that is robustly aligned to the virtual model. The alignment transformation serves as an input for the new pose prediction. Detailed experiments including the evaluation of the overlay accuracy show that our approach can accurately and robustly track complex objects at interactive rates.

On-site semi-automatic calibration and registration of a projector-camera system using arbitrary objects with known geometry.

In the Shader Lamps concept, a projector-camera system augments physical objects with projected virtual textures, provided that a precise intrinsic and extrinsic calibration of the system is available. Calibrating such systems has been an elaborate and lengthy task in the past and required a special calibration apparatus. Self-calibration methods in turn are able to estimate calibration parameters automatically with no effort. However they inherently lack global scale and are fairly sensitive to input data. We propose a new semi-automatic calibration approach for projector-camera systems that - unlike existing auto-calibration approaches - additionally recovers the necessary global scale by projecting on an arbitrary object of known geometry. To this end our method combines surface registration with bundle adjustment optimization on points reconstructed from structured light projections to refine a solution that is computed from the decomposition of the fundamental matrix. In simulations on virtual data and experiments with real data we demonstrate that our approach estimates the global scale robustly and is furthermore able to improve incorrectly guessed intrinsic and extrinsic calibration parameters thus outperforming comparable metric rectification algorithms.

Semi-parametric color reproduction method for optical see-through head-mounted displays.

The fundamental issues in Augmented Reality (AR) are on how to naturally mediate the reality with virtual content as seen by users. In AR applications with Optical See-Through Head-Mounted Displays (OST-HMD), the issues often raise the problem of rendering color on the OST-HMD consistently to input colors. However, due to various display constraints and eye properties, it is still a challenging task to indistinguishably reproduce the colors on OST-HMDs. An approach to solve this problem is to pre-process the input color so that a user perceives the output color on the display to be the same as the input. We propose a color calibration method for OST-HMDs. We start from modeling the physical optics in the rendering and perception process between the HMD and the eye. We treat the color distortion as a semi-parametric model which separates the non-linear color distortion and the linear color shift. We demonstrate that calibrated images regain their original appearance on two OST-HMD setups with both synthetic and real datasets. Furthermore, we analyze the limitations of the proposed method and remaining problems of the color reproduction in OST-HMDs. We then discuss how to realize more practical color reproduction methods for future HMD-eye system.

Light-Field Correction for Spatial Calibration of Optical See-Through Head-Mounted Displays.

A critical requirement for AR applications with Optical See-Through Head-Mounted Displays (OST-HMD) is to project 3D information correctly into the current viewpoint of the user - more particularly, according to the user's eye position. Recently-proposed interaction-free calibration methods [16], [17] automatically estimate this projection by tracking the user's eye position, thereby freeing users from tedious manual calibrations. However, the method is still prone to contain systematic calibration errors. Such errors stem from eye-/HMD-related factors and are not represented in the conventional eye-HMD model used for HMD calibration. This paper investigates one of these factors - the fact that optical elements of OST-HMDs distort incoming world-light rays before they reach the eye, just as corrective glasses do. Any OST-HMD requires an optical element to display a virtual screen. Each such optical element has different distortions. Since users see a distorted world through the element, ignoring this distortion degenerates the projection quality. We propose a light-field correction method, based on a machine learning technique, which compensates the world-scene distortion caused by OST-HMD optics. We demonstrate that our method reduces the systematic error and significantly increases the calibration accuracy of the interaction-free calibration.

Corneal-Imaging Calibration for Optical See-Through Head-Mounted Displays.

In recent years optical see-through head-mounted displays (OST-HMDs) have moved from conceptual research to a market of mass-produced devices with new models and applications being released continuously. It remains challenging to deploy augmented reality (AR) applications that require consistent spatial visualization. Examples include maintenance, training and medical tasks, as the view of the attached scene camera is shifted from the user's view. A calibration step can compute the relationship between the HMD-screen and the user's eye to align the digital content. However, this alignment is only viable as long as the display does not move, an assumption that rarely holds for an extended period of time. As a consequence, continuous recalibration is necessary. Manual calibration methods are tedious and rarely support practical applications. Existing automated methods do not account for user-specific parameters and are error prone. We propose the combination of a pre-calibrated display with a per-frame estimation of the user's cornea position to estimate the individual eye center and continuously recalibrate the system. With this, we also obtain the gaze direction, which allows for instantaneous uncalibrated eye gaze tracking, without the need for additional hardware and complex illumination. Contrary to existing methods, we use simple image processing and do not rely on iris tracking, which is typically noisy and can be ambiguous. Evaluation with simulated and real data shows that our approach achieves a more accurate and stable eye pose estimation, which results in an improved and practical calibration with a largely improved distribution of projection error.

Subjective Evaluation of a Semi-Automatic Optical See-Through Head-Mounted Display Calibration Technique.

With the growing availability of optical see-through (OST) head-mounted displays (HMDs) there is a present need for robust, uncomplicated, and automatic calibration methods suited for non-expert users. This work presents the results of a user study which both objectively and subjectively examines registration accuracy produced by three OST HMD calibration methods: (1) SPAAM, (2) Degraded SPAAM, and (3) Recycled INDICA, a recently developed semi-automatic calibration method. Accuracy metrics used for evaluation include subject provided quality values and error between perceived and absolute registration coordinates. Our results show all three calibration methods produce very accurate registration in the horizontal direction but caused subjects to perceive the distance of virtual objects to be closer than intended. Surprisingly, the semi-automatic calibration method produced more accurate registration vertically and in perceived object distance overall. User assessed quality values were also the highest for Recycled INDICA, particularly when objects were shown at distance. The results of this study confirm that Recycled INDICA is capable of producing equal or superior on-screen registration compared to common OST HMD calibration methods. We also identify a potential hazard in using reprojection error as a quantitative analysis technique to predict registration accuracy. We conclude with discussing the further need for examining INDICA calibration in binocular HMD systems, and the present possibility for creation of a closed-loop continuous calibration method for OST Augmented Reality.

Precise Haptic Device Co-Location for Visuo-Haptic Augmented Reality.

Visuo-haptic augmented reality systems enable users to see and touch digital information that is embedded in the real world. PHANToM haptic devices are often employed to provide haptic feedback. Precise co-location of computer-generated graphics and the haptic stylus is necessary to provide a realistic user experience. Previous work has focused on calibration procedures that compensate the non-linear position error caused by inaccuracies in the joint angle sensors. In this article we present a more complete procedure that additionally compensates for errors in the gimbal sensors and improves position calibration. The proposed procedure further includes software-based temporal alignment of sensor data and a method for the estimation of a reference for position calibration, resulting in increased robustness against haptic device initialization and external tracker noise. We designed our procedure to require minimal user input to maximize usability. We conducted an extensive evaluation with two different PHANToMs, two different optical trackers, and a mechanical tracker. Compared to state-of-the-art calibration procedures, our approach significantly improves the co-location of the haptic stylus. This results in higher fidelity visual and haptic augmentations, which are crucial for fine-motor tasks in areas such as medical training simulators, assembly planning tools, or rapid prototyping applications.

Beaming into the rat world: enabling real-time interaction between rat and human each at their own scale.

Immersive virtual reality (IVR) typically generates the illusion in participants that they are in the displayed virtual scene where they can experience and interact in events as if they were really happening. Teleoperator (TO) systems place people at a remote physical destination embodied as a robotic device, and where typically participants have the sensation of being at the destination, with the ability to interact with entities there. In this paper, we show how to combine IVR and TO to allow a new class of application. The participant in the IVR is represented in the destination by a physical robot (TO) and simultaneously the remote place and entities within it are represented to the participant in the IVR. Hence, the IVR participant has a normal virtual reality experience, but where his or her actions and behaviour control the remote robot and can therefore have physical consequences. Here, we show how such a system can be deployed to allow a human and a rat to operate together, but the human interacting with the rat on a human scale, and the rat interacting with the human on the rat scale. The human is represented in a rat arena by a small robot that is slaved to the human's movements, whereas the tracked rat is represented to the human in the virtual reality by a humanoid avatar. We describe the system and also a study that was designed to test whether humans can successfully play a game with the rat. The results show that the system functioned well and that the humans were able to interact with the rat to fulfil the tasks of the game. This system opens up the possibility of new applications in the life sciences involving participant observation of and interaction with animals but at human scale.

Online estimation of the target registration error for n-ocular optical tracking systems.

For current surgical navigation systems optical tracking is state of the art. The accuracy of these tracking systems is currently determined statically for the case of full visibility of all tracking targets. We propose a dynamic determination of the accuracy based on the visibility and geometry of the tracking setup. This real time estimation of accuracy has a multitude of applications. For multiple camera systems it allows reducing line of sight problems and guaranteeing a certain accuracy. The visualization of these accuracies allows surgeons to perform the procedures taking to the tracking accuracy into account. It also allows engineers to design tracking setups interactively guaranteeing a certain accuracy. Our model is an extension to the state of the art models of Fitzpatrick et al. and Hoff et al. We model the error in the camera sensor plane. The error is propagated using the internal camera parameter, camera poses, tracking target poses, target geometry and marker visibility, in order to estimate the final accuracy of the tracked instrument.