Playing Flappy Bird Based on Motion Recognition Using a Transformer Model and LIDAR Sensor.

A transformer neural network is employed in the present study to predict Q-values in a simulated environment using reinforcement learning techniques. The goal is to teach an agent to navigate and excel in the Flappy Bird game, which became a popular model for control in machine learning approaches. Unlike most top existing approaches that use the game's rendered image as input, our main contribution lies in using sensory input from LIDAR, which is represented by the ray casting method. Specifically, we focus on understanding the temporal context of measurements from a ray casting perspective and optimizing potentially risky behavior by considering the degree of the approach to objects identified as obstacles. The agent learned to use the measurements from ray casting to avoid collisions with obstacles. Our model substantially outperforms related approaches. Going forward, we aim to apply this approach in real-world scenarios.

A Fast and Interactive Augmented Reality System for PET/CT-guided Intervention of Neuroblastoma.

Neuroblastoma is the most common type of extracranial solid tumor in children and can often result in death if not treated. High-intensity focused ultrasound (HIFU) is a non-invasive technique for treating tissue that is deep within the body. It avoids the use of ionizing radiation, avoiding long-term side-effects of these treatments. The goal of this project was to develop the rendering component of an augmented reality (AR) system with potential applications for image-guided HIFU treatment of neuroblastoma. Our project focuses on taking 3D models of neuroblastoma lesions obtained from PET/CT and displaying them in our AR system in near real-time for use by physicians. We used volume ray casting with raster graphics as our preferred rendering method, as it allows for the real-time editing of our 3D radiologic data. Some unique features of our AR system include intuitive hand gestures and virtual user interfaces that allow the user to interact with the rendered data and process PET/CT images for optimal visualization. We implemented the feature to set a custom transfer function, set custom intensity cutoff points, and region-of-interest extraction via cutting planes. In the future, we hope to incorporate this work as part of a complete system for focused ultrasound treatment by adding ultrasound simulation, visualization, and deformable registration.

Gaze-based attention network analysis in a virtual reality classroom.

This article provides a step-by-step guideline for measuring and analyzing visual attention in 3D virtual reality (VR) environments based on eye-tracking data. We propose a solution to the challenges of obtaining relevant eye-tracking information in a dynamic 3D virtual environment and calculating interpretable indicators of learning and social behavior. With a method called "gaze-ray casting," we simulated 3D-gaze movements to obtain information about the gazed objects. This information was used to create graphical models of visual attention, establishing attention networks. These networks represented participants' gaze transitions between different entities in the VR environment over time. Measures of centrality, distribution, and interconnectedness of the networks were calculated to describe the network structure. The measures, derived from graph theory, allowed for statistical inference testing and the interpretation of participants' visual attention in 3D VR environments. Our method provides useful insights when analyzing students' learning in a VR classroom, as reported in a corresponding evaluation article with N = 274 participants. •Guidelines on implementing gaze-ray casting in VR using the Unreal Engine and the HTC VIVE Pro Eye.•Creating gaze-based attention networks and analyzing their network structure.•Implementation tutorials and the Open Source software code are provided via OSF: https://osf.io/pxjrc/?view_only=1b6da45eb93e4f9eb7a138697b941198.

Feasibility and tolerability of ophthalmic virtual reality as a medical communication tool in children and young people.

PURPOSE: Virtual reality (VR) can be useful in explaining diseases and complications that affect children in order to improve medical communications with this vulnerable patient group. So far, children and young people's responses to high-end medical VR environments have never been assessed. METHODS: An unprecedented number of 320 children and young people were given the opportunity to interact with a VR application displaying original ophthalmic volume data via a commercially available tethered head-mounted display (HMD). Participants completed three surveys: demographics and experience with VR, usability and perceived utility of this technology and the Simulator Sickness Questionnaire. The second survey also probed participants for suggestions on improvements and whether this system could be useful for increasing engagement in science. RESULTS: A total of 206 sets of surveys were received. 165 children and young people (84 female) aged 12-18 years (mean, 15 years) completed surveys that could be used for analysis. 69 participants (47.59%) were VR-naïve, and 76 (52.41%) reported that they had previous VR experience. Results show that VR facilitated understanding of ophthalmological complications and was reasonably tolerated. Lastly, exposure to VR raised children and young people's awareness and interest in science. CONCLUSIONS: The VR platform used was successfully utilized and was well accepted in children to display and interact with volume-rendered 3D ophthalmological data. Virtual reality (VR) is suitable as a novel image display platform in ophthalmology to engage children and young people.

An active constraint approach to identify essential spectral information in noisy data.

Multivariate curve resolution (MCR) methods aim at extracting pure component profiles from mixed spectral data and can be applied to high-dimensional data, e.g., from process spectroscopy or hyperspectral imaging techniques. One often observes that some parts of this data, namely certain rows and columns of the data matrix, are considered essential for MCR outcomes, while other parts are of minor importance. Some methods for determining essential data are known, but all have different disadvantages concerning the application for noisy data. This work presents a new approach on how to detect the essential information for noisy, experimental spectral data. Active nonnegativity constraints in combination with duality arguments are the key ingredients for determining essential spectra and frequency channels. The new approach is conceptually simple, computationally cheap and stable with respect to noise. The algorithm is tested for noisy experimental Raman, UV-Vis and FTIR-SEC data.

High-Performance Virtual Reality Volume Rendering of Original Optical Coherence Tomography Point-Cloud Data Enhanced With Real-Time Ray Casting.

PURPOSE: Feasibility testing of a novel volume renders technology to display optical coherence tomography data (OCT) in a virtual reality (VR) environment. METHODS: A VR program was written in C++/OpenGL to import and display volumetric OCT data in real time with 180 frames per second using a high-end computer and a tethered head-mounted display. Following exposure, participants completed a Simulator Sickness Questionnaire (SSQ) to assess for nausea, disorientation, and oculomotor disturbances. A user evaluation study of this software was conducted to explore the potential utility of this application. RESULTS: Fifty-seven subjects completed the user testing (34 males and 23 females). Mean age was 48.5 years (range, 21-77 years). Mean acquired work experience of the 35 ophthalmologists (61.40%) included in the group was 15.46 years (range, 1-37 years). Twenty-nine participants were VR-naïve. The SSQ showed a mean total score of 5.8 (SD = 9.44) indicating that the system was well tolerated and produced minimal side effects. No difference was reported between VR-naïve participants and experienced users. Overall, immersed subjects reported an enjoyable VR-OCT presence effect. CONCLUSIONS: A usable and satisfying VR imaging technique was developed to display and interact with original OCT data. TRANSLATIONAL RELEVANCE: An advanced high-end VR image display method was successfully developed to provide new views and interactions in an ultra high-speed projected digital scenery using point-cloud OCT data. This represents the next generation of OCT image display technology and a new tool for patient engagement, medical education, professional training, and telecommunications.

A novel approach to segmentation and measurement of medical image using level set methods.

The study proposes a novel approach for segmentation and visualization plus value-added surface area and volume measurements for brain medical image analysis. The proposed method contains edge detection and Bayesian based level set segmentation, surface and volume rendering, and surface area and volume measurements for 3D objects of interest (i.e., brain tumor, brain tissue, or whole brain). Two extensions based on edge detection and Bayesian level set are first used to segment 3D objects. Ray casting and a modified marching cubes algorithm are then adopted to facilitate volume and surface visualization of medical-image dataset. To provide physicians with more useful information for diagnosis, the surface area and volume of an examined 3D object are calculated by the techniques of linear algebra and surface integration. Experiment results are finally reported in terms of 3D object extraction, surface and volume rendering, and surface area and volume measurements for medical image analysis.