Visualization for epidemiological modelling: challenges, solutions, reflections and recommendations.

We report on an ongoing collaboration between epidemiological modellers and visualization researchers by documenting and reflecting upon knowledge constructs-a series of ideas, approaches and methods taken from existing visualization research and practice-deployed and developed to support modelling of the COVID-19 pandemic. Structured independent commentary on these efforts is synthesized through iterative reflection to develop: evidence of the effectiveness and value of visualization in this context; open problems upon which the research communities may focus; guidance for future activity of this type and recommendations to safeguard the achievements and promote, advance, secure and prepare for future collaborations of this kind. In describing and comparing a series of related projects that were undertaken in unprecedented conditions, our hope is that this unique report, and its rich interactive supplementary materials, will guide the scientific community in embracing visualization in its observation, analysis and modelling of data as well as in disseminating findings. Equally we hope to encourage the visualization community to engage with impactful science in addressing its emerging data challenges. If we are successful, this showcase of activity may stimulate mutually beneficial engagement between communities with complementary expertise to address problems of significance in epidemiology and beyond. See https://ramp-vis.github.io/RAMPVIS-PhilTransA-Supplement/. This article is part of the theme issue 'Technical challenges of modelling real-life epidemics and examples of overcoming these'.

Immersive Virtual Reality for the Cognitive Rehabilitation of Stroke Survivors.

We present the results of a double-blind phase 2b randomized control trial that used a custom built virtual reality environment for the cognitive rehabilitation of stroke survivors. A stroke causes damage to the brain and problem solving, memory and task sequencing are commonly affected. The brain can recover to some extent, however, and stroke patients have to relearn how to carry out activities of daily living. We have created an application called VIRTUE to enable such activities to be practiced using immersive virtual reality. Gamification techniques enhance the motivation of patients such as by making the level of difficulty of a task increase over time. The design and implementation of VIRTUE is described together with the results of the trial conducted within the Stroke Unit of a large hospital. We report on the safety and acceptability of VIRTUE. We have also observed particular benefits of VR treatment for stroke survivors that experienced more severe cognitive impairment, and an encouraging reduction in time spent in the hospital for all patients that received the VR treatment.

VRIA: A Web-Based Framework for Creating Immersive Analytics Experiences.

We present VRIA, a Web-based framework for creating Immersive Analytics (IA) experiences in Virtual Reality. VRIA is built upon WebVR, A-Frame, React and D3.js, and offers a visualization creation workflow which enables users, of different levels of expertise, to rapidly develop Immersive Analytics experiences for the Web. The use of these open-standards Web-based technologies allows us to implement VR experiences in a browser and offers strong synergies with popular visualization libraries, through the HTML Document Object Model (DOM). This makes VRIA ubiquitous and platform-independent. Moreover, by using WebVR's progressive enhancement, the experiences VRIA creates are accessible on a plethora of devices. We elaborate on our motivation for focusing on open-standards Web technologies, present the VRIA creation workflow and detail the underlying mechanics of our framework. We also report on techniques and optimizations necessary for implementing Immersive Analytics experiences on the Web, discuss scalability implications of our framework, and present a series of use case applications to demonstrate the various features of VRIA. Finally, we discuss current limitations of our framework, the lessons learned from its development, and outline further extensions.

De-smokeGCN: Generative Cooperative Networks for Joint Surgical Smoke Detection and Removal.

Surgical smoke removal algorithms can improve the quality of intra-operative imaging and reduce hazards in image-guided surgery, a highly desirable post-process for many clinical applications. These algorithms also enable effective computer vision tasks for future robotic surgery. In this article, we present a new unsupervised learning framework for high-quality pixel-wise smoke detection and removal. One of the well recognized grand challenges in using convolutional neural networks (CNNs) for medical image processing is to obtain intra-operative medical imaging datasets for network training and validation, but availability and quality of these datasets are scarce. Our novel training framework does not require ground-truth image pairs. Instead, it learns purely from computer-generated simulation images. This approach opens up new avenues and bridges a substantial gap between conventional non-learning based methods and which requiring prior knowledge gained from extensive training datasets. Inspired by the Generative Adversarial Network (GAN), we have developed a novel generative-collaborative learning scheme that decomposes the de-smoke process into two separate tasks: smoke detection and smoke removal. The detection network is used as prior knowledge, and also as a loss function to maximize its support for training of the smoke removal network. Quantitative and qualitative studies show that the proposed training framework outperforms the state-of-the-art de-smoking approaches including the latest GAN framework (such as PIX2PIX). Although trained on synthetic images, experimental results on clinical images have proved the effectiveness of the proposed network for detecting and removing surgical smoke on both simulated and real-world laparoscopic images.

An Information-Theoretic Approach to the Cost-benefit Analysis of Visualization in Virtual Environments.

Visualization and virtual environments (VEs) have been two interconnected parallel strands in visual computing for decades. Some VEs have been purposely developed for visualization applications, while many visualization applications are exemplary showcases in general-purpose VEs. Because of the development and operation costs of VEs, the majority of visualization applications in practice have yet to benefit from the capacity of VEs. In this paper, we examine this status quo from an information-theoretic perspective. Our objectives are to conduct cost-benefit analysis on typical VE systems (including augmented and mixed reality, theater-based systems, and large powerwalls), to explain why some visualization applications benefit more from VEs than others, and to sketch out pathways for the future development of visualization applications in VEs. We support our theoretical propositions and analysis using theories and discoveries in the literature of cognitive sciences and the practical evidence reported in the literatures of visualization and VEs.

SLAM-based dense surface reconstruction in monocular Minimally Invasive Surgery and its application to Augmented Reality.

BACKGROUND AND OBJECTIVE: While Minimally Invasive Surgery (MIS) offers considerable benefits to patients, it also imposes big challenges on a surgeon's performance due to well-known issues and restrictions associated with the field of view (FOV), hand-eye misalignment and disorientation, as well as the lack of stereoscopic depth perception in monocular endoscopy. Augmented Reality (AR) technology can help to overcome these limitations by augmenting the real scene with annotations, labels, tumour measurements or even a 3D reconstruction of anatomy structures at the target surgical locations. However, previous research attempts of using AR technology in monocular MIS surgical scenes have been mainly focused on the information overlay without addressing correct spatial calibrations, which could lead to incorrect localization of annotations and labels, and inaccurate depth cues and tumour measurements. In this paper, we present a novel intra-operative dense surface reconstruction framework that is capable of providing geometry information from only monocular MIS videos for geometry-aware AR applications such as site measurements and depth cues. We address a number of compelling issues in augmenting a scene for a monocular MIS environment, such as drifting and inaccurate planar mapping. METHODS: A state-of-the-art Simultaneous Localization And Mapping (SLAM) algorithm used in robotics has been extended to deal with monocular MIS surgical scenes for reliable endoscopic camera tracking and salient point mapping. A robust global 3D surface reconstruction framework has been developed for building a dense surface using only unorganized sparse point clouds extracted from the SLAM. The 3D surface reconstruction framework employs the Moving Least Squares (MLS) smoothing algorithm and the Poisson surface reconstruction framework for real time processing of the point clouds data set. Finally, the 3D geometric information of the surgical scene allows better understanding and accurate placement AR augmentations based on a robust 3D calibration. RESULTS: We demonstrate the clinical relevance of our proposed system through two examples: (a) measurement of the surface; (b) depth cues in monocular endoscopy. The performance and accuracy evaluations of the proposed framework consist of two steps. First, we have created a computer-generated endoscopy simulation video to quantify the accuracy of the camera tracking by comparing the results of the video camera tracking with the recorded ground-truth camera trajectories. The accuracy of the surface reconstruction is assessed by evaluating the Root Mean Square Distance (RMSD) of surface vertices of the reconstructed mesh with that of the ground truth 3D models. An error of 1.24 mm for the camera trajectories has been obtained and the RMSD for surface reconstruction is 2.54 mm, which compare favourably with previous approaches. Second, in vivo laparoscopic videos are used to examine the quality of accurate AR based annotation and measurement, and the creation of depth cues. These results show the potential promise of our geometry-aware AR technology to be used in MIS surgical scenes. CONCLUSIONS: The results show that the new framework is robust and accurate in dealing with challenging situations such as the rapid endoscopy camera movements in monocular MIS scenes. Both camera tracking and surface reconstruction based on a sparse point cloud are effective and operated in real-time. This demonstrates the potential of our algorithm for accurate AR localization and depth augmentation with geometric cues and correct surface measurements in MIS with monocular endoscopes.

The Implementation and Validation of a Virtual Environment for Training Powered Wheelchair Manoeuvres.

Navigating a powered wheelchair and avoiding collisions is often a daunting task for new wheelchair users. It takes time and practice to gain the coordination needed to become a competent driver and this can be even more of a challenge for someone with a disability. We present a cost-effective virtual reality (VR) application that takes advantage of consumer level VR hardware. The system can be easily deployed in an assessment centre or for home use, and does not depend on a specialized high-end virtual environment such as a Powerwall or CAVE. This paper reviews previous work that has used virtual environments technology for training tasks, particularly wheelchair simulation. We then describe the implementation of our own system and the first validation study carried out using thirty three able bodied volunteers. The study results indicate that at a significance level of 5 percent then there is an improvement in driving skills from the use of our VR system. We thus have the potential to develop the competency of a wheelchair user whilst avoiding the risks inherent to training in the real world. However, the occurrence of cybersickness is a particular problem in this application that will need to be addressed.

Real-time geometry-aware augmented reality in minimally invasive surgery.

The potential of augmented reality (AR) technology to assist minimally invasive surgery (MIS) lies in its computational performance and accuracy in dealing with challenging MIS scenes. Even with the latest hardware and software technologies, achieving both real-time and accurate augmented information overlay in MIS is still a formidable task. In this Letter, the authors present a novel real-time AR framework for MIS that achieves interactive geometric aware AR in endoscopic surgery with stereo views. The authors' framework tracks the movement of the endoscopic camera and simultaneously reconstructs a dense geometric mesh of the MIS scene. The movement of the camera is predicted by minimising the re-projection error to achieve a fast tracking performance, while the three-dimensional mesh is incrementally built by a dense zero mean normalised cross-correlation stereo-matching method to improve the accuracy of the surface reconstruction. The proposed system does not require any prior template or pre-operative scan and can infer the geometric information intra-operatively in real time. With the geometric information available, the proposed AR framework is able to interactively add annotations, localisation of tumours and vessels, and measurement labelling with greater precision and accuracy compared with the state-of-the-art approaches.

A Cost-Effective Virtual Environment for Simulating and Training Powered Wheelchairs Manoeuvres.

Control of a powered wheelchair is often not intuitive, making training of new users a challenging and sometimes hazardous task. Collisions, due to a lack of experience can result in injury for the user and other individuals. By conducting training activities in virtual reality (VR), we can potentially improve driving skills whilst avoiding the risks inherent to the real world. However, until recently VR technology has been expensive and limited the commercial feasibility of a general training solution. We describe Wheelchair-Rift, a cost effective prototype simulator that makes use of the Oculus Rift head mounted display and the Leap Motion hand tracking device. It has been assessed for face validity by a panel of experts from a local Posture and Mobility Service. Initial results augur well for our cost-effective training solution.

Combination of 3D skin surface texture features and 2D ABCD features for improved melanoma diagnosis.

Two-dimensional asymmetry, border irregularity, colour variegation and diameter (ABCD) features are important indicators currently used for computer-assisted diagnosis of malignant melanoma (MM); however, they often prove to be insufficient to make a convincing diagnosis. Previous work has demonstrated that 3D skin surface normal features in the form of tilt and slant pattern disruptions are promising new features independent from the existing 2D ABCD features. This work investigates that whether improved lesion classification can be achieved by combining the 3D features with the 2D ABCD features. Experiments using a nonlinear support vector machine classifier show that many combinations of the 2D ABCD features and the 3D features can give substantially better classification accuracy than using (1) single features and (2) many combinations of the 2D ABCD features. The best 2D and 3D feature combination includes the overall 3D skin surface disruption, the asymmetry and all the three colour channel features. It gives an overall 87.8 % successful classification, which is better than the best single feature with 78.0 % and the best 2D feature combination with 83.1 %. These demonstrate that (1) the 3D features have additive values to improve the existing lesion classification and (2) combining the 3D feature with all the 2D features does not lead to the best lesion classification. The two ABCD features not selected by the best 2D and 3D combination, namely (1) the border feature and (2) the diameter feature, were also studied in separate experiments. It found that inclusion of either feature in the 2D and 3D combination can successfully classify 3 out of 4 lesion groups. The only one group not accurately classified by either feature can be classified satisfactorily by the other. In both cases, they have shown better classification performances than those without the 3D feature in the combinations. This further demonstrates that (1) the 3D feature can be used to improve the existing 2D-based diagnosis and (2) including the 3D feature with subsets of the 2D features can be used in distinguishing different benign lesion classes from MM. It is envisaged that classification performance may be further improved if different 2D and 3D feature subsets demonstrated in this study are used in different stages to target different benign lesion classes in future studies.

VCath: a tablet-based neurosurgery training tool.

VCath is a neurosurgery training tool for the catheterization of the lateral ventricle that has been designed for use on tablet devices. We believe this is the first use of a tablet (iPad) for this purpose and demonstrates future utility for this approach, particularly for Objective Structured Clinical Exams (OSCEs). This paper outlines the implementation and use of VCath.

3D measuring tool for estimating femoroacetabular impingement.

Osteoarthritis of the hip is commonly caused by the repetitive contact between abnormal skeletal prominences between the anterosuperior femoral head-neck junction and the rim of the acetabular socket. Current methods for estimating femoroacetabular impingement by analyzing the sphericity of the femoral head require manual measurements which are both inaccurate and open to interpretation. In this research we provide a prototype software tool for improving this estimation.

A directed particle system for optimised visualization of blood flow in complex networks.

This paper introduces a novel technique for the visualization of blood (or other fluid) flowing through a complex 3D network of vessels. The Directed Particle System (DPS) approach is loosely based on the computer graphics concept of flocking agents. It has been developed and optimised to provide effective real time visualization and qualitative simulation of fluid flow. There are many potential applications of DPS, and one example - a decision support tool for coronary collateralization - is discussed.

Medical mentoring via the evolving world wide web.

OBJECTIVES: Mentoring, for physicians and surgeons in training, is advocated as an essential adjunct in work-based learning, providing support in career and non-career related issues. The World Wide Web (WWW) has evolved, as a technology, to become more interactive and person centric, tailoring itself to the individual needs of the user. This changing technology may open new avenues to foster mentoring in medicine. DESIGN, SYSTEMATIC REVIEW, MAIN OUTCOME MEASURES: A search of the MEDLINE database from 1950 to 2012 using the PubMed interface, combined with manual cross-referencing was performed using the following strategy: ("mentors"[MeSH Terms] OR "mentors"[All Fields] OR "mentor"[All Fields]) AND ("internet"[MeSH Terms] OR "internet"[All Fields]) AND ("medicine"[MeSH Terms] OR "medicine"[All Fields]) AND ("humans"[MeSH Terms] AND English[lang]). Abstracts were screened for relevance (UJ) to the topic; eligibility for inclusion was simply on screening for relevance to online mentoring and web-based technologies. RESULTS: Forty-five papers were found, of which 16 were relevant. All studies were observational in nature. To date, all medical mentoring applications utilizing the World Wide Web have enjoyed some success limited by Web 1.0 and 2.0 technologies. CONCLUSIONS: With the evolution of the WWW through 1.0, 2.0 and 3.0 generations, the potential for meaningful tele- and distance mentoring has greatly improved. Some engagement has been made with these technological advancements, however further work is required to fully realize the potential of these technologies.

Surgical Trainee Opinions in the United Kingdom Regarding a Three-Dimensional Virtual Mentoring Environment (MentorSL) in Second Life: Pilot Study.

BACKGROUND: Medical mentoring is becoming increasingly complex with the evolving needs of trainees and the complexities of their personal and social lives. The Internet is an enabling technology, which increasingly facilitates interaction with multiple people at a distance. Web 2.0 and 3.0 technology shows promise in furthering this facilitation. OBJECTIVE: The objective of our study was to establish opinions among doctors in postgraduate surgical training regarding mentoring and whether these doctors would readily accept virtual mentoring following a brief experience. METHODS: On the 12th of February 2012, an introductory teaching class was arranged by The London Postgraduate School of Surgery for doctors in training. Participants were introduced to a novel virtual mentoring system and asked to complete a questionnaire regarding their opinions before and after the demonstration. RESULTS: A total of 57 junior doctors attended. Among them, 35 completed questionnaires pre- and postdemonstration. Regarding usefulness of a 3D virtual environment for mentoring, 6/35 (17%) agreed or strongly agreed and 20/35 (57%) were unsure prior to the session. Following 20 minutes using MentorSL, this significantly increased to 14/35 (40%) agreeing or strongly agreeing with 11/35 (31%) unsure (P<.001). Prior to using MentorSL, regarding usefulness of voice communication for virtual mentoring, 11/35 (31%) agreed or strongly agreed and 18/35 (51%) were unsure. Following 20 minutes using MentorSL, 19/35 (54%) agreed or strongly agreed and 10/35 (29%) were unsure of usefulness. Regarding ease of use of navigation, search mentor, meeting scheduling, and voice communication features, 17/35 (49%), 13/35 (37%), 15/35 (43%), and 16/35 (46%) participants agreed or strongly agreed, respectively. Regarding usefulness of telementoring, 24/35 (69%) agreed or strongly agreed, increasing to 28/35 (80%) following the introduction. For usefulness of multiple mentors, initially 24/35 (69%) agreed or strongly agreed increasing to 29/35 (83%). For overall satisfaction, 30/35 (86%) reported good or adequate and 19/35 (54%) agreed or strongly agreed with using the system again. CONCLUSIONS: These data suggest that a short introduction on how to use virtual systems may result in significant participation and use of virtual mentoring systems.

A method to compute respiration parameters for patient-based simulators.

We propose a method to automatically tune a patient-based virtual environment training simulator for abdominal needle insertion. The key attributes to be customized in our framework are the elasticity of soft-tissues and the respiratory model parameters. The estimation is based on two 3D Computed Tomography (CT) scans of the same patient at two different time steps. Results are presented on four patients and show that our new method leads to better results than our previous studies with manually tuned parameters.

Realistic visualization of living brain tissue.

This paper presents an advanced method of visualizing the surface appearance of living brain tissue. We have been granted access to the operating theatre during neurosurgical procedures to obtain colour data via calibrated photography of exposed brain tissue. The specular reflectivity of the brain's surface is approximated by analyzing a gelatine layer applied to animal flesh. This provides data for a bidirectional reflectance distribution function (BRDF) that is then used the rendering process. Rendering is achieved in realtime by utilizing the GPU, and includes support for ambient occlusion, advanced texturing, sub surface scattering and specularity. Our goal is to investigate whether realistic visualizations of living anatomy can be produced and so provide added value to anatomy education.

Modification of commercial force feedback hardware for needle insertion simulation.

A SensAble Omni force feedback device has been modified to increase the face validity of a needle insertion simulation. The new end effector uses a real needle hub and shortened needle shaft in place of the Omni's pre-fitted pen shaped end effector. This modification facilitates correct procedural training through the simulation of co-located visual and haptic cues in an augmented reality approach to simulation. The development of the new end effector is described and a pictorial guide to its manufacture and the fitting process is provided. Initial results from face validation studies bode well for the fidelity of this low cost device.