Fast-track virtual reality software to facilitate 3-dimensional reconstruction in congenital heart disease.

OBJECTIVES: Two limitations of the clinical use of 3-dimensional (3D) reconstruction and virtual reality systems are the relatively high cost and the amount of experience required to use hardware and software to effectively explore medical images. We have tried to simplify the process and validate a new tool developed for this purpose with a novel software package. METHODS: Five patients with right partial anomalous pulmonary venous return with adequate preoperative images acquired with magnetic resonance imaging were enrolled. Five volunteers with no previous experience in the field of 3D reconstruction were instructed to use the software after viewing a short video tutorial. Users were then asked to create a 3D model of each patient's heart using DIVA software. Their results were compared quantitatively and qualitatively with a benchmark reconstruction performed by an experienced user. RESULTS: All our participants recreated 3D models in a relatively short time, maintaining a good overall quality (average quality score ≥ 3 on a scale of 1-5). The overall trend of all the parameters analysed showed a statistical improvement between case 1 and case 5, as users became more and more experienced. CONCLUSIONS: DIVA is a simple software program that allows accurate 3D reconstruction in a relatively short time ("fast-track" virtual reality). In this study, we demonstrated the potential use of DIVA by inexperienced users, with a significant improvement in quality and time after a few cases were performed. Further studies are needed to confirm the potential application of this technology on a larger scale.

Breast Magnetic Resonance Image Analysis for Surgeons Using Virtual Reality: A Comparative Study.

PURPOSE: The treatment of breast cancer, the leading cause of cancer and cancer mortality among women worldwide, is mainly on the basis of surgery. In this study, we describe the use of a medical image visualization tool on the basis of virtual reality (VR), entitled DIVA, in the context of breast cancer tumor localization among surgeons. The aim of this study was to evaluate the speed and accuracy of surgeons using DIVA for medical image analysis of breast magnetic resonance image (MRI) scans relative to standard image slice-based visualization tools. MATERIALS AND METHODS: In our study, residents and practicing surgeons used two breast MRI reading modalities: the common slice-based radiology interface and the DIVA system in its VR mode. Metrics measured were compared in relation to postoperative anatomical-pathologic reports. RESULTS: Eighteen breast surgeons from the Institut Curie performed all the analysis presented. The MRI analysis time was significantly lower with the DIVA system than with the slice-based visualization for residents, practitioners, and subsequently the entire group (P < .001). The accuracy of determination of which breast contained the lesion significantly increased with DIVA for residents (P = .003) and practitioners (P = .04). There was little difference between the DIVA and slice-based visualization for the determination of the number of lesions. The accuracy of quadrant determination was significantly improved by DIVA for practicing surgeons (P = .01) but not significantly for residents (P = .49). CONCLUSION: This study indicates that the VR visualization of medical images systematically improves surgeons' analysis of preoperative breast MRI scans across several different metrics irrespective of surgeon seniority.

Fast-track virtual reality for cardiac imaging in congenital heart disease.

BACKGROUND AND AIM OF THE STUDY: We sought to evaluate the appropriateness of cardiac anatomy renderings by a new virtual reality (VR) technology, entitled DIVA, directly applicable to raw magnetic resonance imaging (MRI) data without intermediate segmentation steps in comparison to standard three-dimensional (3D) rendering techniques (3D PDF and 3D printing). Differences in post-processing times were also evaluated. METHODS: We reconstructed 3D (STL, 3D-PDF, and 3D printed ones) and VR models of three patients with different types of complex congenital heart disease (CHD). We then asked a senior pediatric heart surgeon to compare and grade the results obtained. RESULTS: All anatomical structures were well visualized in both VR and 3D PDF/printed models. Ventricular-arterial connections and their relationship with the great vessels were better visualized with the VR model (Case 2); aortic arch anatomy and details were also better visualized by the VR model (Case 3). The median post-processing time to get VR models using DIVA was 5 min in comparison to 8 h (range 8-12 h including printing time) for 3D models (PDF/printed). CONCLUSIONS: VR directly applied to non-segmented 3D-MRI data set is a promising technique for 3D advanced modeling in CHD. It is systematically more consistent and faster when compared to standard 3D-modeling techniques.

Partial breast resection for multifocal lower quadrant breast tumour using virtual reality.

Oncoplastic surgery allows an increase in the number of indications for conservative breast cancer treatments. However, uncertainty as to whether it can be performed still exists in certain situations such as with multicentric or multifocal lesions, even when the breast volume can accommodate it. With the aid of a virtual reality software, DIVA, allowing the precise visualisation of tumours and breast volumes based entirely on the patient's MRI, we report the ability to rapidly confirm and secure an indication for partial surgery of multiple lesions in a 31-year-old patient. With the described approach, the patient did not have to suffer significant disfigurement from cancerous breast surgery without compromising safety.

Towards Human in the Loop Analysis of Complex Point Clouds: Advanced Visualizations, Quantifications, and Communication Features in Virtual Reality.

Multiple fields in biological and medical research produce large amounts of point cloud data with high dimensionality and complexity. In addition, a large set of experiments generate point clouds, including segmented medical data or single-molecule localization microscopy. In the latter, individual molecules are observed within their natural cellular environment. Analyzing this type of experimental data is a complex task and presents unique challenges, where providing extra physical dimensions for visualization and analysis could be beneficial. Furthermore, whether highly noisy data comes from single-molecule recordings or segmented medical data, the necessity to guide analysis with user intervention creates both an ergonomic challenge to facilitate this interaction and a computational challenge to provide fluid interactions as information is being processed. Several applications, including our software DIVA for image stack and our platform Genuage for point clouds, have leveraged Virtual Reality (VR) to visualize and interact with data in 3D. While the visualization aspects can be made compatible with different types of data, quantifications, on the other hand, are far from being standard. In addition, complex analysis can require significant computational resources, making the real-time VR experience uncomfortable. Moreover, visualization software is mainly designed to represent a set of data points but lacks flexibility in manipulating and analyzing the data. This paper introduces new libraries to enhance the interaction and human-in-the-loop analysis of point cloud data in virtual reality and integrate them into the open-source platform Genuage. We first detail a new toolbox of communication tools that enhance user experience and improve flexibility. Then, we introduce a mapping toolbox allowing the representation of physical properties in space overlaid on a 3D mesh while maintaining a point cloud dedicated shader. We introduce later a new and programmable video capture tool in VR and desktop modes for intuitive data dissemination. Finally, we highlight the protocols that allow simultaneous analysis and fluid manipulation of data with a high refresh rate. We illustrate this principle by performing real-time inference of random walk properties of recorded trajectories with a pre-trained Graph Neural Network running in Python.

New Approach to Accelerated Image Annotation by Leveraging Virtual Reality and Cloud Computing.

Three-dimensional imaging is at the core of medical imaging and is becoming a standard in biological research. As a result, there is an increasing need to visualize, analyze and interact with data in a natural three-dimensional context. By combining stereoscopy and motion tracking, commercial virtual reality (VR) headsets provide a solution to this critical visualization challenge by allowing users to view volumetric image stacks in a highly intuitive fashion. While optimizing the visualization and interaction process in VR remains an active topic, one of the most pressing issue is how to utilize VR for annotation and analysis of data. Annotating data is often a required step for training machine learning algorithms. For example, enhancing the ability to annotate complex three-dimensional data in biological research as newly acquired data may come in limited quantities. Similarly, medical data annotation is often time-consuming and requires expert knowledge to identify structures of interest correctly. Moreover, simultaneous data analysis and visualization in VR is computationally demanding. Here, we introduce a new procedure to visualize, interact, annotate and analyze data by combining VR with cloud computing. VR is leveraged to provide natural interactions with volumetric representations of experimental imaging data. In parallel, cloud computing performs costly computations to accelerate the data annotation with minimal input required from the user. We demonstrate multiple proof-of-concept applications of our approach on volumetric fluorescent microscopy images of mouse neurons and tumor or organ annotations in medical images.

DIVA, a 3D virtual reality platform, improves undergraduate craniofacial trauma education.

Craniofacial fractures management is challenging to teach due to the complex anatomy of the head, even when using three-dimensional CT-scan images. DIVA is a software allowing the straightforward visualization of CT-scans in a user-friendly three-dimensional virtual reality environment. Here, we assess DIVA as an educational tool for craniofacial trauma for undergraduate medical students. Three craniofacial trauma cases (jaw fracture, naso-orbital-ethmoid complex fracture and Le Fort 3 fracture) were submitted to 50 undergraduate medical students, who had to provide diagnoses and treatment plans. Each student then filled an 8-item questionnaire assessing satisfaction, potential benefit, ease of use and tolerance. Additionally, 4 postgraduate students were requested to explore these cases and to place 6 anatomical landmarks on both virtual reality renderings and usual slice-based three-dimensional CT-scan visualizations. High degrees of satisfaction (98%) without specific tolerance issues (86%) were reported. The potential benefit in a better understanding of craniofacial trauma using virtual reality was reported by almost all students (98%). Virtual reality allowed a reliable localization of key anatomical landmarks when compared with standard three-dimensional CT-scan visualization. Virtual reality interfaces such DIVA are beneficial to medical students for a better understanding of craniofacial trauma and allow a reliable rendering of craniofacial anatomy.

Genuage: visualize and analyze multidimensional single-molecule point cloud data in virtual reality.

Experimentally recorded point cloud data, such as those generated by single-molecule localization microscopy, are continuously increasing in size and dimension. Gaining an intuitive understanding and facilitating the analysis of such multidimensional data remains challenging. Here we report a new open-source software platform, Genuage, that enables the easy perception of, interaction with and analysis of multidimensional point clouds in virtual reality. Genuage is compatible with arbitrary multidimensional data extending beyond single-molecule localization microscopy.

DIVA: Natural Navigation Inside 3D Images Using Virtual Reality.

As three-dimensional microscopy becomes commonplace in biological research, there is an increasing need for researchers to be able to view experimental image stacks in a natural three-dimensional viewing context. Through stereoscopy and motion tracking, commercial virtual reality headsets provide a solution to this important visualization challenge by allowing researchers to view volumetric objects in an entirely intuitive fashion. With this motivation, we present DIVA, a user-friendly software tool that automatically creates detailed three-dimensional reconstructions of raw experimental image stacks that are integrated in virtual reality. In DIVA's immersive virtual environment, users can view, manipulate and perform volumetric measurements on their microscopy images as they would to real physical objects. In contrast to similar solutions, our software provides high-quality volume rendering with native TIFF file compatibility. We benchmark the software with diverse image types including those generated by confocal, light-sheet and electron microscopy. DIVA is available at https://diva.pasteur.fr and will be regularly updated.

Virtual Reality: Beyond Visualization.

Virtual reality (VR) has recently become an affordable technology. A wide range of options are available to access this unique visualization medium, from simple cardboard inserts for smartphones to truly advanced headsets tracked by external sensors. While it is now possible for any research team to gain access to VR, we can still question what it brings to scientific research. Visualization and the ability to navigate complex three-dimensional data are undoubtedly a gateway to many scientific applications; however, we are convinced that data treatment and numerical simulations, especially those mixing interactions with data, human cognition, and automated algorithms will be the future of VR in scientific research. Moreover, VR might soon merit the same level of attention to imaging data as machine learning currently has. In this short perspective, we discuss approaches that employ VR in scientific research based on some concrete examples.

Author Correction: Single molecule localisation microscopy reveals how HIV-1 Gag proteins sense membrane virus assembly sites in living host CD4 T cells.

A correction to this article has been published and is linked from the HTML and PDF versions of this paper. The error has been fixed in the paper.

Single molecule localisation microscopy reveals how HIV-1 Gag proteins sense membrane virus assembly sites in living host CD4 T cells.

Monitoring virus assembly at the nanoscale in host cells remains a major challenge. Human immunodeficiency virus type 1 (HIV-1) components are addressed to the plasma membrane where they assemble to form spherical particles of 100 nm in diameter. Interestingly, HIV-1 Gag protein expression alone is sufficient to produce virus-like particles (VLPs) that resemble the immature virus. Here, we monitored VLP formation at the plasma membrane of host CD4+ T cells using a newly developed workflow allowing the analysis of long duration recordings of single-molecule Gag protein localisation and movement. Comparison of Gag assembling platforms in CD4+ T cells expressing wild type or assembly-defective Gag mutant proteins showed that VLP formation lasts roughly 15 minutes with an assembly time of 5 minutes. Trapping energy maps, built from membrane associated Gag protein movements, showed that one third of the assembling energy is due to direct Gag capsid-capsid interaction while the remaining two thirds require the nucleocapsid-RNA interactions. Finally, we show that the viral RNA genome does not increase the attraction of Gag at the membrane towards the assembling site but rather acts as a spatiotemporal coordinator of the membrane assembly process.

PSF engineering in multifocus microscopy for increased depth volumetric imaging.

Imaging and localizing single molecules with high accuracy in a 3D volume is a challenging task. Here we combine multifocal microscopy, a recently developed volumetric imaging technique, with point spread function engineering to achieve an increased depth for single molecule imaging. Applications in 3D single molecule localization-based super-resolution imaging is shown over an axial depth of 4 µm as well as for the tracking of diffusing beads in a fluid environment over 8 µm.

Multifocus microscopy with precise color multi-phase diffractive optics applied in functional neuronal imaging.

Multifocus microscopy (MFM) allows high-resolution instantaneous three-dimensional (3D) imaging and has been applied to study biological specimens ranging from single molecules inside cells nuclei to entire embryos. We here describe pattern designs and nanofabrication methods for diffractive optics that optimize the light-efficiency of the central optical component of MFM: the diffractive multifocus grating (MFG). We also implement a "precise color" MFM layout with MFGs tailored to individual fluorophores in separate optical arms. The reported advancements enable faster and brighter volumetric time-lapse imaging of biological samples. In live microscopy applications, photon budget is a critical parameter and light-efficiency must be optimized to obtain the fastest possible frame rate while minimizing photodamage. We provide comprehensive descriptions and code for designing diffractive optical devices, and a detailed methods description for nanofabrication of devices. Theoretical efficiencies of reported designs is ≈90% and we have obtained efficiencies of > 80% in MFGs of our own manufacture. We demonstrate the performance of a multi-phase MFG in 3D functional neuronal imaging in living C. elegans.

A Primer on the Bayesian Approach to High-Density Single-Molecule Trajectories Analysis.

Tracking single molecules in living cells provides invaluable information on their environment and on the interactions that underlie their motion. New experimental techniques now permit the recording of large amounts of individual trajectories, enabling the implementation of advanced statistical tools for data analysis. In this primer, we present a Bayesian approach toward treating these data, and we discuss how it can be fruitfully employed to infer physical and biochemical parameters from single-molecule trajectories.

Dynamics of CRISPR-Cas9 genome interrogation in living cells.

The RNA-guided CRISPR-associated protein Cas9 is used for genome editing, transcriptional modulation, and live-cell imaging. Cas9-guide RNA complexes recognize and cleave double-stranded DNA sequences on the basis of 20-nucleotide RNA-DNA complementarity, but the mechanism of target searching in mammalian cells is unknown. Here, we use single-particle tracking to visualize diffusion and chromatin binding of Cas9 in living cells. We show that three-dimensional diffusion dominates Cas9 searching in vivo, and off-target binding events are, on average, short-lived (<1 second). Searching is dependent on the local chromatin environment, with less sampling and slower movement within heterochromatin. These results reveal how the bacterial Cas9 protein interrogates mammalian genomes and navigates eukaryotic chromatin structure.

Whole-cell, multicolor superresolution imaging using volumetric multifocus microscopy.

Single molecule-based superresolution imaging has become an essential tool in modern cell biology. Because of the limited depth of field of optical imaging systems, one of the major challenges in superresolution imaging resides in capturing the 3D nanoscale morphology of the whole cell. Despite many previous attempts to extend the application of photo-activated localization microscopy (PALM) and stochastic optical reconstruction microscopy (STORM) techniques into three dimensions, effective localization depths do not typically exceed 1.2 µm. Thus, 3D imaging of whole cells (or even large organelles) still demands sequential acquisition at different axial positions and, therefore, suffers from the combined effects of out-of-focus molecule activation (increased background) and bleaching (loss of detections). Here, we present the use of multifocus microscopy for volumetric multicolor superresolution imaging. By simultaneously imaging nine different focal planes, the multifocus microscope instantaneously captures the distribution of single molecules (either fluorescent proteins or synthetic dyes) throughout an ∼ 4-µm-deep volume, with lateral and axial localization precisions of ∼ 20 and 50 nm, respectively. The capabilities of multifocus microscopy to rapidly image the 3D organization of intracellular structures are illustrated by superresolution imaging of the mammalian mitochondrial network and yeast microtubules during cell division.

Accessing the third dimension in localization-based super-resolution microscopy.

Only a few years after its inception, localization-based super-resolution microscopy has become widely employed in biological studies. Yet, it is primarily used in two-dimensional imaging and accessing the organization of cellular structures at the nanoscale in three dimensions (3D) still poses important challenges. Here, we review optical and computational techniques that enable the 3D localization of individual emitters and the reconstruction of 3D super-resolution images. These techniques are grouped into three main categories: PSF engineering, multiple plane imaging and interferometric approaches. We provide an overview of their technical implementation as well as commentary on their applicability. Finally, we discuss future trends in 3D localization-based super-resolution microscopy.