Learning to deep learning: statistics and a paradigm test in selecting a UNet architecture to enhance MRI.

OBJECTIVE: This study aims to assess the statistical significance of training parameters in 240 dense UNets (DUNets) used for enhancing low Signal-to-Noise Ratio (SNR) and undersampled MRI in various acquisition protocols. The objective is to determine the validity of differences between different DUNet configurations and their impact on image quality metrics. MATERIALS AND METHODS: To achieve this, we trained all DUNets using the same learning rate and number of epochs, with variations in 5 acquisition protocols, 24 loss function weightings, and 2 ground truths. We calculated evaluation metrics for two metric regions of interest (ROI). We employed both Analysis of Variance (ANOVA) and Mixed Effects Model (MEM) to assess the statistical significance of the independent parameters, aiming to compare their efficacy in revealing differences and interactions among fixed parameters. RESULTS: ANOVA analysis showed that, except for the acquisition protocol, fixed variables were statistically insignificant. In contrast, MEM analysis revealed that all fixed parameters and their interactions held statistical significance. This emphasizes the need for advanced statistical analysis in comparative studies, where MEM can uncover finer distinctions often overlooked by ANOVA. DISCUSSION: These findings highlight the importance of utilizing appropriate statistical analysis when comparing different deep learning models. Additionally, the surprising effectiveness of the UNet architecture in enhancing various acquisition protocols underscores the potential for developing improved methods for characterizing and training deep learning models. This study serves as a stepping stone toward enhancing the transparency and comparability of deep learning techniques for medical imaging applications.

A Holographic Augmented Reality Interface for Visualizing of MRI Data and Planning of Neurosurgical Procedures.

The recent introduction of wireless head-mounted displays (HMD) promises to enhance 3D image visualization by immersing the user into 3D morphology. This work introduces a prototype holographic augmented reality (HAR) interface for the 3D visualization of magnetic resonance imaging (MRI) data for the purpose of planning neurosurgical procedures. The computational platform generates a HAR scene that fuses pre-operative MRI sets, segmented anatomical structures, and a tubular tool for planning an access path to the targeted pathology. The operator can manipulate the presented images and segmented structures and perform path-planning using voice and gestures. On-the-fly, the software uses defined forbidden-regions to prevent the operator from harming vital structures. In silico studies using the platform with a HoloLens HMD assessed its functionality and the computational load and memory for different tasks. A preliminary qualitative evaluation revealed that holographic visualization of high-resolution 3D MRI data offers an intuitive and interactive perspective of the complex brain vasculature and anatomical structures. This initial work suggests that immersive experiences may be an unparalleled tool for planning neurosurgical procedures.

A Platform Integrating Acquisition, Reconstruction, Visualization, and Manipulator Control Modules for MRI-Guided Interventions.

This work presents a platform that integrates a customized MRI data acquisition scheme with reconstruction and three-dimensional (3D) visualization modules along with a module for controlling an MRI-compatible robotic device to facilitate the performance of robot-assisted, MRI-guided interventional procedures. Using dynamically-acquired MRI data, the computational framework of the platform generates and updates a 3D model representing the area of the procedure (AoP). To image structures of interest in the AoP that do not reside inside the same or parallel slices, the MRI acquisition scheme was modified to collect a multi-slice set of intraoblique to each other slices; which are termed composing slices. Moreover, this approach interleaves the collection of the composing slices so the same k-space segments of all slices are collected during similar time instances. This time matching of the k-space segments results in spatial matching of the imaged objects in the individual composing slices. The composing slices were used to generate and update the 3D model of the AoP. The MRI acquisition scheme was evaluated with computer simulations and experimental studies. Computer simulations demonstrated that k-space segmentation and time-matched interleaved acquisition of these segments provide spatial matching of the structures imaged with composing slices. Experimental studies used the platform to image the maneuvering of an MRI-compatible manipulator that carried tubing filled with MRI contrast agent. In vivo experimental studies to image the abdomen and contrast enhanced heart on free-breathing subjects without cardiac triggering demonstrated spatial matching of imaged anatomies in the composing planes. The described interventional MRI framework could assist in performing real-time MRI-guided interventions.

Let UNet Play an Adversarial Game: Investigating the Effect of Adversarial Training in Enhancing Low-Resolution MRI.

Adversarial training has attracted much attention in enhancing the visual realism of images, but its efficacy in clinical imaging has not yet been explored. This work investigated adversarial training in a clinical context, by training 206 networks on the OASIS-1 dataset for improving low-resolution and low signal-to-noise ratio (SNR) magnetic resonance images. Each network corresponded to a different combination of perceptual and adversarial loss weights and distinct learning rate values. For each perceptual loss weighting, we identified its corresponding adversarial loss weighting that minimized structural disparity. Each optimally weighted adversarial loss yielded an average SSIM reduction of 1.5%. We further introduced a set of new metrics to assess other clinically relevant image features: Gradient Error (GE) to measure structural disparities; Sharpness to compute edge clarity; and Edge-Contrast Error (ECE) to quantify any distortion of the pixel distribution around edges. Including adversarial loss increased structural enhancement in visual inspection, which correlated with statistically consistent GE reductions (p-value << 0.05). This also resulted in increased Sharpness; however, the level of statistical significance was dependent on the perceptual loss weighting. Additionally, adversarial loss yielded ECE reductions for smaller perceptual loss weightings, while showing non-significant increases (p-value >> 0.05) when these weightings were higher, demonstrating that the increased Sharpness does not adversely distort the pixel distribution around the edges in the image. These studies clearly suggest that adversarial training significantly improves the performance of an MRI enhancement pipeline, and highlights the need for systematic studies of hyperparameter optimization and investigation of alternative image quality metrics.

Deep learning methods for automatic evaluation of delayed enhancement-MRI. The results of the EMIDEC challenge.

A key factor for assessing the state of the heart after myocardial infarction (MI) is to measure whether the myocardium segment is viable after reperfusion or revascularization therapy. Delayed enhancement-MRI or DE-MRI, which is performed 10 min after injection of the contrast agent, provides high contrast between viable and nonviable myocardium and is therefore a method of choice to evaluate the extent of MI. To automatically assess myocardial status, the results of the EMIDEC challenge that focused on this task are presented in this paper. The challenge's main objectives were twofold. First, to evaluate if deep learning methods can distinguish between non-infarct and pathological exams, i.e. exams with or without hyperenhanced area. Second, to automatically calculate the extent of myocardial infarction. The publicly available database consists of 150 exams divided into 50 cases without any hyperenhanced area after injection of a contrast agent and 100 cases with myocardial infarction (and then with a hyperenhanced area on DE-MRI), whatever their inclusion in the cardiac emergency department. Along with MRI, clinical characteristics are also provided. The obtained results issued from several works show that the automatic classification of an exam is a reachable task (the best method providing an accuracy of 0.92), and the automatic segmentation of the myocardium is possible. However, the segmentation of the diseased area needs to be improved, mainly due to the small size of these areas and the lack of contrast with the surrounding structures.

A novel virtual reality environment for preoperative planning and simulation of image guided intracardiac surgeries with robotic manipulators.

The evolution of image-guided and robot-assisted procedures can be beneficial to intracardiac interventions. This paper proposes a novel approach and a virtual reality system for preoperative planning and intraoperative guidance of cardiac procedures, and for investigating the kinematics and control of a virtual robotic manipulator, based on MRI CINE images. The system incorporates dedicated software modules for processing MR images, generating dynamic trajectories in the continuously changing environment of a beating heart, controlling a specific generic virtual manipulator along those trajectories, and a virtual reality interface that fuses all those information. The proposed system is applied for the simulation of accessing the aortic valve annulus via a small incision on the apex by maneuvering a robotic manipulator through an access corridor that safely transverses the left ventricle (LV) of the beating heart.

A modular and scalable computational framework for interactive immersion into imaging data with a holographic augmented reality interface.

BACKGROUND AND OBJECTIVE: Modern imaging scanners produce an ever-growing body of 3D/4D multimodal data requiring image analytics and visualization of fused images, segmentations, and information. For the latter, augmented reality (AR) with head-mounted displays (HMDs) has shown potential. This work describes a framework (FI3D) for interactive immersion with data, integration of image processing and analytics, and rendering and fusion with an AR interface. METHODS: The FI3D was designed and endowed with modules to communicate with peripherals, including imaging scanners and HMDs, and to provide computational power for data acquisition and processing. The core of FI3D is deployed to a dedicated computational unit that performs the computationally demanding processes in real-time, and the HMD is used as a display output peripheral and an input peripheral through gestures and voice commands. FI3D offers user-made processing and analysis dedicated modules. Users can customize and optimize these for a particular workflow while incorporating current or future libraries. RESULTS: The FI3D framework was used to develop a workflow for processing, rendering, and visualization of CINE MRI cardiac sets. In this version, the data were loaded from a remote database, and the endocardium and epicardium of the left ventricle (LV) were segmented using a machine learning model and transmitted to a HoloLens HMD to be visualized in 4D. Performance results show that the system is capable of maintaining an image stream of one image per second with a resolution of 512 × 512. Also, it can modify visual properties of the holograms at 1 update per 16 milliseconds (62.5 Hz) while providing enough resources for the segmentation and surface reconstruction tasks without hindering the HMD. CONCLUSIONS: We provide a system design and framework to be used as a foundation for medical applications that benefit from AR visualization, removing several technical challenges from the developmental pipeline.

End-user evaluation of software-generated intervention planning environment for transrectal magnetic resonance-guided prostate biopsies.

BACKGROUND: This study presents user evaluation studies to assess the effect of information rendered by an interventional planning software on the operator's ability to plan transrectal magnetic resonance (MR)-guided prostate biopsies using actuated robotic manipulators. METHODS: An intervention planning software was developed based on the clinical workflow followed for MR-guided transrectal prostate biopsies. The software was designed to interface with a generic virtual manipulator and simulate an intervention environment using 2D and 3D scenes. User studies were conducted with urologists using the developed software to plan virtual biopsies. RESULTS: User studies demonstrated that urologists with prior experience in using 3D software completed the planning less time. 3D scenes were required to control all degrees-of-freedom of the manipulator, while 2D scenes were sufficient for planar motion of the manipulator. CONCLUSIONS: The study provides insights on using 2D versus 3D environment from a urologist's perspective for different operational modes of MR-guided prostate biopsy systems.

Evaluation of how users interface with holographic augmented reality surgical scenes: Interactive planning MR-Guided prostate biopsies.

BACKGROUND: User interfaces play a vital role in the planning and execution of an interventional procedure. The objective of this study is to investigate the effect of using different user interfaces for planning transrectal robot-assisted MR-guided prostate biopsy (MRgPBx) in an augmented reality (AR) environment. METHOD: End-user studies were conducted by simulating an MRgPBx system with end- and side-firing modes. The information from the system to the operator was rendered on HoloLens as an output interface. Joystick, mouse/keyboard, and holographic menus were used as input interfaces to the system. RESULTS: The studies indicated that using a joystick improved the interactive capacity and enabled operator to plan MRgPBx in less time. It efficiently captures the operator's commands to manipulate the augmented environment representing the state of MRgPBx system. CONCLUSIONS: The study demonstrates an alternative to conventional input interfaces to interact and manipulate an AR environment within the context of MRgPBx planning.

Manipulator-driven selection of semi-active MR-visible markers.

BACKGROUND: A method for the identification of semi-active fiducial magnetic resonance (MR) markers is presented based on selectively optically tuning and detuning them. METHODS: Four inductively coupled solenoid coils with photoresistors were connected to light sources. A microcontroller timed the optical tuning/detuning of coils and image collection. The markers were tested on an MR manipulator linking the microcontroller to the manipulator control to visibly select the marker subset according to the actuated joint. RESULTS: In closed-loop control, the average and maximum were 0.76° ± 0.41° and 1.18° errors for a rotational joint, and 0.87 mm ± 0.26 mm and 1.13 mm for the prismatic joint. CONCLUSIONS: This technique is suitable for MR-compatible actuated devices that use semi-active MR-compatible markers.

Performance of interventions with manipulator-driven real-time MR guidance: implementation and initial in vitro tests.

The purpose of this work was to implement and assess the performance of interventions inside a cylindrical magnetic resonance imaging (MRI) scanner with an MR-compatible manipulator system and manipulator-driven real-time MR guidance. The interventional system is based on a seven degree-of-freedom MR-compatible manipulator, which offers man-in-the-loop control either with a graphical user interface or with a master/slave device. The position and the orientation of the interventional tool are sent to an MR scanner for a manipulator-driven dynamic update of the imaging plane to track, visualize and guide the motion of an end-effector. Studies on phantoms were performed with a cylindrical 1.5-T scanner using an operator-managed triggered gradient-recalled echo (GRE) or a computer-managed dynamic True Fast Imaging with Steady Precession (TrueFISP). Targets were acquired with an accuracy of 3.2 mm and with an in-plane path orientation of 2.5 degrees relative to the prescribed one. Path planning, including negotiation of obstacles and needle bending, was successfully performed. The signal-to-noise ratio (SNR) of TrueFISP was 25.3+/-2.1 when the manipulator was idle and was 18.6+/-2.4 during its operation. The SNR of triggered GRE (acquired only when the manipulator was idle) was 61.3+/-1.8. In conclusion, this study shows the feasibility of performing manually directed interventions inside cylindrical MR scanners with real-time MRI.

Tagging of the magnetization with the transition zones of 360 degrees rotations generated by a tandem of two adiabatic DANTE inversion sequences.

A new technique is presented for generating myocardial tagging using the signal intensity minima of the transition zones between the bands of 0 degrees and 360 degrees rotations, induced by a tandem of two adiabatic delays alternating with nutations for tailored excitation (DANTE) inversion sequences. With this approach, the underlying matrix corresponds to magnetization that has experienced 0 degrees or 360 degrees rotations. The DANTE sequences were implemented from adiabatic parent pulses for insensitivity of the underlying matrix to B(1) inhomogeneity. The performance of the proposed tagging technique is demonstrated theoretically with computer simulations and experimentally on phantom and on the canine heart, using a surface coil for both RF transmission and signal reception. The simulations and the experimental data demonstrated uniform grid contrast and sharp tagging profiles over a twofold variation of the B(1) field magnitude.

GPU-accelerated interactive visualization and planning of neurosurgical interventions.

Advances in computational methods and hardware platforms provide efficient processing of medical-imaging datasets for surgical planning. For neurosurgical interventions employing a straight access path, planning entails selecting a path from the scalp to the target area that's of minimal risk to the patient. A proposed GPU-accelerated method enables interactive quantitative estimation of the risk for a particular path. It exploits acceleration spatial data structures and efficient implementation of algorithms on GPUs. In evaluations of its computational efficiency and scalability, it achieved interactive rates even for high-resolution meshes. A user study and feedback from neurosurgeons identified this methods' potential benefits for preoperative planning and intraoperative replanning.

A novel, general-purpose, MR-compatible, manually actuated robotic manipulation system for minimally invasive interventions under direct MRI guidance.

BACKGROUND: Performing minimally invasive interventions under direct MRI guidance offers significant advantages. Required accessibility to the patient inside the MRI scanner is fairly limited, and employment of robotic assistance has been proposed. The development of MR-compatible robotic systems entails engineering challenges related to geometric constraints and the magnetic nature of the scanning environment. METHODS: A novel, general-purpose, MR-compatible robotic manipulation system has been developed for the performance of minimally invasive interventions inside a cylindrical scanner under direct MRI guidance. The system is endowed with five degrees of freedom (DOF), is characterized by a unique kinematics structure and is manually actuated. RESULTS: The prototype system was shown to exhibit the required MR-compatibility characteristics and a task-space positioning ability of approximately 5 mm. Needle targeting testing demonstrated a 93% success rate in acquiring a 5 mm spherical target. Phantom testing was performed inside a 3 T scanner and results are reported for an experimental study simulating MRI-guided, manipulator-assisted, MR arthrography. CONCLUSIONS: Robotic assistance provided by the developed manipulator may effectively facilitate the performance of various MRI-guided, minimally invasive interventions inside a cylindrical scanner.

A framework for integrating real-time MRI with robot control: application to simulated transapical cardiac interventions.

The advent of intraoperative real-time image guidance has led to the emergence of new surgical interventional paradigms including image-guided robot assistance. Most often the use of an intraoperative imaging modality is limited to visual perception of the area of procedure. In this study, we propose a framework for performing robot-assisted interventions with real-time magnetic resonance imaging (rtMRI) guidance. The described computational core of this framework, processes on-the-fly rtMRI, integrates the processed information with robot control and renders it on the human-machine interfaces. This information is rendered on a visualization and force-feedback interface for enhanced perception of a dynamic area of procedure and for assisting the operator in the safe and accurate maneuvering of a robotic manipulator. The framework was experimentally tested by applying it to a simulated Transapical aortic valve implantation with a virtual robotic manipulator. rtMRI data were processed on-the-fly in a rolling-window scheme and together with a multithreaded and multihardware implementation, the core delivered appropriate speed of 20 Hz for visualization and 1000 Hz for force feedback. The experimental results demonstrate significant improvement in the simulated task by both decreasing the duration of the procedure by half and increasing safety in the presence of cardiac and breathing motion by reducing the duration or incidents the operator collides with the tissue.

A system for endoscopic mechanically scanned localized proton MR and light-induced fluorescence emission spectroscopies.

Molecular and near-cellular modalities offer new opportunities in assessing living tissue in situ, and multimodality approaches, which offer complementary information, may lead to improved characterization of tissue pathophysiology benefiting diagnosis and focal therapy. However, many such modalities are limited by their low penetration through tissue, which has led to minimally invasive trans-cannula approaches to place the corresponding sensors locally at the area of interest. This work presents a system for performing localized fluorescence emission and proton magnetic resonance (MR) spectroscopies via endoscopic access. The in-house developed side-firing 1.9-mm wide dual-sensor integrates a three-fiber optical sensor for fluorescence emission optical spectroscopy and a 1-mm circular radiofrequency (RF) coil for localized MR proton spectroscopy. An MR-compatible manipulator was developed for carrying and mechanically translating the dual-sensor along a linear access channel. The hardware and software control of the system allows reconfigurable synchronization of the manipulator-assisted translation of the sensor, and MR and optical data collection. The manipulator serves as the mechanical link for the three modalities and MR images, MR spectra and optical spectra are inherently co-registered to the MR scanner coordinate system. These spectra were then used to generate spatio-spectral maps of the fluorophores and proton MR-signal sources in three-compartment phantoms with optically- and MR-visible, and distinguishable, materials. These data demonstrate a good spatial match between MR images, MR spectra and optical spectra along the scanned path. In addition to basic research, such a system may have clinical applications for assessing and characterizing cancer in situ, as well as guiding focal therapies.

An approach for preoperative planning and performance of MR-guided interventions demonstrated with a manual manipulator in a 1.5T MRI scanner.

PURPOSE: The aim of this work was to develop and test a general methodology for the planning and performance of robot-assisted, MR-guided interventions. This methodology also includes the employment of software tools with appropriately tailored routines to effectively exploit the capabilities of MRI and address the relevant spatial limitations. METHODS: The described methodology consists of: (1) patient-customized feasibility study that focuses on the geometric limitations imposed by the gantry, the robotic hardware, and interventional tools, as well as the patient; (2) stereotactic preoperative planning for initial positioning of the manipulator and alignment of its end-effector with a selected target; and (3) real-time, intraoperative tool tracking and monitoring of the actual intervention execution. Testing was performed inside a standard 1.5T MRI scanner in which the MR-compatible manipulator is deployed to provide the required access. RESULTS: A volunteer imaging study demonstrates the application of the feasibility stage. A phantom study on needle targeting is also presented, demonstrating the applicability and effectiveness of the proposed preoperative and intraoperative stages of the methodology. For this purpose, a manually actuated, MR-compatible robotic manipulation system was used to accurately acquire a prescribed target through alternative approaching paths. CONCLUSIONS: The methodology presented and experimentally examined allows the effective performance of MR-guided interventions. It is suitable for, but not restricted to, needle-targeting applications assisted by a robotic manipulation system, which can be deployed inside a cylindrical scanner to provide the required access to the patient facilitating real-time guidance and monitoring.

Simulations and experimental demonstration of coupling molecular and macroscopic level modalities with a robotic manipulator.

Established and emerging molecular and cellular modalities, such as optical imaging and spectroscopy, offer new opportunities for assessing tissue pathophysiology in situ. A challenge with such applications is their limited tissue penetration and low sensitivity that can be addressed with trans-needle or trans-catheter access. In this work, we describe the use of an actuated manipulator to physically manipulate such sensors to scan an area of interest generating 1-D scans while registering them to a guiding modality. Simulations were performed for a miniature RF coil to determine the voxel size, and experimental studies were conducted using a miniature RF coil manipulated by the MR-compatible device. The experimental results on phantom studies show that potential diagnostic information can be collected by using this methodology. This system was pursued to address a critical limitation of emerging molecular and near-cellular modalities; the limited tissue penetration.

MR-based real time path planning for cardiac operations with transapical access.

Minimally invasive surgeries (MIS) have been perpetually evolving due to their potential high impact on improving patient management and overall cost effectiveness. Currently, MIS are further strengthened by the incorporation of magnetic resonance imaging (MRI) for amended visualization and high precision. Motivated by the fact that real-time MRI is emerging as a feasible modality especially for guiding interventions and surgeries in the beating heart; in this paper we introduce a real-time path planning algorithm for intracardiac procedures. Our approach creates a volumetric safety zone inside a beating heart and updates it on-the-fly using real-time MRI during the deployment of a robotic device. In order to prove the concept and assess the feasibility of the introduced method, a realistic operational scenario of transapical aortic valve replacement in a beating heart is chosen as the virtual case study.