3D vertebrae labeling in spine CT: an accurate, memory-efficient (Ortho2D) framework.

Purpose.Accurate localization and labeling of vertebrae in computed tomography (CT) is an important step toward more quantitative, automated diagnostic analysis and surgical planning. In this paper, we present a framework (called Ortho2D) for vertebral labeling in CT in a manner that is accurate and memory-efficient.Methods. Ortho2D uses two independent faster R-convolutional neural network networks to detect and classify vertebrae in orthogonal (sagittal and coronal) CT slices. The 2D detections are clustered in 3D to localize vertebrae centroids in the volumetric CT and classify the region (cervical, thoracic, lumbar, or sacral) and vertebral level. A post-process sorting method incorporates the confidence in network output to refine classifications and reduce outliers. Ortho2D was evaluated on a publicly available dataset containing 302 normal and pathological spine CT images with and without surgical instrumentation. Labeling accuracy and memory requirements were assessed in comparison to other recently reported methods. The memory efficiency of Ortho2D permitted extension to high-resolution CT to investigate the potential for further boosts to labeling performance.Results. Ortho2D achieved overall vertebrae detection accuracy of 97.1%, region identification accuracy of 94.3%, and individual vertebral level identification accuracy of 91.0%. The framework achieved 95.8% and 83.6% level identification accuracy in images without and with surgical instrumentation, respectively. Ortho2D met or exceeded the performance of previously reported 2D and 3D labeling methods and reduced memory consumption by a factor of ∼50 (at 1 mm voxel size) compared to a 3D U-Net, allowing extension to higher resolution datasets than normally afforded. The accuracy of level identification increased from 80.1% (for standard/low resolution CT) to 95.1% (for high-resolution CT).Conclusions. The Ortho2D method achieved vertebrae labeling performance that is comparable to other recently reported methods with significant reduction in memory consumption, permitting further performance boosts via application to high-resolution CT.

Data-Driven Deformable 3D-2D Registration for Guiding Neuroelectrode Placement in Deep Brain Stimulation.

Purpose: Deep brain stimulation is a neurosurgical procedure used in treatment of a growing spectrum of movement disorders. Inaccuracies in electrode placement, however, can result in poor symptom control or adverse effects and confound variability in clinical outcomes. A deformable 3D-2D registration method is presented for high-precision 3D guidance of neuroelectrodes. Methods: The approach employs a model-based, deformable algorithm for 3D-2D image registration. Variations in lead design are captured in a parametric 3D model based on a B-spline curve. The registration is solved through iterative optimization of 16 degrees-of-freedom that maximize image similarity between the 2 acquired radiographs and simulated forward projections of the neuroelectrode model. The approach was evaluated in phantom models with respect to pertinent imaging parameters, including view selection and imaging dose. Results: The results demonstrate an accuracy of (0.2 ± 0.2) mm in 3D localization of individual electrodes. The solution was observed to be robust to changes in pertinent imaging parameters, which demonstrate accurate localization with ≥20° view separation and at 1/10th the dose of a standard fluoroscopy frame. Conclusions: The presented approach provides the means for guiding neuroelectrode placement from 2 low-dose radiographic images in a manner that accommodates potential deformations at the target anatomical site. Future work will focus on improving runtime though learning-based initialization, application in reducing reconstruction metal artifacts for 3D verification of placement, and extensive evaluation in clinical data from an IRB study underway.

Data-Driven Detection and Registration of Spine Surgery Instrumentation in Intraoperative Images.

Purpose: Conventional model-based 3D-2D registration algorithms can be challenged by limited capture range, model validity, and stringent intraoperative runtime requirements. In this work, a deep convolutional neural network was used to provide robust initialization of a registration algorithm (known-component registration, KC-Reg) for 3D localization of spine surgery implants, combining the speed and global support of data-driven approaches with the previously demonstrated accuracy of model-based registration. Methods: The approach uses a Faster R-CNN architecture to detect and localize a broad variety and orientation of spinal pedicle screws in clinical images. Training data were generated using projections from 17 clinical cone-beam CT scans and a library of screw models to simulate implants. Network output was processed to provide screw count and 2D poses. The network was tested on two test datasets of 2,000 images, each depicting real anatomy and realistic spine surgery instrumentation - one dataset involving the same patient data as in the training set (but with different screws, poses, image noise, and affine transformations) and one dataset with five patients unseen in the test data. Assessment of device detection was quantified in terms of accuracy and specificity, and localization accuracy was evaluated in terms of intersection-over-union (IOU) and distance between true and predicted bounding box coordinates. Results: The overall accuracy of pedicle screw detection was ~86.6% (85.3% for the same-patient dataset and 87.8% for the many-patient dataset), suggesting that the screw detection network performed reasonably well irrespective of disparate, complex anatomical backgrounds. The precision of screw detection was ~92.6% (95.0% and 90.2% for the respective same-patient and many-patient datasets). The accuracy of screw localization was within 1.5 mm (median difference of bounding box coordinates), and median IOU exceeded 0.85. For purposes of initializing a 3D-2D registration algorithm, the accuracy was observed to be well within the typical capture range of KC-Reg.1. Conclusions: Initial evaluation of network performance indicates sufficient accuracy to integrate with algorithms for implant registration, guidance, and verification in spine surgery. Such capability is of potential use in surgical navigation, robotic assistance, and data-intensive analysis of implant placement in large retrospective datasets. Future work includes correspondence of multiple views, 3D localization, screw classification, and expansion of the training dataset to a broader variety of anatomical sites, number of screws, and types of implants.

Image-Guided Robotic K-Wire Placement for Orthopaedic Trauma Surgery.

Purpose: We report the initial development of an image-based solution for robotic assistance of pelvic fracture fixation. The approach uses intraoperative radiographs, preoperative CT, and an end effector of known design to align the robot with target trajectories in CT. The method extends previous work to solve the robot-to-patient registration from a single radiographic view (without C-arm rotation) and addresses the workflow challenges associated with integrating robotic assistance in orthopaedic trauma surgery in a form that could be broadly applicable to isocentric or non-isocentric C-arms. Methods: The proposed method uses 3D-2D known-component registration to localize a robot end effector with respect to the patient by: (1) exploiting the extended size and complex features of pelvic anatomy to register the patient; and (2) capturing multiple end effector poses using precise robotic manipulation. These transformations, along with an offline hand-eye calibration of the end effector, are used to calculate target robot poses that align the end effector with planned trajectories in the patient CT. Geometric accuracy of the registrations was independently evaluated for the patient and the robot in phantom studies. Results: The resulting translational difference between the ground truth and patient registrations of a pelvis phantom using a single (AP) view was 1.3 mm, compared to 0.4 mm using dual (AP+Lat) views. Registration of the robot in air (i.e., no background anatomy) with five unique end effector poses achieved mean translational difference ~1.4 mm for K-wire placement in the pelvis, comparable to tracker-based margins of error (commonly ~2 mm). Conclusions: The proposed approach is feasible based on the accuracy of the patient and robot registrations and is a preliminary step in developing an image-guided robotic guidance system that more naturally fits the workflow of fluoroscopically guided orthopaedic trauma surgery. Future work will involve end-to-end development of the proposed guidance system and assessment of the system with delivery of K-wires in cadaver studies.

A line fiducial method for geometric calibration of cone-beam CT systems with diverse scan trajectories.

Modern cone-beam CT systems, especially C-arms, are capable of diverse source-detector orbits. However, geometric calibration of these systems using conventional configurations of spherical fiducials (BBs) may be challenged for novel source-detector orbits and system geometries. In part, this is because the BB configurations are designed with careful forethought regarding the intended orbit so that BB marker projections do not overlap in projection views. Examples include helical arrangements of BBs (Rougee et al 1993 Proc. SPIE 1897 161-9) such that markers do not overlap in projections acquired from a circular orbit and circular arrangements of BBs (Cho et al 2005 Med. Phys. 32 968-83). As a more general alternative, this work proposes a calibration method based on an array of line-shaped, radio-opaque wire segments. With this method, geometric parameter estimation is accomplished by relating the 3D line equations representing the wires to the 2D line equations of their projections. The use of line fiducials simplifies many challenges with fiducial recognition and extraction in an orbit-independent manner. For example, their projections can overlap only mildly, for any gantry pose, as long as the wires are mutually non-coplanar in 3D. The method was tested in application to circular and non-circular trajectories in simulation and in real orbits executed using a mobile C-arm prototype for cone-beam CT. Results indicated high calibration accuracy, as measured by forward and backprojection/triangulation error metrics. Triangulation errors on the order of microns and backprojected ray deviations uniformly less than 0.2 mm were observed in both real and simulated orbits. Mean forward projection errors less than 0.1 mm were observed in a comprehensive sweep of different C-arm gantry angulations. Finally, successful integration of the method into a CT imaging chain was demonstrated in head phantom scans.

Fundamental limits of image registration performance: Effects of image noise and resolution in CT-guided interventions.

PURPOSE: In image-guided procedures, image acquisition is often performed primarily for the task of geometrically registering information from another image dataset, rather than detection / visualization of a particular feature. While the ability to detect a particular feature in an image has been studied extensively with respect to image quality characteristics (noise, resolution) and is an ongoing, active area of research, comparatively little has been accomplished to relate such image quality characteristics to registration performance. METHODS: To establish such a framework, we derived Cramer-Rao lower bounds (CRLB) for registration accuracy, revealing the underlying dependencies on image variance and gradient strength. The CRLB was analyzed as a function of image quality factors (in particular, dose) for various similarity metrics and compared to registration accuracy using CT images of an anthropomorphic head phantom at various simulated dose levels. Performance was evaluated in terms of root mean square error (RMSE) of the registration parameters. RESULTS: Analysis of the CRLB shows two primary dependencies: 1) noise variance (related to dose); and 2) sum of squared image gradients (related to spatial resolution and image content). Comparison of the measured RMSE to the CRLB showed that the best registration method, RMSE achieved the CRLB to within an efficiency factor of 0.21, and optimal estimators followed the predicted inverse proportionality between registration performance and radiation dose. CONCLUSIONS: Analysis of the CRLB for image registration is an important step toward understanding and evaluating an intraoperative imaging system with respect to a registration task. While the CRLB is optimistic in absolute performance, it reveals a basis for relating the performance of registration estimators as a function of noise content and may be used to guide acquisition parameter selection (e.g., dose) for purposes of intraoperative registration.

C-arm Positioning Using Virtual Fluoroscopy for Image-Guided Surgery.

INTRODUCTION: Fluoroscopically guided procedures often involve repeated acquisitions for C-arm positioning at the cost of radiation exposure and time in the operating room. A virtual fluoroscopy system is reported with the potential of reducing dose and time spent in C-arm positioning, utilizing three key advances: robust 3D-2D registration to a preoperative CT; real-time forward projection on GPU; and a motorized mobile C-arm with encoder feedback on C-arm orientation. METHOD: Geometric calibration of the C-arm was performed offline in two rotational directions (orbit α, orbit ß). Patient registration was performed using image-based 3D-2D registration with an initially acquired radiograph of the patient. This approach for patient registration eliminated the requirement for external tracking devices inside the operating room, allowing virtual fluoroscopy using commonly available systems in fluoroscopically guided procedures within standard surgical workflow. Geometric accuracy was evaluated in terms of projection distance error (PDE) in anatomical fiducials. A pilot study was conducted to evaluate the utility of virtual fluoroscopy to aid C-arm positioning in image guided surgery, assessing potential improvements in time, dose, and agreement between the virtual and desired view. RESULTS: The overall geometric accuracy of DRRs in comparison to the actual radiographs at various C-arm positions was PDE (mean ± std) = 1.6 ± 1.1 mm. The conventional approach required on average 8.0 ± 4.5 radiographs spent "fluoro hunting" to obtain the desired view. Positioning accuracy improved from 2.6° ± 2.3° (in α) and 4.1° ± 5.1° (in ß) in the conventional approach to 1.5° ± 1.3° and 1.8° ± 1.7°, respectively, with the virtual fluoroscopy approach. CONCLUSION: Virtual fluoroscopy could improve accuracy of C-arm positioning and save time and radiation dose in the operating room. Such a system could be valuable to training of fluoroscopy technicians as well as intraoperative use in fluoroscopically guided procedures.

Multi-stage 3D-2D registration for correction of anatomical deformation in image-guided spine surgery.

A multi-stage image-based 3D-2D registration method is presented that maps annotations in a 3D image (e.g. point labels annotating individual vertebrae in preoperative CT) to an intraoperative radiograph in which the patient has undergone non-rigid anatomical deformation due to changes in patient positioning or due to the intervention itself. The proposed method (termed msLevelCheck) extends a previous rigid registration solution (LevelCheck) to provide an accurate mapping of vertebral labels in the presence of spinal deformation. The method employs a multi-stage series of rigid 3D-2D registrations performed on sets of automatically determined and increasingly localized sub-images, with the final stage achieving a rigid mapping for each label to yield a locally rigid yet globally deformable solution. The method was evaluated first in a phantom study in which a CT image of the spine was acquired followed by a series of 7 mobile radiographs with increasing degree of deformation applied. Second, the method was validated using a clinical data set of patients exhibiting strong spinal deformation during thoracolumbar spine surgery. Registration accuracy was assessed using projection distance error (PDE) and failure rate (PDE > 20 mm-i.e. label registered outside vertebra). The msLevelCheck method was able to register all vertebrae accurately for all cases of deformation in the phantom study, improving the maximum PDE of the rigid method from 22.4 mm to 3.9 mm. The clinical study demonstrated the feasibility of the approach in real patient data by accurately registering all vertebral labels in each case, eliminating all instances of failure encountered in the conventional rigid method. The multi-stage approach demonstrated accurate mapping of vertebral labels in the presence of strong spinal deformation. The msLevelCheck method maintains other advantageous aspects of the original LevelCheck method (e.g. compatibility with standard clinical workflow, large capture range, and robustness against mismatch in image content) and extends capability to cases exhibiting strong changes in spinal curvature.

Intraoperative evaluation of device placement in spine surgery using known-component 3D-2D image registration.

Intraoperative x-ray radiography/fluoroscopy is commonly used to assess the placement of surgical devices in the operating room (e.g. spine pedicle screws), but qualitative interpretation can fail to reliably detect suboptimal delivery and/or breach of adjacent critical structures. We present a 3D-2D image registration method wherein intraoperative radiographs are leveraged in combination with prior knowledge of the patient and surgical components for quantitative assessment of device placement and more rigorous quality assurance (QA) of the surgical product. The algorithm is based on known-component registration (KC-Reg) in which patient-specific preoperative CT and parametric component models are used. The registration performs optimization of gradient similarity, removes the need for offline geometric calibration of the C-arm, and simultaneously solves for multiple component bodies, thereby allowing QA in a single step (e.g. spinal construct with 4-20 screws). Performance was tested in a spine phantom, and first clinical results are reported for QA of transpedicle screws delivered in a patient undergoing thoracolumbar spine surgery. Simultaneous registration of ten pedicle screws (five contralateral pairs) demonstrated mean target registration error (TRE) of 1.1 ± 0.1 mm at the screw tip and 0.7 ± 0.4° in angulation when a prior geometric calibration was used. The calibration-free formulation, with the aid of component collision constraints, achieved TRE of 1.4 ± 0.6 mm. In all cases, a statistically significant improvement (p < 0.05) was observed for the simultaneous solutions in comparison to previously reported sequential solution of individual components. Initial application in clinical data in spine surgery demonstrated TRE of 2.7 ± 2.6 mm and 1.5 ± 0.8°. The KC-Reg algorithm offers an independent check and quantitative QA of the surgical product using radiographic/fluoroscopic views acquired within standard OR workflow. Such intraoperative assessment could improve quality and safety, provide the opportunity to revise suboptimal constructs in the OR, and reduce the frequency of revision surgery.

Registration of MRI to intraoperative radiographs for target localization in spinal interventions.

Decision support to assist in target vertebra localization could provide a useful aid to safe and effective spine surgery. Previous solutions have shown 3D-2D registration of preoperative CT to intraoperative radiographs to reliably annotate vertebral labels for assistance during level localization. We present an algorithm (referred to as MR-LevelCheck) to perform 3D-2D registration based on a preoperative MRI to accommodate the increasingly common clinical scenario in which MRI is used instead of CT for preoperative planning. Straightforward adaptation of gradient/intensity-based methods appropriate to CT-to-radiograph registration is confounded by large mismatch and noncorrespondence in image intensity between MRI and radiographs. The proposed method overcomes such challenges with a simple vertebrae segmentation step using vertebra centroids as seed points (automatically defined within existing workflow). Forwards projections are computed using segmented MRI and registered to radiographs via gradient orientation (GO) similarity and the CMA-ES (covariance-matrix-adaptation evolutionary-strategy) optimizer. The method was tested in an IRB-approved study involving 10 patients undergoing cervical, thoracic, or lumbar spine surgery following preoperative MRI. The method successfully registered each preoperative MRI to intraoperative radiographs and maintained desirable properties of robustness against image content mismatch and large capture range. Robust registration performance was achieved with projection distance error (PDE) (median ± IQR) = 4.3 ± 2.6 mm (median ± IQR) and 0% failure rate. Segmentation accuracy for the continuous max-flow method yielded dice coefficient = 88.1 ± 5.2, accuracy = 90.6 ± 5.7, RMSE = 1.8 ± 0.6 mm, and contour affinity ratio (CAR) = 0.82 ± 0.08. Registration performance was found to be robust for segmentation methods exhibiting RMSE <3 mm and CAR >0.50. The MR-LevelCheck method provides a potentially valuable extension to a previously developed decision support tool for spine surgery target localization by extending its utility to preoperative MRI while maintaining characteristics of accuracy and robustness.

Automatic Masking for Robust 3D-2D Image Registration in Image-Guided Spine Surgery.

During spinal neurosurgery, patient-specific information, planning, and annotation such as vertebral labels can be mapped from preoperative 3D CT to intraoperative 2D radiographs via image-based 3D-2D registration. Such registration has been shown to provide a potentially valuable means of decision support in target localization as well as quality assurance of the surgical product. However, robust registration can be challenged by mismatch in image content between the preoperative CT and intraoperative radiographs, arising, for example, from anatomical deformation or the presence of surgical tools within the radiograph. In this work, we develop and evaluate methods for automatically mitigating the effect of content mismatch by leveraging the surgical planning data to assign greater weight to anatomical regions known to be reliable for registration and vital to the surgical task while removing problematic regions that are highly deformable or often occluded by surgical tools. We investigated two approaches to assigning variable weight (i.e., "masking") to image content and/or the similarity metric: (1) masking the preoperative 3D CT ("volumetric masking"); and (2) masking within the 2D similarity metric calculation ("projection masking"). The accuracy of registration was evaluated in terms of projection distance error (PDE) in 61 cases selected from an IRB-approved clinical study. The best performing of the masking techniques was found to reduce the rate of gross failure (PDE > 20 mm) from 11.48% to 5.57% in this challenging retrospective data set. These approaches provided robustness to content mismatch and eliminated distinct failure modes of registration. Such improvement was gained without additional workflow and has motivated incorporation of the masking methods within a system under development for prospective clinical studies.

3D-2D image registration for target localization in spine surgery: investigation of similarity metrics providing robustness to content mismatch.

In image-guided spine surgery, robust three-dimensional to two-dimensional (3D-2D) registration of preoperative computed tomography (CT) and intraoperative radiographs can be challenged by the image content mismatch associated with the presence of surgical instrumentation and implants as well as soft-tissue resection or deformation. This work investigates image similarity metrics in 3D-2D registration offering improved robustness against mismatch, thereby improving performance and reducing or eliminating the need for manual masking. The performance of four gradient-based image similarity metrics (gradient information (GI), gradient correlation (GC), gradient information with linear scaling (GS), and gradient orientation (GO)) with a multi-start optimization strategy was evaluated in an institutional review board-approved retrospective clinical study using 51 preoperative CT images and 115 intraoperative mobile radiographs. Registrations were tested with and without polygonal masks as a function of the number of multistarts employed during optimization. Registration accuracy was evaluated in terms of the projection distance error (PDE) and assessment of failure modes (PDE > 30 mm) that could impede reliable vertebral level localization. With manual polygonal masking and 200 multistarts, the GC and GO metrics exhibited robust performance with 0% gross failures and median PDE < 6.4 mm (±4.4 mm interquartile range (IQR)) and a median runtime of 84 s (plus upwards of 1-2 min for manual masking). Excluding manual polygonal masks and decreasing the number of multistarts to 50 caused the GC-based registration to fail at a rate of >14%; however, GO maintained robustness with a 0% gross failure rate. Overall, the GI, GC, and GS metrics were susceptible to registration errors associated with content mismatch, but GO provided robust registration (median PDE = 5.5 mm, 2.6 mm IQR) without manual masking and with an improved runtime (29.3 s). The GO metric improved the registration accuracy and robustness in the presence of strong image content mismatch. This capability could offer valuable assistance and decision support in spine level localization in a manner consistent with clinical workflow.

Deformable 3D-2D Registration of Known Components for Image Guidance in Spine Surgery.

A 3D-2D image registration method is reported for guiding the placement of surgical devices (e.g., K-wires). The solution registers preoperative CT (and planning data therein) to intraoperative radiographs and computes the pose, shape, and deformation parameters of devices (termed "components") known to be in the radiographic scene. The deformable known-component registration (dKC-Reg) method was applied in experiments emulating spine surgery to register devices (K-wires and spinal fixation rods) undergoing realistic deformation. A two-stage registration process (i) resolves patient pose from individual radiographs and (ii) registers components represented as polygonal meshes based on a B-spline model. The registration result can be visualized as overlay of the component in CT analogous to surgical navigation but without conventional trackers or fiducials. Target registration error in the tip and orientation of deformable K-wires was (1.5±0.9)mm and 0.6∘±0.2∘, respectively. For spinal fixation rods, the registered components achieved Hausdorff distance of 3.4 mm. Future work includes testing in cadaver and clinical data and extension to more generalized deformation and component models.