Self-supervised representation learning for surgical activity recognition.

PURPOSE: Virtual reality-based simulators have the potential to become an essential part of surgical education. To make full use of this potential, they must be able to automatically recognize activities performed by users and assess those. Since annotations of trajectories by human experts are expensive, there is a need for methods that can learn to recognize surgical activities in a data-efficient way. METHODS: We use self-supervised training of deep encoder-decoder architectures to learn representations of surgical trajectories from video data. These representations allow for semi-automatic extraction of features that capture information about semantically important events in the trajectories. Such features are processed as inputs of an unsupervised surgical activity recognition pipeline. RESULTS: Our experiments document that the performance of hidden semi-Markov models used for recognizing activities in a simulated myomectomy scenario benefits from using features extracted from representations learned while training a deep encoder-decoder network on the task of predicting the remaining surgery progress. CONCLUSION: Our work is an important first step in the direction of making efficient use of features obtained from deep representation learning for surgical activity recognition in settings where only a small fraction of the existing data is annotated by human domain experts and where those annotations are potentially incomplete.

A modified McKinnon-Bates (MKB) algorithm for improved 4D cone-beam computed tomography (CBCT) of the lung.

PURPOSE: Four-dimensional (4D) cone-beam computed tomography (CBCT) of the lung is an effective tool for motion management in radiotherapy but presents a challenge because of slow gantry rotation times. Sorting the individual projections by breathing phase and using an established technique such as Feldkamp-Davis-Kress (FDK) to generate corresponding phase-correlated (PC) three-dimensional (3D) images results in reconstructions (FDK-PC) that often contain severe streaking artifacts due to the sparse angular sampling distributions. These can be reduced by further slowing down the gantry at the expense of incurring unwanted increases in scan times and dose. A computationally efficient alternative is the McKinnon-Bates (MKB) reconstruction algorithm that has shown promise in reducing view aliasing-induced streaking but can produce ghosting artifacts that reduce contrast and impede the determination of motion trajectories. The purpose of this work was to identify and correct shortcomings in the MKB algorithm. METHODS: In the general MKB approach, a time-averaged 3D prior image is first reconstructed. The prior is then forward-projected at the same angles as the original projection data creating time-averaged reprojections. These reprojections are subsequently subtracted from the original (unblurred) projections to create motion-encoded difference projections. The difference projections are reconstructed into PC difference images that are added to the well-sampled 3D prior to create the higher quality 4D image. The cause of the ghosting in the traditional 4D MKB images was studied and traced to motion-induced streaking in the prior that, when reprojected, has the undesirable effect of re-encoding for motion in what should be a purely time-averaged reprojection. A new method, designated as the modified McKinnon-Bates (mMKB) algorithm, was developed based on destreaking the prior. This was coupled with a postprocessing 4D bilateral filter for noise suppression and edge preservation (mMKBbf ). The algorithms were tested with the 4D XCAT phantom using four simulated scan times (57, 60, 120, 180 s) and with two in vivo thorax studies (acquisition time of 60 and 90 s). Contrast-to-noise ratios (CNRs) of the target lesions and overall visual quality of the images were assessed. RESULTS: Prior destreaking (mMKB algorithm) reduced ghosting artifacts and increased CNRs for all cases, with the biggest impacts seen in the end inhale (EI) and end exhale (EE) phases of the respiratory cycle. For the XCAT phantom, mMKB lesion CNR was 44% higher than the MKB lesion CNR and was 81% higher than the FDK-PC lesion CNR (EI and EE phases). The bilateral filter provided a further average CNR improvement of 87% with the highest increases associated with longer scan times. Across all phases and scan times, the maximum mMKBbf -to-FDK-PC CNR improvement was over 300%. In vivo results agreed with XCAT results. Significantly less ghosting was observed throughout the mMKB images including near the lesions-of-interest and the diaphragm allowing for, in one case, visualization of a small tumor with nearly 30 mm of motion. The maximum FDK-PC-to-MKBbf CNR improvement for Patient 1's lesion was 261% and for Patient 2's lesion was 318%. CONCLUSIONS: The 4D mMKB algorithm yields good quality coronal and sagittal images in the thorax that may provide sufficient information for patient verification.

Artifact-resistant motion estimation with a patient-specific artifact model for motion-compensated cone-beam CT.

PURPOSE: In image-guided radiation therapy (IGRT) valuable information for patient positioning, dose verification, and adaptive treatment planning is provided by an additional kV imaging unit. However, due to the limited gantry rotation speed during treatment the typical acquisition time is quite long. Tomographic images of the thorax suffer from motion blurring or, if a gated 4D reconstruction is performed, from significant streak artifacts. Our purpose is to provide a method that reliably estimates respiratory motion in presence of severe artifacts. The estimated motion vector fields are then used for motion-compensated image reconstruction to provide high quality respiratory-correlated 4D volumes for on-board cone-beam CT (CBCT) scans. METHODS: The proposed motion estimation method consists of a model that explicitly addresses image artifacts because in presence of severe artifacts state-of-the-art registration methods tend to register artifacts rather than anatomy. Our artifact model, e.g., generates streak artifacts very similar to those included in the gated 4D CBCT images, but it does not include respiratory motion. In combination with a registration strategy, the model gives an error estimate that is used to compensate the corresponding errors of the motion vector fields that are estimated from the gated 4D CBCT images. The algorithm is tested in combination with a cyclic registration approach using temporal constraints and with a standard 3D-3D registration approach. A qualitative and quantitative evaluation of the motion-compensated results was performed using simulated rawdata created on basis of clinical CT data. Further evaluation includes patient data which were scanned with an on-board CBCT system. RESULTS: The model-based motion estimation method is nearly insensitive to image artifacts of gated 4D reconstructions as they are caused by angular undersampling. The motion is accurately estimated and our motion-compensated image reconstruction algorithm can correct for it. Motion artifacts of 3D standard reconstruction are significantly reduced, while almost no new artifacts are introduced. CONCLUSIONS: Using the artifact model allows to accurately estimate and compensate for patient motion, even if the initial reconstructions are of very low image quality. Using our approach together with a cyclic registration algorithm yields a combination which shows almost no sensitivity to sparse-view artifacts and thus ensures both high spatial and high temporal resolution.

Self-adapting cyclic registration for motion-compensated cone-beam CT in image-guided radiation therapy.

PURPOSE: In image-guided radiation therapy an additional kV imaging system next to the linear particle accelerator provides information for an accurate patient positioning. However, the acquisition time of the system is much longer than the patient's breathing cycle due to the low gantry rotation speed. Our purpose is a cyclic registration in the context of motion-compensated image reconstruction that provides high quality respiratory-correlated 4D volumes for on-board flat panel detector cone-beam CT scans. METHODS: Based on the small motion assumption, widely used within registration algorithms, a strategy is developed for motion estimation. In this strategy temporal restrictions are incorporated, for example, the cyclic motion patterns of respiration. The resultant cyclic registration method is to show less sensitivity on image artifacts, in particular on artifacts due to projection data sparsification. Using a new cyclic registration method a motion estimation is performed on respiratory-correlated reconstructions, and the obtained motion vector fields are used for motion compensation. RESULTS: The proposed cyclic registration is evaluated in the context of motion-compensated image reconstruction using simulated data and patient data. Motion artifacts of 3D standard reconstructions can be significantly reduced by the resulting cyclic motion compensation. The method outperforms the respiratory-correlated reconstructions regarding sparse-view artifacts and maintains the high temporal resolution at the same time. Image artifacts show only minor to almost no effect on the motion estimation using the cyclic registration. CONCLUSIONS: The cyclic motion compensation approach provides respiratory-correlated volumes with high image quality. The cyclic motion estimation is of such low sensitivity to sparse-view artifacts, that it is capable to determine high quality motion vector fields based on reconstructions of low sampled data.

Correction for patient table-induced scattered radiation in cone-beam computed tomography (CBCT).

PURPOSE: In image-guided radiotherapy, an artifact typically seen in axial slices of x-ray cone-beam computed tomography (CBCT) reconstructions is a dark region or "black hole" situated below the scan isocenter. The authors trace the cause of the artifact to scattered radiation produced by radiotherapy patient tabletops and show it is linked to the use of the offset-detector acquisition mode to enlarge the imaging field-of-view. The authors present a hybrid scatter kernel superposition (SKS) algorithm to correct for scatter from both the object-of-interest and the tabletop. METHODS: Monte Carlo simulations and phantom experiments were first performed to identify the source of the black hole artifact. For correction, a SKS algorithm was developed that uses separate kernels to estimate scatter from the patient tabletop and the object-of-interest. Each projection is divided into two regions, one defined by the shadow cast by the tabletop on the imager and one defined by the unshadowed region. The region not shadowed by the tabletop is processed using the recently developed fast adaptive scatter kernel superposition (fASKS) method which employs asymmetric kernels that best model scatter transport through bodylike objects. The shadowed region is convolved with a combination of slab-derived symmetric SKS kernels and asymmetric fASKS kernels. The composition of the hybrid kernels is projection-angle-dependent. To test the algorithm, pelvis phantom and in vivo data were acquired using a CBCT test stand, a Varian Acuity simulator, and a Varian On-Board Imager, all of which have similar geometries and components. Artifact intensities and Hounsfield unit (HU) accuracies in the reconstructions were assessed before and after the correction. RESULTS: The hybrid kernel algorithm provided effective correction and produced substantially better scatter estimates than the symmetric SKS or asymmetric fASKS methods alone. HU nonuniformities in the reconstructed pelvis phantom were reduced from 220 to 50 HU (i.e., 22%-5%). In the in vivo scans, the black hole artifact was reduced by up to 147 HU, a 73% improvement, and anatomical details in the prostate and rectum areas were made considerably more visible. CONCLUSIONS: Radiotherapy tabletops, which are generally flatter and larger than those used for diagnostic CT, can produce significant scatter-related artifacts. The proposed hybrid SKS algorithm accurately estimates scatter from both the object-of-interest and the patient tabletop, and resulting image uniformities and HU accuracies are greatly improved.

An investigation of 4D cone-beam CT algorithms for slowly rotating scanners.

PURPOSE: To evaluate several algorithms for 4D cone-beam computed tomography (4D CBCT) with slow rotating devices. 4D CBCT is used to perform phase-correlated (PC) reconstructions of moving objects, such as breathing patients, for example. Such motion phase-dependent reconstructions are especially useful for updating treatment plans in radiation therapy. The treatment plan can be registered more precisely to the motion of the tumor and, in consequence, the irradiation margins for the treatment, the so-called planning target volume, can be reduced significantly METHODS: In the study, several algorithms were evaluated for kilovoltage cone-beam CT units attached to linear particle accelerators. The reconstruction algorithms were the conventional PC reconstruction, the McKinnon-Bates (MKB) algorithm, the prior image constrained compressed sensing (PICCS) approach, a total variation minimization (ASD-POCS) algorithm, and the auto-adaptive phase correlation (AAPC) algorithm. For each algorithm, the same motion-affected raw data were used, i.e., one simulated and one measured data set. The reconstruction results from the authors' implementation of these algorithms were evaluated regarding their noise and artifact levels, their residual motion blur, and their computational complexity and convergence. RESULTS: In general, it turned out that the residual motion blur was lowest in those algorithms which exclusively use data from a single motion phase. Algorithms using the image from the full data set as initialization or as a reference for the reconstruction were not capable of fully removing the motion blurring. The iterative algorithms, especially approaches based on total variation minimization, showed lower noise and artifact levels but were computationally complex. The conventional methods based on a single filtered backprojection were computationally inexpensive but suffered from higher noise and streak artifacts which limit the usability. In contrast, these methods showed to be less demanding and more predictable in their outcome than the total variation minimization based approaches. CONCLUSIONS: The reconstruction algorithms including at least one iterative step can reduce the 4 CBCT specific artifacts. Nevertheless, the algorithms that use the full data set, at least for initialization, such as MKB and PICCS in the authors' implementation, are only a trade-off and may not fully achieve the optimal temporal resolution. A predictable image quality as seen in conventional reconstruction methods, i.e., without total variation minimization, is a desirable property for reconstruction algorithms. Fast, iterative approaches such as the MKB can therefore be seen as a suitable tradeoff.

Cone-beam CT image reconstruction with extended z range.

In circular cone-beam CT the Feldkamp [Feldkamp-Davis-Kress (FDK)] algorithm is the most prominent image reconstruction algorithm. For example, in radiation oncology images reconstructed with the Feldkamp algorithm are used for accurate patient positioning. The scan and reconstruction volumes are limited by the size of the flat panel detector. Flat panel detectors, however, are expensive and difficult to manufacture in large size. For numerous treatment techniques, extending this scan volume would be very beneficial. In most applications, data from 360 degrees or more are available. However, usually only those slices are reconstructed where each pixel is seen under the full 360 degree range. Yet for a 360 degree scan there are regions that are seen by less than 360 degrees, namely, those that lie further off the plane of the circular source trajectory. Performing a reconstruction also for those slices where all voxels are seen at least by 180 degrees will extend the z range and therefore increase the dose usage. In this work a new method is presented that reconstructs also those slices where some or all pixels receive less than 360 degrees but at least 180 degrees of the data. The procedure significantly increases the longitudinal range of the reconstructed volume. As opposed to the existing techniques, the proposed method does not necessitate any multiple convolutions or multiple backprojections, lending itself therefore for a very efficient implementation. To validate the abilities of the extended reconstruction, the authors performed an evaluation of the image quality by using simulated and measured CT data. The method shows good image quality on simulated phantom data as well as on clinical patient scans. Image noise and spatial resolution behave as expected. This means that the noise equals FDK values in the normal region and increases in the extended region due to reduced data redundancies. The extended Feldkamp demonstrates its ability to extend the reconstructable z range and appears to be useful in clinical practice.

Autoadaptive phase-correlated (AAPC) reconstruction for 4D CBCT.

PURPOSE: Kilovoltage cone-beam computed tomography (CBCT) is widely used in image-guided radiation therapy for exact patient positioning prior to the treatment. However, producing time series of volumetric images (4D CBCT) of moving anatomical structures remains challenging. The presented work introduces a novel method, combining high temporal resolution inside anatomical regions with strong motion and image quality improvement in regions with little motion. METHODS: In the proposed method, the projections are divided into regions that are subject to motion and regions at rest. The latter ones will be shared among phase bins, leading thus to an overall reduction in artifacts and noise. An algorithm based on the concept of optical flow was developed to analyze motion-induced changes between projections. The technique was optimized to distinguish patient motion and motion deriving from gantry rotation. The effectiveness of the method is shown in numerical simulations and patient data. RESULTS: The images reconstructed from the presented method yield an almost the same temporal resolution in the moving volume segments as a conventional phase-correlated reconstruction, while reducing the noise in the motionless regions down to the level of a standard reconstruction without phase correlation. The proposed simple motion segmentation scheme is yet limited to rotation speeds of less than 3 degrees/s. CONCLUSIONS: The method reduces the noise in the reconstruction and increases the image quality. More data are introduced for each phase-correlated reconstruction, and therefore the applied dose is used more efficiently.

2D-spatially-selective real-time magnetic resonance imaging for the assessment of microvascular function and its relation to the cardiovascular risk profile.

BACKGROUND: While local endothelial dysfunction of conduit arteries is well recognized as an early step in atherogenesis, contradictory observations are reported with regard to alterations in the microcirculation and their association with cardiovascular risk factors (RFs). A real-time MR approach was developed to investigate the relationship between the RFs profile and microcirculatory alterations assessed as impairment of reactive hyperemic flow in the leg circulation. METHODS: The MR technique was applied to patients (n = 17, Pats1) with 1.8 +/- 0.8 RFs but without peripheral arterial occlusive disease (PAD), to age-matched healthy controls (n = 13, Con1), to young controls (n = 12, 23 +/- 4 y), and to patients with RFs and PAD (n = 8, Pats2). RESULTS: Superficial femoral artery (SFA) peak hyperemic flow in Pats1 was reduced vs Con1 (24.6 +/- 4.2 vs 30.4 +/- 7.3 mL min-1 100 mL-1 calf tissue, p < 0.02), and minimal vascular resistance increased incrementally with the number of RFs and with Framingham and Procam risk scores. Flow-mediated vasodilation (FMD) of the SFA was blunted in both Pats1 and Con1 (-0.5 +/- 3.4% and +0.6 +/- 3.2%, respectively, both ns vs 0). In young controls, peak hyperemic flow (30.1 +/- 3.3 mL min-1. 100 mL-1) and endothelium-independent vasodilation (9.2 +/- 10.0%) were preserved, while FMD was minimal (2.0 +/- 5.9%,p < 0.02 vs endothelium-independent vasodilation). In Pats2, peak hyperemic flow was severely reduced (12.2 +/- 3.6 mL min-1 100 mL-1, p < 0.0003 vs Con1 and Pats1), and both FMD and endothelium-independent vasodilation were absent. CONCLUSIONS: Reactive hyperemic flow in the SFA, reflecting microcirculatory function of the lower limb, gradually decreases with increasing cardiovascular risk suggesting a role for microvascular dysfunction in atherogenesis. The presented MR approach might become a valuable tool to study (micro)-vascular pathophysiology.

Assessing arterial blood flow and vessel area variations using real-time zonal phase-contrast MRI.

PURPOSE: To measure peripheral artery function using a real-time phase-contrast (PC)-MRI sequence with tailored image-processing algorithms for flow computation. MATERIALS AND METHODS: An approach to real-time flow measurements was developed based on two-dimensional spatially selective excitation pulses and consecutive tailored processing of the data to derive blood flow and vessel area variations. The data acquisition strategy allows for flow measurements at high spatial and temporal resolutions of 1 mm(2) and 50 msec, respectively. In postprocessing the vessel area is automatically extracted using correlation measures in conjunction with morphological image operators. By means of in vitro and in vivo validations, it is shown that the current methods provide accurate and reproducible measurements of flow and vessel area variations. RESULTS: In vitro the comparison between the lumen area measured with the presented method and the values obtained by caliper gauge measurement showed a difference of 3.4% +/- 3.4% (mean +/- 2 SD). Similarly, the comparison between the stroke volumes determined with the presented method and by stopwatch and bucket measurements yielded a difference of 6.1% +/- 2.1%. In vivo the results from the real-time measurements for lumen area and stroke volume were compared with those from a gated PC-MRI technique with differences of 4.8% +/- 14% and 3.0% +/- 24.7%, respectively. CONCLUSION: The presented method constitutes a reliable tool set for quantifying the variations of blood flow and lumen area in the superficial femoral artery during reactive hyperemia and for studying their correlation with cardiovascular risk factors.

Calibration of echo-planar 2D-selective RF excitation pulses.

Echo-planar radiofrequency (RF) pulses (EPP) are increasingly being used for 2D-selective excitation in MRI. Pulse schemes of this kind are susceptible to eddy-current effects, timing imperfections, and anisotropy of the gradient system. As a consequence, practical EPP implementations have been restricted to robust fly-back strategies that use only every other leg of the echo-planar trajectory for RF transmission. The present work is dedicated to enabling forward-backward EPP with RF transmission during each k-space segment, hence doubling the pulses' time efficiency. This is accomplished by comprehensive pulse calibration based on preparatory measurements of the system imperfections, including potential gradient anisotropy. The effectiveness of the method is demonstrated in vitro and in vivo. By doubling the speed of k-space coverage, the proposed method enhances the potential of EPP for numerous applications. For example, motion-sensitive techniques benefit from shorter feasible echo times (TEs) and improved excitation profiles resulting from reduced in-pulse motion. In sequences with fast repetition, shorter EPP help reduce the overall scan duration. Alternatively, the higher time efficiency of forward-backward EPP can enhance their spatial selectivity.

Single breath-hold extended free-breathing navigator-gated three-dimensional coronary MRA.

PURPOSE: To acquire the center of k-space while extending three-dimensional free-breathing navigator-gated coronary magnetic resonance (MR) angiography by an initial single breath-hold. MATERIALS AND METHODS: This approach was successfully applied in eight healthy adult subjects. Resulting images were compared with conventionally acquired free-breathing navigator-gated MR angiograms. RESULTS: The acquisition of k-space center during the single breath-hold resulted in a 26% increase (P < 0.05) of signal-to-noise ratio. Visible length of the right coronary artery, as well as contrast-to-noise ratio between the blood and the myocardial muscle, were identical. CONCLUSION: The breath-hold extension was shown to be a valuable technique that may be combined with first-pass contrast-enhanced MR imaging.