Machine Learning of Multi-Modal Tumor Imaging Reveals Trajectories of Response to Precision Treatment.

The standard assessment of response to cancer treatments is based on gross tumor characteristics, such as tumor size or glycolysis, which provide very indirect information about the effect of precision treatments on the pharmacological targets of tumors. Several advanced imaging modalities allow for the visualization of targeted tumor hallmarks. Descriptors extracted from these images can help establishing new classifications of precision treatment response. We propose a machine learning (ML) framework to analyze metabolic-anatomical-vascular imaging features from positron emission tomography, ultrafast Doppler, and computed tomography in a mouse model of paraganglioma undergoing anti-angiogenic treatment with sunitinib. Imaging features from the follow-up of sunitinib-treated (n = 8, imaged once-per-week/6-weeks) and sham-treated (n = 8, imaged once-per-week/3-weeks) mice groups were dimensionally reduced and analyzed with hierarchical clustering Analysis (HCA). The classes extracted from HCA were used with 10 ML classifiers to find a generalized tumor stage prediction model, which was validated with an independent dataset of sunitinib-treated mice. HCA provided three stages of treatment response that were validated using the best-performing ML classifier. The Gaussian naive Bayes classifier showed the best performance, with a training accuracy of 98.7 and an average area under curve of 100. Our results show that metabolic-anatomical-vascular markers allow defining treatment response trajectories that reflect the efficacy of an anti-angiogenic drug on the tumor target hallmark.

Optical biopsy mapping on endoscopic image mosaics with a marker-free probe.

Gastric cancer is the second leading cause of cancer-related deaths worldwide. Early diagnosis significantly increases the chances of survival; therefore, improved assisted exploration and screening techniques are necessary. Previously, we made use of an augmented multi-spectral endoscope by inserting an optical probe into the instrumentation channel. However, the limited field of view and the lack of markings left by optical biopsies on the tissue complicate the navigation and revisit of the suspect areas probed in-vivo. In this contribution two innovative tools are introduced to significantly increase the traceability and monitoring of patients in clinical practice: (i) video mosaicing to build a more comprehensive and panoramic view of large gastric areas; (ii) optical biopsy targeting and registration with the endoscopic images. The proposed optical flow-based mosaicing technique selects images that minimize texture discontinuities and is robust despite the lack of texture and illumination variations. The optical biopsy targeting is based on automatic tracking of a free-marker probe in the endoscopic view using deep learning to dynamically estimate its pose during exploration. The accuracy of pose estimation is sufficient to ensure a precise overlapping of the standard white-light color image and the hyperspectral probe image, assuming that the small target area of the organ is almost flat. This allows the mapping of all spatio-temporally tracked biopsy sites onto the panoramic mosaic. Experimental validations are carried out from videos acquired on patients in hospital. The proposed technique is purely software-based and therefore easily integrable into clinical practice. It is also generic and compatible to any imaging modality that connects to a fiberscope.

Pose estimation of a markerless fiber bundle for endoscopic optical biopsy.

Purpose: We present a markerless vision-based method for on-the-fly three-dimensional (3D) pose estimation of a fiberscope instrument to target pathologic areas in the endoscopic view during exploration. Approach: A 2.5-mm-diameter fiberscope is inserted through the endoscope's operating channel and connected to an additional camera to perform complementary observation of a targeted area such as a multimodal magnifier. The 3D pose of the fiberscope is estimated frame-by-frame by maximizing the similarity between its silhouette (automatically detected in the endoscopic view using a deep learning neural network) and a cylindrical shape bound to a kinematic model reduced to three degrees-of-freedom. An alignment of the cylinder axis, based on Plücker coordinates from the straight edges detected in the image, makes convergence faster and more reliable. Results: The performance on simulations has been validated with a virtual trajectory mimicking endoscopic exploration and on real images of a chessboard pattern acquired with different endoscopic configurations. The experiments demonstrated a good accuracy and robustness of the proposed algorithm with errors of 0.33 ± 0.68 mm in distance position and 0.32 ± 0.11 deg in axis orientation for the 3D pose estimation, which reveals its superiority over previous approaches. This allows multimodal image registration with sufficient accuracy of < 3 pixels . Conclusion: Our pose estimation pipeline was executed on simulations and patterns; the results demonstrate the robustness of our method and the potential of fiber-optical instrument image-based tracking for pose estimation and multimodal registration. It can be fully implemented in software and therefore easily integrated into a routine clinical environment.

Total attenuation compensation for backscatter coefficient estimation using full angular spatial compounding.

The backscatter coefficient (BSC) quantifies the frequency-dependent reflectivity of tissues. Accurate estimation of the BSC is only possible with the knowledge of the attenuation coefficient slope (ACS) of the tissues under examination. In this study, the use of attenuation maps constructed using full angular spatial compounding (FASC) is proposed for attenuation compensation when imaging integrated BSCs. Experimental validation of the proposed approach was obtained using two cylindrical physical phantoms with off-centered inclusions having different ACS and BSC values than the background, and in a phantom containing an ex vivo chicken breast sample embedded in an agar matrix. With the phantom data, three different ACS maps were employed for attenuation compensation: (1) a ground truth ACS map constructed using insertion loss techniques, (2) the estimated ACS map using FASC attenuation imaging, and (3) a uniform ACS map with a value of 0.5 dBcm\protect \relax \special {t4ht=-}1MHz\protect \relax \special {t4ht=-}1, which is commonly used to represent attenuation in soft tissues. Comparable results were obtained when using the ground truth and FASC-estimated ACS maps in term of inclusion detectability and estimation accuracy, with averaged fractional error below 2.8 dB in both phantoms. Conversely, the use of the homogeneous ACS map resulted in higher levels of fractional error (>10 dB), which demonstrates the importance of an accurate attenuation compensation. The results with the ex vivo tissue sample were consistent with the observations using the physical phantoms, with the FASC-derived ACS map providing comparable BSC images to those formed using the ground truth ACS map and more accurate than those BSC images formed using a uniform ACS. These results suggest that BSCs can be reliably estimated using FASC when a self-consistent attenuation compensation stemming from prior estimation of an accurate ACS map is used.

Markerless tracking of micro-endoscope for optical biopsy in stomach.

This paper presents a landmark-free approach to estimate the fiberscope pose during endoscopic exploration for in-vivo optical biopsy. The fiberscope pose is estimated by fitting the projection of a virtual 3D cylinder into the endoscopic images. The cylinder axis is estimated based on the apparent contours using Plücker coordinates and its insertion is estimated by maximizing the similarity between binary masks. The performance of the method is evaluated on simulations: the mean Euclidian distance of fiberscopic tip between estimated pose and ground truth is 0.158 ± 0.113 mm. The in-vivo performance is assessed in two endoscopic sequences by comparing automatic RCF and manual segmentations in terms of angular deviation of the axis and Euclidian distance between the tip location. The estimation of the relative position of both cameras allows to perform registration between the two image modalities.