Machine learning analysis of multispectral imaging and clinical risk factors to predict amputation wound healing.

OBJECTIVE: Prediction of amputation wound healing is challenging due to the multifactorial nature of critical limb ischemia and lack of objective assessment tools. Up to one-third of amputations require revision to a more proximal level within 1 year. We tested a novel wound imaging system to predict amputation wound healing at initial evaluation. METHODS: Patients planned to undergo amputation due to critical limb ischemia were prospectively enrolled. Clinicians evaluated the patients in traditional fashion, and all clinical decisions for amputation level were determined by the clinician's judgement. Multispectral images of the lower extremity were obtained preoperatively using a novel wound imaging system. Clinicians were blinded to the machine analysis. A standardized wound healing assessment was performed on postoperative day 30 by physical exam to determine whether the amputation site achieved complete healing. If operative revision or higher level of amputation was required, this was undertaken based solely upon the provider's clinical judgement. A machine learning algorithm combining the multispectral imaging data with patient clinical risk factors was trained and tested using cross-validation to measure the wound imaging system's accuracy of predicting amputation wound healing. RESULTS: A total of 22 patients undergoing 25 amputations (10 toe, five transmetatarsal, eight below-knee, and two above-knee amputations) were enrolled. Eleven amputations (44%) were non-healing after 30 days. The machine learning algorithm had 91% sensitivity and 86% specificity for prediction of non-healing amputation sites (area under curve, 0.89). CONCLUSIONS: This pilot study suggests that a machine learning algorithm combining multispectral wound imaging with patient clinical risk factors may improve prediction of amputation wound healing and therefore decrease the need for reoperation and incidence of delayed healing. We propose that this, in turn, may offer significant cost savings to the patient and health system in addition to decreasing length of stay for patients.

A Bayesian hidden Potts mixture model for analyzing lung cancer pathology images.

Digital pathology imaging of tumor tissues, which captures histological details in high resolution, is fast becoming a routine clinical procedure. Recent developments in deep-learning methods have enabled the identification, characterization, and classification of individual cells from pathology images analysis at a large scale. This creates new opportunities to study the spatial patterns of and interactions among different types of cells. Reliable statistical approaches to modeling such spatial patterns and interactions can provide insight into tumor progression and shed light on the biological mechanisms of cancer. In this article, we consider the problem of modeling a pathology image with irregular locations of three different types of cells: lymphocyte, stromal, and tumor cells. We propose a novel Bayesian hierarchical model, which incorporates a hidden Potts model to project the irregularly distributed cells to a square lattice and a Markov random field prior model to identify regions in a heterogeneous pathology image. The model allows us to quantify the interactions between different types of cells, some of which are clinically meaningful. We use Markov chain Monte Carlo sampling techniques, combined with a double Metropolis-Hastings algorithm, in order to simulate samples approximately from a distribution with an intractable normalizing constant. The proposed model was applied to the pathology images of $205$ lung cancer patients from the National Lung Screening trial, and the results show that the interaction strength between tumor and stromal cells predicts patient prognosis (P = $0.005$). This statistical methodology provides a new perspective for understanding the role of cell-cell interactions in cancer progression.

Microvessel prediction in H&E Stained Pathology Images using fully convolutional neural networks.

BACKGROUND: Pathological angiogenesis has been identified in many malignancies as a potential prognostic factor and target for therapy. In most cases, angiogenic analysis is based on the measurement of microvessel density (MVD) detected by immunostaining of CD31 or CD34. However, most retrievable public data is generally composed of Hematoxylin and Eosin (H&E)-stained pathology images, for which is difficult to get the corresponding immunohistochemistry images. The role of microvessels in H&E stained images has not been widely studied due to their complexity and heterogeneity. Furthermore, identifying microvessels manually for study is a labor-intensive task for pathologists, with high inter- and intra-observer variation. Therefore, it is important to develop automated microvessel-detection algorithms in H&E stained pathology images for clinical association analysis. RESULTS: In this paper, we propose a microvessel prediction method using fully convolutional neural networks. The feasibility of our proposed algorithm is demonstrated through experimental results on H&E stained images. Furthermore, the identified microvessel features were significantly associated with the patient clinical outcomes. CONCLUSIONS: This is the first study to develop an algorithm for automated microvessel detection in H&E stained pathology images.

Three-dimensional image authentication scheme using sparse phase information in double random phase encoded integral imaging.

In recent years, many studies have focused on authentication of two-dimensional (2D) images using double random phase encryption techniques. However, there has been little research on three-dimensional (3D) imaging systems, such as integral imaging, for 3D image authentication. We propose a 3D image authentication scheme based on a double random phase integral imaging method. All of the 2D elemental images captured through integral imaging are encrypted with a double random phase encoding algorithm and only partial phase information is reserved. All the amplitude and other miscellaneous phase information in the encrypted elemental images is discarded. Nevertheless, we demonstrate that 3D images from integral imaging can be authenticated at different depths using a nonlinear correlation method. The proposed 3D image authentication algorithm can provide enhanced information security because the decrypted 2D elemental images from the sparse phase cannot be easily observed by the naked eye. Additionally, using sparse phase images without any amplitude information can greatly reduce data storage costs and aid in image compression and data transmission.

Automated tracking of temporal displacements of a red blood cell obtained by time-lapse digital holographic microscopy.

Red blood cell (RBC) phase images that are numerically reconstructed by digital holographic microscopy (DHM) can describe the cell structure and dynamics information beneficial for a quantitative analysis of RBCs. However, RBCs investigated with time-lapse DHM undergo temporal displacements when their membranes are loosely attached to the substrate during sedimentation on a glass surface or due to the microscope drift. Therefore, we need to develop a tracking algorithm to localize the same RBC among RBC image sequences and dynamically monitor its biophysical cell parameters; this information is helpful for studies on RBC-related diseases and drug tests. Here, we propose a method, which is a combination of the mean-shift algorithm and Kalman filter, to track a single RBC and demonstrate that the optical path length of the single RBC can be continually extracted from the tracked RBC. The Kalman filter is utilized to predict the target RBC position in the next frame. Then, the mean-shift algorithm starts execution from the predicted location, and a robust kernel, which is adaptive to changes in the RBC scale, shape, and direction, is designed to improve the accuracy of the tracking. Finally, the tracked RBC is segmented and parameters such as the RBC location are extracted to update the Kalman filter and the kernel function for mean-shift tracking; the characteristics of the target RBC are dynamically observed. Experimental results show the feasibility of the proposed algorithm.

Efficient asymmetric image authentication schemes based on photon counting-double random phase encoding and RSA algorithms.

Recently, double random phase encoding (DRPE) has been integrated with the photon counting (PC) imaging technique for the purpose of secure image authentication. In this scheme, the same key should be securely distributed and shared between the sender and receiver, but this is one of the most vexing problems of symmetric cryptosystems. In this study, we propose an efficient asymmetric image authentication scheme by combining the PC-DRPE and RSA algorithms, which solves key management and distribution problems. The retrieved image from the proposed authentication method contains photon-limited encrypted data obtained by means of PC-DRPE. Therefore, the original image can be protected while the retrieved image can be efficiently verified using a statistical nonlinear correlation approach. Experimental results demonstrate the feasibility of our proposed asymmetric image authentication method.

Automated multi-parameter measurement of cardiomyocytes dynamics with digital holographic microscopy.

Compounds tested during drug development may have adverse effects on the heart; therefore all new chemical entities have to undergo extensive preclinical assessment for cardiac liability. Conventional intensity-based imaging techniques are not robust enough to provide detailed information for cell structure and the captured images result in low-contrast, especially to cell with semi-transparent or transparent feature, which would affect the cell analysis. In this paper we show, for the first time, that digital holographic microscopy (DHM) integrated with information processing algorithms automatically provide dynamic quantitative phase profiles of beating cardiomyocytes. We experimentally demonstrate that relevant parameters of cardiomyocytes can be obtained by our automated algorithm based on DHM phase signal analysis and used to characterize the physiological state of resting cardiomyocytes. Our study opens the possibility of automated quantitative analysis of cardiomyocyte dynamics suitable for further drug safety testing and compounds selection as a new paradigm in drug toxicity screens.

Avalanche and bit independence characteristics of double random phase encoding in the Fourier and Fresnel domains.

In this work, we evaluate the avalanche effect and bit independence properties of the double random phase encoding (DRPE) algorithm in the Fourier and Fresnel domains. Experimental results show that DRPE has excellent bit independence characteristics in both the Fourier and Fresnel domains. However, DRPE achieves better avalanche effect results in the Fresnel domain than in the Fourier domain. DRPE gives especially poor avalanche effect results in the Fourier domain when only one bit is changed in the plaintext or in the encryption key. Despite this, DRPE shows satisfactory avalanche effect results in the Fresnel domain when any other number of bits changes in the plaintext or in the encryption key. To the best of our knowledge, this is the first report on the avalanche effect and bit independence behaviors of optical encryption approaches for bit units.

Simultaneous reconstruction of multiple depth images without off-focus points in integral imaging using a graphics processing unit.

The reconstruction of multiple depth images with a ray back-propagation algorithm in three-dimensional (3D) computational integral imaging is computationally burdensome. Further, a reconstructed depth image consists of a focus and an off-focus area. Focus areas are 3D points on the surface of an object that are located at the reconstructed depth, while off-focus areas include 3D points in free-space that do not belong to any object surface in 3D space. Generally, without being removed, the presence of an off-focus area would adversely affect the high-level analysis of a 3D object, including its classification, recognition, and tracking. Here, we use a graphics processing unit (GPU) that supports parallel processing with multiple processors to simultaneously reconstruct multiple depth images using a lookup table containing the shifted values along the x and y directions for each elemental image in a given depth range. Moreover, each 3D point on a depth image can be measured by analyzing its statistical variance with its corresponding samples, which are captured by the two-dimensional (2D) elemental images. These statistical variances can be used to classify depth image pixels as either focus or off-focus points. At this stage, the measurement of focus and off-focus points in multiple depth images is also implemented in parallel on a GPU. Our proposed method is conducted based on the assumption that there is no occlusion of the 3D object during the capture stage of the integral imaging process. Experimental results have demonstrated that this method is capable of removing off-focus points in the reconstructed depth image. The results also showed that using a GPU to remove the off-focus points could greatly improve the overall computational speed compared with using a CPU.

A multispectral photon-counting double random phase encoding scheme for image authentication.

In this paper, we propose a new method for color image-based authentication that combines multispectral photon-counting imaging (MPCI) and double random phase encoding (DRPE) schemes. The sparsely distributed information from MPCI and the stationary white noise signal from DRPE make intruder attacks difficult. In this authentication method, the original multispectral RGB color image is down-sampled into a Bayer image. The three types of color samples (red, green and blue color) in the Bayer image are encrypted with DRPE and the amplitude part of the resulting image is photon counted. The corresponding phase information that has nonzero amplitude after photon counting is then kept for decryption. Experimental results show that the retrieved images from the proposed method do not visually resemble their original counterparts. Nevertheless, the original color image can be efficiently verified with statistical nonlinear correlations. Our experimental results also show that different interpolation algorithms applied to Bayer images result in different verification effects for multispectral RGB color images.

Feasibility of intra-operative image guidance in burn excision surgery with multispectral imaging and deep learning.

BACKGROUND: Exposing a healthy wound bed for skin grafting is an important step during burn surgery to ensure graft take and maintain good functional outcomes. Currently, the removal of non-viable tissue in the burn wound bed during excision is determined by expert clinician judgment. Using a porcine model of tangential burn excision, we investigated the effectiveness of an intraoperative multispectral imaging device combined with artificial intelligence to aid clinician judgment for the excision of non-viable tissue. METHODS: Multispectral imaging data was obtained from serial tangential excisions of thermal burn injuries and used to train a deep learning algorithm to identify the presence and location of non-viable tissue in the wound bed. Following algorithm development, we studied the ability of two surgeons to estimate wound bed viability, both unaided and aided by the imaging device. RESULTS: The deep learning algorithm was 87% accurate in identifying the viability of a burn wound bed. When paired with the surgeons, this device significantly improved their abilities to determine the viability of the wound bed by 25% (p = 0.03). Each time a surgeon changed their decision after seeing the AI model output, it was always a change from an incorrect decision to excise more tissue to a correct decision to stop excision. CONCLUSION: This study provides insight into the feasibility of image-guided burn excision, its effect on surgeon decision making, and suggests further investigation of a real-time imaging system for burn surgery could reduce over-excision of burn wounds.

Automated quantitative analysis of 3D morphology and mean corpuscular hemoglobin in human red blood cells stored in different periods.

Quantitative phase (QP) images of red blood cells (RBCs), which are obtained by off-axis digital holographic microscopy, can provide quantitative information about three-dimensional (3D) morphology of human RBCs and the characteristic properties such as mean corpuscular hemoglobin (MCH) and MCH surface density (MCHSD). In this paper, we investigate modifications of the 3D morphology and MCH in RBCs induced by the period of storage time for the purpose of classification of RBCs with different periods of storage by using off-axis digital holographic microscopy. The classification of RBCs based on the duration of storage is highly relevant because a long storage of blood before transfusion may alter the functionality of RBCs and, therefore, cause complications in patients. To analyze any changes in the 3D morphology and MCH of RBCs due to storage, we use data sets from RBC samples stored for 8, 13, 16, 23, 27, 30, 34, 37, 40, 47, and 57 days, respectively. The data sets consist of more than 3,300 blood cells in eleven classes, with more than 300 blood cells per class. The classes indicate the storage period of RBCs and are listed in chronological order. Using the RBCs donated by healthy persons, the off-axis digital holographic microscopy reconstructs several quantitative phase images of RBC samples stored for eleven different periods. We employ marker-controlled watershed transform to remove the background in the RBC quantitative phase images obtained by the off-axis digital holographic microscopy. More than 300 single RBCs are extracted from the segmented quantitative phase images for each class. Such a large number of RBC samples enable us to obtain statistical distributions of the characteristic properties of RBCs after a specific period of storage. Experimental results show that the 3D morphology of the RBCs, in contrast to MCH, is essentially related to the aging of the RBCs.

Clinical Investigation of a Rapid Non-invasive Multispectral Imaging Device Utilizing an Artificial Intelligence Algorithm for Improved Burn Assessment.

Currently, the incorrect judgment of burn depth remains common even among experienced surgeons. Contributing to this problem are change in burn appearance throughout the first week requiring periodic evaluation until a confident diagnosis can be made. To overcome these issues, we investigated the feasibility of an artificial intelligence algorithm trained with multispectral images of burn injuries to predict burn depth rapidly and accurately, including burns of indeterminate depth. In a feasibility study, 406 multispectral images of burns were collected within 72 hours of injury and then serially for up to 7 days. Simultaneously, the subject's clinician indicated whether the burn was of indeterminate depth. The final depth of burned regions within images were agreed upon by a panel of burn practitioners using biopsies and 21-day healing assessments as reference standards. We compared three convolutional neural network architectures and an ensemble in their capability to automatically highlight areas of nonhealing burn regions within images. The top algorithm was the ensemble with 81% sensitivity, 100% specificity, and 97% positive predictive value (PPV). Its sensitivity and PPV were found to increase in a sigmoid shape during the first week postburn, with the inflection point at day 2.5. Additionally, when burns were labeled as indeterminate, the algorithm's sensitivity, specificity, PPV, and negative predictive value were: 70%, 100%, 97%, and 100%. These results suggest multispectral imaging combined with artificial intelligence is feasible for detecting nonhealing burn tissue and could play an important role in aiding the earlier diagnosis of indeterminate burns.

Automated statistical quantification of three-dimensional morphology and mean corpuscular hemoglobin of multiple red blood cells.

In this paper, we present an automated approach to quantify information about three-dimensional (3D) morphology, hemoglobin content and density of mature red blood cells (RBCs) using off-axis digital holographic microscopy (DHM) and statistical algorithms. The digital hologram of RBCs is recorded by a CCD camera using an off-axis interferometry setup and quantitative phase images of RBCs are obtained by a numerical reconstruction algorithm. In order to remove unnecessary parts and obtain clear targets in the reconstructed phase image with many RBCs, the marker-controlled watershed segmentation algorithm is applied to the phase image. Each RBC in the segmented phase image is three-dimensionally investigated. Characteristic properties such as projected cell surface, average phase, sphericity coefficient, mean corpuscular hemoglobin (MCH) and MCH surface density of each RBC is quantitatively measured. We experimentally demonstrate that joint statistical distributions of the characteristic parameters of RBCs can be obtained by our algorithm and efficiently used as a feature pattern to discriminate between RBC populations that differ in shape and hemoglobin content. Our study opens the possibility of automated RBC quantitative analysis suitable for the rapid classification of a large number of RBCs from an individual blood specimen, which is a fundamental step to develop a diagnostic approach based on DHM.

Automatic calculation of tree diameter from stereoscopic image pairs using digital image processing.

Automatic operations play an important role in societies by saving time and improving efficiency. In this paper, we apply the digital image processing method to the field of lumbering to automatically calculate tree diameters in order to reduce culler work and enable a third party to verify tree diameters. To calculate the cross-sectional diameter of a tree, the image was first segmented by the marker-controlled watershed transform algorithm based on the hue saturation intensity (HSI) color model. Then, the tree diameter was obtained by measuring the area of every isolated region in the segmented image. Finally, the true diameter was calculated by multiplying the diameter computed in the image and the scale, which was derived from the baseline and disparity of correspondence points from stereoscopic image pairs captured by rectified configuration cameras.

High-throughput label-free cell detection and counting from diffraction patterns with deep fully convolutional neural networks.

SIGNIFICANCE: Digital holographic microscopy (DHM) is a promising technique for the study of semitransparent biological specimen such as red blood cells (RBCs). It is important and meaningful to detect and count biological cells at the single cell level in biomedical images for biomarker discovery and disease diagnostics. However, the biological cell analysis based on phase information of images is inefficient due to the complexity of numerical phase reconstruction algorithm applied to raw hologram images. New cell study methods based on diffraction pattern directly are desirable. AIM: Deep fully convolutional networks (FCNs) were developed on raw hologram images directly for high-throughput label-free cell detection and counting to assist the biological cell analysis in the future. APPROACH: The raw diffraction patterns of RBCs were recorded by use of DHM. Ground-truth mask images were labeled based on phase images reconstructed from RBC holograms using numerical reconstruction algorithm. A deep FCN, which is UNet, was trained on the diffraction pattern images to achieve the label-free cell detection and counting. RESULTS: The implemented deep FCNs provide a promising way to high-throughput and label-free counting of RBCs with a counting accuracy of 99% at a throughput rate of greater than 288 cells per second and 200 µm × 200 µm field of view at the single cell level. Compared to convolutional neural networks, the FCNs can get much better results in terms of accuracy and throughput rate. CONCLUSIONS: High-throughput label-free cell detection and counting were successfully achieved from diffraction patterns with deep FCNs. It is a promising approach for biological specimen analysis based on raw hologram directly.

Automated three-dimensional microbial sensing and recognition using digital holography and statistical sampling.

We overview an approach to providing automated three-dimensional (3D) sensing and recognition of biological micro/nanoorganisms integrating Gabor digital holographic microscopy and statistical sampling methods. For 3D data acquisition of biological specimens, a coherent beam propagates through the specimen and its transversely and longitudinally magnified diffraction pattern observed by the microscope objective is optically recorded with an image sensor array interfaced with a computer. 3D visualization of the biological specimen from the magnified diffraction pattern is accomplished by using the computational Fresnel propagation algorithm. For 3D recognition of the biological specimen, a watershed image segmentation algorithm is applied to automatically remove the unnecessary background parts in the reconstructed holographic image. Statistical estimation and inference algorithms are developed to the automatically segmented holographic image. Overviews of preliminary experimental results illustrate how the holographic image reconstructed from the Gabor digital hologram of biological specimen contains important information for microbial recognition.

ConvPath: A software tool for lung adenocarcinoma digital pathological image analysis aided by a convolutional neural network.

BACKGROUND: The spatial distributions of different types of cells could reveal a cancer cell's growth pattern, its relationships with the tumor microenvironment and the immune response of the body, all of which represent key "hallmarks of cancer". However, the process by which pathologists manually recognize and localize all the cells in pathology slides is extremely labor intensive and error prone. METHODS: In this study, we developed an automated cell type classification pipeline, ConvPath, which includes nuclei segmentation, convolutional neural network-based tumor cell, stromal cell, and lymphocyte classification, and extraction of tumor microenvironment-related features for lung cancer pathology images. To facilitate users in leveraging this pipeline for their research, all source scripts for ConvPath software are available at https://qbrc.swmed.edu/projects/cnn/. FINDINGS: The overall classification accuracy was 92.9% and 90.1% in training and independent testing datasets, respectively. By identifying cells and classifying cell types, this pipeline can convert a pathology image into a "spatial map" of tumor, stromal and lymphocyte cells. From this spatial map, we can extract features that characterize the tumor micro-environment. Based on these features, we developed an image feature-based prognostic model and validated the model in two independent cohorts. The predicted risk group serves as an independent prognostic factor, after adjusting for clinical variables that include age, gender, smoking status, and stage. INTERPRETATION: The analysis pipeline developed in this study could convert the pathology image into a "spatial map" of tumor cells, stromal cells and lymphocytes. This could greatly facilitate and empower comprehensive analysis of the spatial organization of cells, as well as their roles in tumor progression and metastasis.

Automated red blood cells extraction from holographic images using fully convolutional neural networks.

In this paper, we present two models for automatically extracting red blood cells (RBCs) from RBCs holographic images based on a deep learning fully convolutional neural network (FCN) algorithm. The first model, called FCN-1, only uses the FCN algorithm to carry out RBCs prediction, whereas the second model, called FCN-2, combines the FCN approach with the marker-controlled watershed transform segmentation scheme to achieve RBCs extraction. Both models achieve good segmentation accuracy. In addition, the second model has much better performance in terms of cell separation than traditional segmentation methods. In the proposed methods, the RBCs phase images are first numerically reconstructed from RBCs holograms recorded with off-axis digital holographic microscopy. Then, some RBCs phase images are manually segmented and used as training data to fine-tune the FCN. Finally, each pixel in new input RBCs phase images is predicted into either foreground or background using the trained FCN models. The RBCs prediction result from the first model is the final segmentation result, whereas the result from the second model is used as the internal markers of the marker-controlled transform algorithm for further segmentation. Experimental results show that the given schemes can automatically extract RBCs from RBCs phase images and much better RBCs separation results are obtained when the FCN technique is combined with the marker-controlled watershed segmentation algorithm.

Automatic extraction of cell nuclei from H&E-stained histopathological images.

Extraction of cell nuclei from hematoxylin and eosin (H&E)-stained histopathological images is an essential preprocessing step in computerized image analysis for disease detection, diagnosis, and prognosis. We present an automated cell nuclei segmentation approach that works with H&E-stained images. A color deconvolution algorithm was first applied to the image to get the hematoxylin channel. Using a morphological operation and thresholding technique on the hematoxylin channel image, candidate target nuclei and background regions were detected, which were then used as markers for a marker-controlled watershed transform segmentation algorithm. Moreover, postprocessing was conducted to split the touching nuclei. For each segmented region from the previous steps, the regional maximum value positions were identified as potential nuclei centers. These maximum values were further grouped into [Formula: see text]-clusters, and the locations within each cluster were connected with the minimum spanning tree technique. Then, these connected positions were utilized as new markers for a watershed segmentation approach. The final number of nuclei at each region was determined by minimizing an objective function that iterated all of the possible [Formula: see text]-values. The proposed method was applied to the pathological images of the tumor tissues from The Cancer Genome Atlas study. Experimental results show that the proposed method can lead to promising results in terms of segmentation accuracy and separation of touching nuclei.