Feasibility of intra-operative image guidance in burn excision surgery with multispectral imaging and deep learning.

BACKGROUND: Exposing a healthy wound bed for skin grafting is an important step during burn surgery to ensure graft take and maintain good functional outcomes. Currently, the removal of non-viable tissue in the burn wound bed during excision is determined by expert clinician judgment. Using a porcine model of tangential burn excision, we investigated the effectiveness of an intraoperative multispectral imaging device combined with artificial intelligence to aid clinician judgment for the excision of non-viable tissue. METHODS: Multispectral imaging data was obtained from serial tangential excisions of thermal burn injuries and used to train a deep learning algorithm to identify the presence and location of non-viable tissue in the wound bed. Following algorithm development, we studied the ability of two surgeons to estimate wound bed viability, both unaided and aided by the imaging device. RESULTS: The deep learning algorithm was 87% accurate in identifying the viability of a burn wound bed. When paired with the surgeons, this device significantly improved their abilities to determine the viability of the wound bed by 25% (p = 0.03). Each time a surgeon changed their decision after seeing the AI model output, it was always a change from an incorrect decision to excise more tissue to a correct decision to stop excision. CONCLUSION: This study provides insight into the feasibility of image-guided burn excision, its effect on surgeon decision making, and suggests further investigation of a real-time imaging system for burn surgery could reduce over-excision of burn wounds.

Clinical Investigation of a Rapid Non-invasive Multispectral Imaging Device Utilizing an Artificial Intelligence Algorithm for Improved Burn Assessment.

Currently, the incorrect judgment of burn depth remains common even among experienced surgeons. Contributing to this problem are change in burn appearance throughout the first week requiring periodic evaluation until a confident diagnosis can be made. To overcome these issues, we investigated the feasibility of an artificial intelligence algorithm trained with multispectral images of burn injuries to predict burn depth rapidly and accurately, including burns of indeterminate depth. In a feasibility study, 406 multispectral images of burns were collected within 72 hours of injury and then serially for up to 7 days. Simultaneously, the subject's clinician indicated whether the burn was of indeterminate depth. The final depth of burned regions within images were agreed upon by a panel of burn practitioners using biopsies and 21-day healing assessments as reference standards. We compared three convolutional neural network architectures and an ensemble in their capability to automatically highlight areas of nonhealing burn regions within images. The top algorithm was the ensemble with 81% sensitivity, 100% specificity, and 97% positive predictive value (PPV). Its sensitivity and PPV were found to increase in a sigmoid shape during the first week postburn, with the inflection point at day 2.5. Additionally, when burns were labeled as indeterminate, the algorithm's sensitivity, specificity, PPV, and negative predictive value were: 70%, 100%, 97%, and 100%. These results suggest multispectral imaging combined with artificial intelligence is feasible for detecting nonhealing burn tissue and could play an important role in aiding the earlier diagnosis of indeterminate burns.

Machine learning analysis of multispectral imaging and clinical risk factors to predict amputation wound healing.

OBJECTIVE: Prediction of amputation wound healing is challenging due to the multifactorial nature of critical limb ischemia and lack of objective assessment tools. Up to one-third of amputations require revision to a more proximal level within 1 year. We tested a novel wound imaging system to predict amputation wound healing at initial evaluation. METHODS: Patients planned to undergo amputation due to critical limb ischemia were prospectively enrolled. Clinicians evaluated the patients in traditional fashion, and all clinical decisions for amputation level were determined by the clinician's judgement. Multispectral images of the lower extremity were obtained preoperatively using a novel wound imaging system. Clinicians were blinded to the machine analysis. A standardized wound healing assessment was performed on postoperative day 30 by physical exam to determine whether the amputation site achieved complete healing. If operative revision or higher level of amputation was required, this was undertaken based solely upon the provider's clinical judgement. A machine learning algorithm combining the multispectral imaging data with patient clinical risk factors was trained and tested using cross-validation to measure the wound imaging system's accuracy of predicting amputation wound healing. RESULTS: A total of 22 patients undergoing 25 amputations (10 toe, five transmetatarsal, eight below-knee, and two above-knee amputations) were enrolled. Eleven amputations (44%) were non-healing after 30 days. The machine learning algorithm had 91% sensitivity and 86% specificity for prediction of non-healing amputation sites (area under curve, 0.89). CONCLUSIONS: This pilot study suggests that a machine learning algorithm combining multispectral wound imaging with patient clinical risk factors may improve prediction of amputation wound healing and therefore decrease the need for reoperation and incidence of delayed healing. We propose that this, in turn, may offer significant cost savings to the patient and health system in addition to decreasing length of stay for patients.

High-throughput label-free cell detection and counting from diffraction patterns with deep fully convolutional neural networks.

SIGNIFICANCE: Digital holographic microscopy (DHM) is a promising technique for the study of semitransparent biological specimen such as red blood cells (RBCs). It is important and meaningful to detect and count biological cells at the single cell level in biomedical images for biomarker discovery and disease diagnostics. However, the biological cell analysis based on phase information of images is inefficient due to the complexity of numerical phase reconstruction algorithm applied to raw hologram images. New cell study methods based on diffraction pattern directly are desirable. AIM: Deep fully convolutional networks (FCNs) were developed on raw hologram images directly for high-throughput label-free cell detection and counting to assist the biological cell analysis in the future. APPROACH: The raw diffraction patterns of RBCs were recorded by use of DHM. Ground-truth mask images were labeled based on phase images reconstructed from RBC holograms using numerical reconstruction algorithm. A deep FCN, which is UNet, was trained on the diffraction pattern images to achieve the label-free cell detection and counting. RESULTS: The implemented deep FCNs provide a promising way to high-throughput and label-free counting of RBCs with a counting accuracy of 99% at a throughput rate of greater than 288 cells per second and 200 µm × 200 µm field of view at the single cell level. Compared to convolutional neural networks, the FCNs can get much better results in terms of accuracy and throughput rate. CONCLUSIONS: High-throughput label-free cell detection and counting were successfully achieved from diffraction patterns with deep FCNs. It is a promising approach for biological specimen analysis based on raw hologram directly.

ConvPath: A software tool for lung adenocarcinoma digital pathological image analysis aided by a convolutional neural network.

BACKGROUND: The spatial distributions of different types of cells could reveal a cancer cell's growth pattern, its relationships with the tumor microenvironment and the immune response of the body, all of which represent key "hallmarks of cancer". However, the process by which pathologists manually recognize and localize all the cells in pathology slides is extremely labor intensive and error prone. METHODS: In this study, we developed an automated cell type classification pipeline, ConvPath, which includes nuclei segmentation, convolutional neural network-based tumor cell, stromal cell, and lymphocyte classification, and extraction of tumor microenvironment-related features for lung cancer pathology images. To facilitate users in leveraging this pipeline for their research, all source scripts for ConvPath software are available at https://qbrc.swmed.edu/projects/cnn/. FINDINGS: The overall classification accuracy was 92.9% and 90.1% in training and independent testing datasets, respectively. By identifying cells and classifying cell types, this pipeline can convert a pathology image into a "spatial map" of tumor, stromal and lymphocyte cells. From this spatial map, we can extract features that characterize the tumor micro-environment. Based on these features, we developed an image feature-based prognostic model and validated the model in two independent cohorts. The predicted risk group serves as an independent prognostic factor, after adjusting for clinical variables that include age, gender, smoking status, and stage. INTERPRETATION: The analysis pipeline developed in this study could convert the pathology image into a "spatial map" of tumor cells, stromal cells and lymphocytes. This could greatly facilitate and empower comprehensive analysis of the spatial organization of cells, as well as their roles in tumor progression and metastasis.

A Bayesian hidden Potts mixture model for analyzing lung cancer pathology images.

Digital pathology imaging of tumor tissues, which captures histological details in high resolution, is fast becoming a routine clinical procedure. Recent developments in deep-learning methods have enabled the identification, characterization, and classification of individual cells from pathology images analysis at a large scale. This creates new opportunities to study the spatial patterns of and interactions among different types of cells. Reliable statistical approaches to modeling such spatial patterns and interactions can provide insight into tumor progression and shed light on the biological mechanisms of cancer. In this article, we consider the problem of modeling a pathology image with irregular locations of three different types of cells: lymphocyte, stromal, and tumor cells. We propose a novel Bayesian hierarchical model, which incorporates a hidden Potts model to project the irregularly distributed cells to a square lattice and a Markov random field prior model to identify regions in a heterogeneous pathology image. The model allows us to quantify the interactions between different types of cells, some of which are clinically meaningful. We use Markov chain Monte Carlo sampling techniques, combined with a double Metropolis-Hastings algorithm, in order to simulate samples approximately from a distribution with an intractable normalizing constant. The proposed model was applied to the pathology images of $205$ lung cancer patients from the National Lung Screening trial, and the results show that the interaction strength between tumor and stromal cells predicts patient prognosis (P = $0.005$). This statistical methodology provides a new perspective for understanding the role of cell-cell interactions in cancer progression.

Microvessel prediction in H&E Stained Pathology Images using fully convolutional neural networks.

BACKGROUND: Pathological angiogenesis has been identified in many malignancies as a potential prognostic factor and target for therapy. In most cases, angiogenic analysis is based on the measurement of microvessel density (MVD) detected by immunostaining of CD31 or CD34. However, most retrievable public data is generally composed of Hematoxylin and Eosin (H&E)-stained pathology images, for which is difficult to get the corresponding immunohistochemistry images. The role of microvessels in H&E stained images has not been widely studied due to their complexity and heterogeneity. Furthermore, identifying microvessels manually for study is a labor-intensive task for pathologists, with high inter- and intra-observer variation. Therefore, it is important to develop automated microvessel-detection algorithms in H&E stained pathology images for clinical association analysis. RESULTS: In this paper, we propose a microvessel prediction method using fully convolutional neural networks. The feasibility of our proposed algorithm is demonstrated through experimental results on H&E stained images. Furthermore, the identified microvessel features were significantly associated with the patient clinical outcomes. CONCLUSIONS: This is the first study to develop an algorithm for automated microvessel detection in H&E stained pathology images.

Three-dimensional image authentication scheme using sparse phase information in double random phase encoded integral imaging.

In recent years, many studies have focused on authentication of two-dimensional (2D) images using double random phase encryption techniques. However, there has been little research on three-dimensional (3D) imaging systems, such as integral imaging, for 3D image authentication. We propose a 3D image authentication scheme based on a double random phase integral imaging method. All of the 2D elemental images captured through integral imaging are encrypted with a double random phase encoding algorithm and only partial phase information is reserved. All the amplitude and other miscellaneous phase information in the encrypted elemental images is discarded. Nevertheless, we demonstrate that 3D images from integral imaging can be authenticated at different depths using a nonlinear correlation method. The proposed 3D image authentication algorithm can provide enhanced information security because the decrypted 2D elemental images from the sparse phase cannot be easily observed by the naked eye. Additionally, using sparse phase images without any amplitude information can greatly reduce data storage costs and aid in image compression and data transmission.

Automated red blood cells extraction from holographic images using fully convolutional neural networks.

In this paper, we present two models for automatically extracting red blood cells (RBCs) from RBCs holographic images based on a deep learning fully convolutional neural network (FCN) algorithm. The first model, called FCN-1, only uses the FCN algorithm to carry out RBCs prediction, whereas the second model, called FCN-2, combines the FCN approach with the marker-controlled watershed transform segmentation scheme to achieve RBCs extraction. Both models achieve good segmentation accuracy. In addition, the second model has much better performance in terms of cell separation than traditional segmentation methods. In the proposed methods, the RBCs phase images are first numerically reconstructed from RBCs holograms recorded with off-axis digital holographic microscopy. Then, some RBCs phase images are manually segmented and used as training data to fine-tune the FCN. Finally, each pixel in new input RBCs phase images is predicted into either foreground or background using the trained FCN models. The RBCs prediction result from the first model is the final segmentation result, whereas the result from the second model is used as the internal markers of the marker-controlled transform algorithm for further segmentation. Experimental results show that the given schemes can automatically extract RBCs from RBCs phase images and much better RBCs separation results are obtained when the FCN technique is combined with the marker-controlled watershed segmentation algorithm.

Lung Cancer Pathological Image Analysis Using a Hidden Potts Model.

Nowadays, many biological data are acquired via images. In this article, we study the pathological images scanned from 205 patients with lung cancer with the goal to find out the relationship between the survival time and the spatial distribution of different types of cells, including lymphocyte, stroma, and tumor cells. Toward this goal, we model the spatial distribution of different types of cells using a modified Potts model for which the parameters represent interactions between different types of cells and estimate the parameters of the Potts model using the double Metropolis-Hastings algorithm. The double Metropolis-Hastings algorithm allows us to simulate samples approximately from a distribution with an intractable normalizing constant. Our numerical results indicate that the spatial interaction between the lymphocyte and tumor cells is significantly associated with the patient's survival time, and it can be used together with the cell count information to predict the survival of the patients.

Automatic extraction of cell nuclei from H&E-stained histopathological images.

Extraction of cell nuclei from hematoxylin and eosin (H&E)-stained histopathological images is an essential preprocessing step in computerized image analysis for disease detection, diagnosis, and prognosis. We present an automated cell nuclei segmentation approach that works with H&E-stained images. A color deconvolution algorithm was first applied to the image to get the hematoxylin channel. Using a morphological operation and thresholding technique on the hematoxylin channel image, candidate target nuclei and background regions were detected, which were then used as markers for a marker-controlled watershed transform segmentation algorithm. Moreover, postprocessing was conducted to split the touching nuclei. For each segmented region from the previous steps, the regional maximum value positions were identified as potential nuclei centers. These maximum values were further grouped into [Formula: see text]-clusters, and the locations within each cluster were connected with the minimum spanning tree technique. Then, these connected positions were utilized as new markers for a watershed segmentation approach. The final number of nuclei at each region was determined by minimizing an objective function that iterated all of the possible [Formula: see text]-values. The proposed method was applied to the pathological images of the tumor tissues from The Cancer Genome Atlas study. Experimental results show that the proposed method can lead to promising results in terms of segmentation accuracy and separation of touching nuclei.

Efficient asymmetric image authentication schemes based on photon counting-double random phase encoding and RSA algorithms.

Recently, double random phase encoding (DRPE) has been integrated with the photon counting (PC) imaging technique for the purpose of secure image authentication. In this scheme, the same key should be securely distributed and shared between the sender and receiver, but this is one of the most vexing problems of symmetric cryptosystems. In this study, we propose an efficient asymmetric image authentication scheme by combining the PC-DRPE and RSA algorithms, which solves key management and distribution problems. The retrieved image from the proposed authentication method contains photon-limited encrypted data obtained by means of PC-DRPE. Therefore, the original image can be protected while the retrieved image can be efficiently verified using a statistical nonlinear correlation approach. Experimental results demonstrate the feasibility of our proposed asymmetric image authentication method.

Cell morphology-based classification of red blood cells using holographic imaging informatics.

We present methods that automatically select a linear or nonlinear classifier for red blood cell (RBC) classification by analyzing the equality of the covariance matrices in Gabor-filtered holographic images. First, the phase images of the RBCs are numerically reconstructed from their holograms, which are recorded using off-axis digital holographic microscopy (DHM). Second, each RBC is segmented using a marker-controlled watershed transform algorithm and the inner part of the RBC is identified and analyzed. Third, the Gabor wavelet transform is applied to the segmented cells to extract a series of features, which then undergo a multivariate statistical test to evaluate the equality of the covariance matrices of the different shapes of the RBCs using selected features. When these covariance matrices are not equal, a nonlinear classification scheme based on quadratic functions is applied; otherwise, a linear classification is applied. We used the stomatocyte, discocyte, and echinocyte RBC for classifier training and testing. Simulation results demonstrated that 10 of the 14 RBC features are useful in RBC classification. Experimental results also revealed that the covariance matrices of the three main RBC groups are not equal and that a nonlinear classification method has a much lower misclassification rate. The proposed automated RBC classification method has the potential for use in drug testing and the diagnosis of RBC-related diseases.

Automated tracking of temporal displacements of a red blood cell obtained by time-lapse digital holographic microscopy.

Red blood cell (RBC) phase images that are numerically reconstructed by digital holographic microscopy (DHM) can describe the cell structure and dynamics information beneficial for a quantitative analysis of RBCs. However, RBCs investigated with time-lapse DHM undergo temporal displacements when their membranes are loosely attached to the substrate during sedimentation on a glass surface or due to the microscope drift. Therefore, we need to develop a tracking algorithm to localize the same RBC among RBC image sequences and dynamically monitor its biophysical cell parameters; this information is helpful for studies on RBC-related diseases and drug tests. Here, we propose a method, which is a combination of the mean-shift algorithm and Kalman filter, to track a single RBC and demonstrate that the optical path length of the single RBC can be continually extracted from the tracked RBC. The Kalman filter is utilized to predict the target RBC position in the next frame. Then, the mean-shift algorithm starts execution from the predicted location, and a robust kernel, which is adaptive to changes in the RBC scale, shape, and direction, is designed to improve the accuracy of the tracking. Finally, the tracked RBC is segmented and parameters such as the RBC location are extracted to update the Kalman filter and the kernel function for mean-shift tracking; the characteristics of the target RBC are dynamically observed. Experimental results show the feasibility of the proposed algorithm.

Automated multi-parameter measurement of cardiomyocytes dynamics with digital holographic microscopy.

Compounds tested during drug development may have adverse effects on the heart; therefore all new chemical entities have to undergo extensive preclinical assessment for cardiac liability. Conventional intensity-based imaging techniques are not robust enough to provide detailed information for cell structure and the captured images result in low-contrast, especially to cell with semi-transparent or transparent feature, which would affect the cell analysis. In this paper we show, for the first time, that digital holographic microscopy (DHM) integrated with information processing algorithms automatically provide dynamic quantitative phase profiles of beating cardiomyocytes. We experimentally demonstrate that relevant parameters of cardiomyocytes can be obtained by our automated algorithm based on DHM phase signal analysis and used to characterize the physiological state of resting cardiomyocytes. Our study opens the possibility of automated quantitative analysis of cardiomyocyte dynamics suitable for further drug safety testing and compounds selection as a new paradigm in drug toxicity screens.

Three-dimensional counting of morphologically normal human red blood cells via digital holographic microscopy.

Counting morphologically normal cells in human red blood cells (RBCs) is extremely beneficial in the health care field. We propose a three-dimensional (3-D) classification method of automatically determining the morphologically normal RBCs in the phase image of multiple human RBCs that are obtained by off-axis digital holographic microscopy (DHM). The RBC holograms are first recorded by DHM, and then the phase images of multiple RBCs are reconstructed by a computational numerical algorithm. To design the classifier, the three typical RBC shapes, which are stomatocyte, discocyte, and echinocyte, are used for training and testing. Nonmain or abnormal RBC shapes different from the three normal shapes are defined as the fourth category. Ten features, including projected surface area, average phase value, mean corpuscular hemoglobin, perimeter, mean corpuscular hemoglobin surface density, circularity, mean phase of center part, sphericity coefficient, elongation, and pallor, are extracted from each RBC after segmenting the reconstructed phase images by using a watershed transform algorithm. Moreover, four additional properties, such as projected surface area, perimeter, average phase value, and elongation, are measured from the inner part of each cell, which can give significant information beyond the previous 10 features for the separation of the RBC groups; these are verified in the experiment by the statistical method of Hotelling's T-quare test. We also apply the principal component analysis algorithm to reduce the dimension number of variables and establish the Gaussian mixture densities using the projected data with the first eight principal components. Consequently, the Gaussian mixtures are used to design the discriminant functions based on Bayesian decision theory. To improve the performance of the Bayes classifier and the accuracy of estimation of its error rate, the leaving-one-out technique is applied. Experimental results show that the proposed method can yield good results for calculating the percentage of each typical normal RBC shape in a reconstructed phase image of multiple RBCs that will be favorable to the analysis of RBC-related diseases. In addition, we show that the discrimination performance for the counting of normal shapes of RBCs can be improved by using 3-D features of an RBC.

Avalanche and bit independence characteristics of double random phase encoding in the Fourier and Fresnel domains.

In this work, we evaluate the avalanche effect and bit independence properties of the double random phase encoding (DRPE) algorithm in the Fourier and Fresnel domains. Experimental results show that DRPE has excellent bit independence characteristics in both the Fourier and Fresnel domains. However, DRPE achieves better avalanche effect results in the Fresnel domain than in the Fourier domain. DRPE gives especially poor avalanche effect results in the Fourier domain when only one bit is changed in the plaintext or in the encryption key. Despite this, DRPE shows satisfactory avalanche effect results in the Fresnel domain when any other number of bits changes in the plaintext or in the encryption key. To the best of our knowledge, this is the first report on the avalanche effect and bit independence behaviors of optical encryption approaches for bit units.

Simultaneous reconstruction of multiple depth images without off-focus points in integral imaging using a graphics processing unit.

The reconstruction of multiple depth images with a ray back-propagation algorithm in three-dimensional (3D) computational integral imaging is computationally burdensome. Further, a reconstructed depth image consists of a focus and an off-focus area. Focus areas are 3D points on the surface of an object that are located at the reconstructed depth, while off-focus areas include 3D points in free-space that do not belong to any object surface in 3D space. Generally, without being removed, the presence of an off-focus area would adversely affect the high-level analysis of a 3D object, including its classification, recognition, and tracking. Here, we use a graphics processing unit (GPU) that supports parallel processing with multiple processors to simultaneously reconstruct multiple depth images using a lookup table containing the shifted values along the x and y directions for each elemental image in a given depth range. Moreover, each 3D point on a depth image can be measured by analyzing its statistical variance with its corresponding samples, which are captured by the two-dimensional (2D) elemental images. These statistical variances can be used to classify depth image pixels as either focus or off-focus points. At this stage, the measurement of focus and off-focus points in multiple depth images is also implemented in parallel on a GPU. Our proposed method is conducted based on the assumption that there is no occlusion of the 3D object during the capture stage of the integral imaging process. Experimental results have demonstrated that this method is capable of removing off-focus points in the reconstructed depth image. The results also showed that using a GPU to remove the off-focus points could greatly improve the overall computational speed compared with using a CPU.

A multispectral photon-counting double random phase encoding scheme for image authentication.

In this paper, we propose a new method for color image-based authentication that combines multispectral photon-counting imaging (MPCI) and double random phase encoding (DRPE) schemes. The sparsely distributed information from MPCI and the stationary white noise signal from DRPE make intruder attacks difficult. In this authentication method, the original multispectral RGB color image is down-sampled into a Bayer image. The three types of color samples (red, green and blue color) in the Bayer image are encrypted with DRPE and the amplitude part of the resulting image is photon counted. The corresponding phase information that has nonzero amplitude after photon counting is then kept for decryption. Experimental results show that the retrieved images from the proposed method do not visually resemble their original counterparts. Nevertheless, the original color image can be efficiently verified with statistical nonlinear correlations. Our experimental results also show that different interpolation algorithms applied to Bayer images result in different verification effects for multispectral RGB color images.

Extraction of target specimens from bioholographic images using interactive graph cuts.

It is necessary to extract target specimens from bioholographic images for high-level analysis such as object identification, recognition, and tracking with the advent of application of digital holographic microscopy to transparent or semi-transparent biological specimens. We present an interactive graph cuts approach to segment the needed target specimens in the reconstructed bioholographic images. This method combines both regional and boundary information and is robust to extract targets with weak boundaries. Moreover, this technique can achieve globally optimal results while minimizing an energy function. We provide a convenient user interface, which can easily differentiate the foreground/background for various types of holographic images, as well as a dynamically modified coefficient, which specifies the importance of the regional and boundary information. The extracted results from our scheme have been compared with those from an advanced level-set-based segmentation method using an unbiased comparison algorithm. Experimental results show that this interactive graph cut technique can not only extract different kinds of target specimens in bioholographic images, but also yield good results when there are multiple similar objects in the holographic image or when the object boundaries are very weak.