Improved accuracy in colorectal cancer tissue decomposition through refinement of established deep learning solutions.

Hematoxylin and eosin-stained biopsy slides are regularly available for colorectal cancer patients. These slides are often not used to define objective biomarkers for patient stratification and treatment selection. Standard biomarkers often pertain to costly and slow genetic tests. However, recent work has shown that relevant biomarkers can be extracted from these images using convolutional neural networks (CNNs). The CNN-based biomarkers predicted colorectal cancer patient outcomes comparably to gold standards. Extracting CNN-biomarkers is fast, automatic, and of minimal cost. CNN-based biomarkers rely on the ability of CNNs to recognize distinct tissue types from microscope whole slide images. The quality of these biomarkers (coined 'Deep Stroma') depends on the accuracy of CNNs in decomposing all relevant tissue classes. Improving tissue decomposition accuracy is essential for improving the prognostic potential of CNN-biomarkers. In this study, we implemented a novel training strategy to refine an established CNN model, which then surpassed all previous solutions . We obtained a 95.6% average accuracy in the external test set and 99.5% in the internal test set. Our approach reduced errors in biomarker-relevant classes, such as Lymphocytes, and was the first to include interpretability methods. These methods were used to better apprehend our model's limitations and capabilities.

Domain-specific transfer learning in the automated scoring of tumor-stroma ratio from histopathological images of colorectal cancer.

Tumor-stroma ratio (TSR) is a prognostic factor for many types of solid tumors. In this study, we propose a method for automated estimation of TSR from histopathological images of colorectal cancer. The method is based on convolutional neural networks which were trained to classify colorectal cancer tissue in hematoxylin-eosin stained samples into three classes: stroma, tumor and other. The models were trained using a data set that consists of 1343 whole slide images. Three different training setups were applied with a transfer learning approach using domain-specific data i.e. an external colorectal cancer histopathological data set. The three most accurate models were chosen as a classifier, TSR values were predicted and the results were compared to a visual TSR estimation made by a pathologist. The results suggest that classification accuracy does not improve when domain-specific data are used in the pre-training of the convolutional neural network models in the task at hand. Classification accuracy for stroma, tumor and other reached 96.1% on an independent test set. Among the three classes the best model gained the highest accuracy (99.3%) for class tumor. When TSR was predicted with the best model, the correlation between the predicted values and values estimated by an experienced pathologist was 0.57. Further research is needed to study associations between computationally predicted TSR values and other clinicopathological factors of colorectal cancer and the overall survival of the patients.

DeepFake knee osteoarthritis X-rays from generative adversarial neural networks deceive medical experts and offer augmentation potential to automatic classification.

Recent developments in deep learning have impacted medical science. However, new privacy issues and regulatory frameworks have hindered medical data sharing and collection. Deep learning is a very data-intensive process for which such regulatory limitations limit the potential for new breakthroughs and collaborations. However, generating medically accurate synthetic data can alleviate privacy issues and potentially augment deep learning pipelines. This study presents generative adversarial neural networks capable of generating realistic images of knee joint X-rays with varying osteoarthritis severity. We offer 320,000 synthetic (DeepFake) X-ray images from training with 5,556 real images. We validated our models regarding medical accuracy with 15 medical experts and for augmentation effects with an osteoarthritis severity classification task. We devised a survey of 30 real and 30 DeepFake images for medical experts. The result showed that on average, more DeepFakes were mistaken for real than the reverse. The result signified sufficient DeepFake realism for deceiving the medical experts. Finally, our DeepFakes improved classification accuracy in an osteoarthritis severity classification task with scarce real data and transfer learning. In addition, in the same classification task, we replaced all real training data with DeepFakes and suffered only a [Formula: see text] loss from baseline accuracy in classifying real osteoarthritis X-rays.

Reflectance Measurement Method Based on Sensor Fusion of Frame-Based Hyperspectral Imager and Time-of-Flight Depth Camera.

Hyperspectral imaging and distance data have previously been used in aerial, forestry, agricultural, and medical imaging applications. Extracting meaningful information from a combination of different imaging modalities is difficult, as the image sensor fusion requires knowing the optical properties of the sensors, selecting the right optics and finding the sensors' mutual reference frame through calibration. In this research we demonstrate a method for fusing data from Fabry-Perot interferometer hyperspectral camera and a Kinect V2 time-of-flight depth sensing camera. We created an experimental application to demonstrate utilizing the depth augmented hyperspectral data to measure emission angle dependent reflectance from a multi-view inferred point cloud. We determined the intrinsic and extrinsic camera parameters through calibration, used global and local registration algorithms to combine point clouds from different viewpoints, created a dense point cloud and determined the angle dependent reflectances from it. The method could successfully combine the 3D point cloud data and hyperspectral data from different viewpoints of a reference colorchecker board. The point cloud registrations gained 0.29-0.36 fitness for inlier point correspondences and RMSE was approx. 2, which refers a quite reliable registration result. The RMSE of the measured reflectances between the front view and side views of the targets varied between 0.01 and 0.05 on average and the spectral angle between 1.5 and 3.2 degrees. The results suggest that changing emission angle has very small effect on the surface reflectance intensity and spectrum shapes, which was expected with the used colorchecker.

Hyperspectral Imaging for Non-invasive Diagnostics of Melanocytic Lesions.

Malignant melanoma poses a clinical diagnostic problem, since a large number of benign lesions are excised to find a single melanoma. This study assessed the accuracy of a novel non-invasive diagnostic technology, hyperspectral imaging, for melanoma detection. Lesions were imaged prior to excision and histopathological analysis. A deep neural network algorithm was trained twice to distinguish between histopathologically verified malignant and benign melanocytic lesions and to classify the separate subgroups. Furthermore, 2 different approaches were used: a majority vote classification and a pixel-wise classification. The study included 325 lesions from 285 patients. Of these, 74 were invasive melanoma, 88 melanoma in situ, 115 dysplastic naevi, and 48 non-dysplastic naevi. The study included a training set of 358,800 pixels and a validation set of 7,313 pixels, which was then tested with a training set of 24,375 pixels. The majority vote classification achieved high overall sensitivity of 95% and a specificity of 92% (95% confidence interval (95% CI) 0.024-0.029) in differentiating malignant from benign lesions. In the pixel-wise classification, the overall sensitivity and specificity were both 82% (95% CI 0.005-0.005). When divided into 4 subgroups, the diagnostic accuracy was lower. Hyperspectral imaging provides high sensitivity and specificity in distinguishing between naevi and melanoma. This novel method still needs further validation.

FPI Based Hyperspectral Imager for the Complex Surfaces-Calibration, Illumination and Applications.

Hyperspectral imaging (HSI) applications for biomedical imaging and dermatological applications have been recently under research interest. Medical HSI applications are non-invasive methods with high spatial and spectral resolution. HS imaging can be used to delineate malignant tumours, detect invasions, and classify lesion types. Typical challenges of these applications relate to complex skin surfaces, leaving some skin areas unreachable. In this study, we introduce a novel spectral imaging concept and conduct a clinical pre-test, the findings of which can be used to develop the concept towards a clinical application. The SICSURFIS spectral imager concept combines a piezo-actuated Fabry-Pérot interferometer (FPI) based hyperspectral imager, a specially designed LED module and several sizes of stray light protection cones for reaching and adapting to the complex skin surfaces. The imager is designed for the needs of photometric stereo imaging for providing the skin surface models (3D) for each captured wavelength. The captured HS images contained 33 selected wavelengths (ranging from 477 nm to 891 nm), which were captured simultaneously with accordingly selected LEDs and three specific angles of light. The pre-test results show that the data collected with the new SICSURFIS imager enable the use of the spectral and spatial domains with surface model information. The imager can reach complex skin surfaces. Healthy skin, basal cell carcinomas and intradermal nevi lesions were classified and delineated pixel-wise with promising results, but further studies are needed. The results were obtained with a convolutional neural network.

Differentiating Malignant from Benign Pigmented or Non-Pigmented Skin Tumours-A Pilot Study on 3D Hyperspectral Imaging of Complex Skin Surfaces and Convolutional Neural Networks.

Several optical imaging techniques have been developed to ease the burden of skin cancer disease on our health care system. Hyperspectral images can be used to identify biological tissues by their diffuse reflected spectra. In this second part of a three-phase pilot study, we used a novel hand-held SICSURFIS Spectral Imager with an adaptable field of view and target-wise selectable wavelength channels to provide detailed spectral and spatial data for lesions on complex surfaces. The hyperspectral images (33 wavelengths, 477-891 nm) provided photometric data through individually controlled illumination modules, enabling convolutional networks to utilise spectral, spatial, and skin-surface models for the analyses. In total, 42 lesions were studied: 7 melanomas, 13 pigmented and 7 intradermal nevi, 10 basal cell carcinomas, and 5 squamous cell carcinomas. All lesions were excised for histological analyses. A pixel-wise analysis provided map-like images and classified pigmented lesions with a sensitivity of 87% and a specificity of 93%, and 79% and 91%, respectively, for non-pigmented lesions. A majority voting analysis, which provided the most probable lesion diagnosis, diagnosed 41 of 42 lesions correctly. This pilot study indicates that our non-invasive hyperspectral imaging system, which involves shape and depth data analysed by convolutional neural networks, is feasible for differentiating between malignant and benign pigmented and non-pigmented skin tumours, even on complex skin surfaces.

Hyperspectral Imaging Reveals Spectral Differences and Can Distinguish Malignant Melanoma from Pigmented Basal Cell Carcinomas: A Pilot Study.

Pigmented basal cell carcinomas can be difficult to distinguish from melanocytic tumours. Hyperspectral imaging is a non-invasive imaging technique that measures the reflectance spectra of skin in vivo. The aim of this prospective pilot study was to use a convolutional neural network classifier in hyperspectral images for differential diagnosis between pigment-ed basal cell carcinomas and melanoma. A total of 26 pigmented lesions (10 pigmented basal cell carcinomas, 12 melanomas in situ, 4 invasive melanomas) were imaged with hyperspectral imaging and excised for histopatho-logical diagnosis. For 2-class classifier (melano-cytic tumours vs pigmented basal cell carcinomas) using the majority of the pixels to predict the class of the whole lesion, the results showed a sensitivity of 100% (95% confidence interval 81-100%), specificity of 90% (95% confidence interval 60-98%) and positive predictive value of 94% (95% confidence interval 73-99%). These results indicate that a convolutional neural network classifier can differentiate melanocytic tumours from pigmented basal cell carcinomas in hyperspectral images. Further studies are warranted in order to confirm these preliminary results, using larger samples and multiple tumour types, including all types of melanocytic lesions.

A Do-It-Yourself Hyperspectral Imager Brought to Practice with Open-Source Python.

Commercial hyperspectral imagers (HSIs) are expensive and thus unobtainable for large audiences or research groups with low funding. In this study, we used an existing do-it-yourself push-broom HSI design for which we provide software to correct for spectral smile aberration without using an optical laboratory. The software also corrects an aberration which we call tilt. The tilt is specific for the particular imager design used, but correcting it may be beneficial for other similar devices. The tilt and spectral smile were reduced to zero in terms of used metrics. The software artifact is available as an open-source Github repository. We also present improved casing for the imager design, and, for those readers interested in building their own HSI, we provide print-ready and modifiable versions of the 3D-models required in manufacturing the imager. To our best knowledge, solving the spectral smile correction problem without an optical laboratory has not been previously reported. This study re-solved the problem with simpler and cheaper tools than those commonly utilized. We hope that this study will promote easier access to hyperspectral imaging for all audiences regardless of their financial status and availability of an optical laboratory.

Rapid Quantification of Microalgae Growth with Hyperspectral Camera and Vegetation Indices.

Spectral cameras are traditionally used in remote sensing of microalgae, but increasingly also in laboratory-scale applications, to study and monitor algae biomass in cultures. Practical and cost-efficient protocols for collecting and analyzing hyperspectral data are currently needed. The purpose of this study was to test a commercial, easy-to-use hyperspectral camera to monitor the growth of different algae strains in liquid samples. Indices calculated from wavebands from transmission imaging were compared against algae abundance and wet biomass obtained from an electronic cell counter, chlorophyll a concentration, and chlorophyll fluorescence. A ratio of selected wavebands containing near-infrared and red turned out to be a powerful index because it was simple to calculate and interpret, yet it yielded strong correlations to abundances strain-specifically (0.85 < r < 0.96, p < 0.001). When all the indices formulated as A/B, A/(A + B) or (A - B)/(A + B), where A and B were wavebands of the spectral camera, were scrutinized, good correlations were found amongst them for biomass of each strain (0.66 < r < 0.98, p < 0.001). Comparison of near-infrared/red index to chlorophyll a concentration demonstrated that small-celled strains had higher chlorophyll absorbance compared to strains with larger cells. The comparison of spectral imaging to chlorophyll fluorescence was done for one strain of green algae and yielded strong correlations (near-infrared/red, r = 0.97, p < 0.001). Consequently, we described a simple imaging setup and information extraction based on vegetation indices that could be used to monitor algae cultures.

Generating Hyperspectral Skin Cancer Imagery using Generative Adversarial Neural Network.

In this study we develop a proof of concept of using generative adversarial neural networks in hyperspectral skin cancer imagery production. Generative adversarial neural network is a neural network, where two neural networks compete. The generator tries to produce data that is similar to the measured data, and the discriminator tries to correctly classify the data as fake or real. This is a reinforcement learning model, where both models get reinforcement based on their performance. In the training of the discriminator we use data measured from skin cancer patients. The aim for the study is to develop a generator for augmenting hyperspectral skin cancer imagery.

Erratum: Multilabel segmentation of cancer cell culture on vascular structures with deep neural networks (Erratum).

[This corrects the article DOI: 10.1117/1.JMI.7.2.024001.].

Multilabel segmentation of cancer cell culture on vascular structures with deep neural networks.

New increasingly complex in vitro cancer cell models are being developed. These new models seem to represent the cell behavior in vivo more accurately and have better physiological relevance than prior models. An efficient testing method for selecting the most optimal drug treatment does not exist to date. One proposed solution to the problem involves isolation of cancer cells from the patients' cancer tissue, after which they are exposed to potential drugs alone or in combinations to find the most optimal medication. To achieve this goal, methods that can efficiently quantify and analyze changes in tested cell are needed. Our study aimed to detect and segment cells and structures from cancer cell cultures grown on vascular structures in phase-contrast microscope images using U-Net neural networks to enable future drug efficacy assessments. We cultivated prostate carcinoma cell lines PC3 and LNCaP on the top of a matrix containing vascular structures. The cells were imaged with a Cell-IQ phase-contrast microscope. Automatic analysis of microscope images could assess the efficacy of tested drugs. The dataset included 36 RGB images and ground-truth segmentations with mutually not exclusive classes. The used method could distinguish vascular structures, cells, spheroids, and cell matter around spheroids in the test images. Some invasive spikes were also detected, but the method could not distinguish the invasive cells in the test images.

Uncertainty quantification on a spatial Markov-chain model for the progression of skin cancer.

A spatial Markov-chain model is formulated for the progression of skin cancer. The model is based on the division of the computational domain into nodal points, that can be in a binary state: either in 'cancer state' or in 'non-cancer state'. The model assigns probabilities for the non-reversible transition from 'non-cancer' state to the 'cancer state' that depend on the states of the neighbouring nodes. The likelihood of transition further depends on the life burden intensity of the UV-rays that the skin is exposed to. The probabilistic nature of the process and the uncertainty in the input data is assessed by the use of Monte Carlo simulations. A good fit between experiments on mice and our model has been obtained.

Delineating margins of lentigo maligna using a hyperspectral imaging system.

Lentigo maligna (LM) is an in situ form of melanoma which can progress into invasive lentigo maligna melanoma (LMM). Variations in the pigmentation and thus visibility of the tumour make assessment of lesion borders challenging. We tested hyperspectral imaging system (HIS) in in vivo preoperative delineation of LM and LMM margins. We compared lesion margins delineated by HIS with those estimated clinically, and confirmed histologically. A total of 14 LMs and 5 LMMs in 19 patients were included. HIS analysis matched the histo-pathological analysis in 18/19 (94.7%) cases while in 1/19 (5.3%) cases HIS showed lesion extension not confirmed by histopathology (false positives). Compared to clinical examination, HIS defined lesion borders more accurately in 10/19 (52.6%) of cases (wider, n = 7 or smaller, n = 3) while in 8/19 (42.1%) cases lesion borders were the same as delineated clinically as confirmed histologically. Thus, HIS is useful for the detection of subclinical LM/LMM borders.

Detecting field cancerization using a hyperspectral imaging system.

BACKGROUND: Field cancerization denotes subclinical abnormalities in a tissue chronically exposed to UV radiation. These abnormalities can be found surrounding the clinically visible actinic keratoses. OBJECTIVES: The aim of this study was to test the feasibility of a hyperspectral imaging system in the detection of multiple clinical and subclinical AKs for early treatment of the affected areas. MATERIALS AND METHODS: Altogether 52 clinical AKs in 12 patients were included in this study. In six patients digital photos were taken of the naive AKs, and again after methylaminolevulinate(MAL)-fluorescence diagnosis which was used to teach HIS to find subclinical lesions. After 2-3 days when the MAL had vanished, the hyperspectral images were taken. Biopsies were taken from clinical AKs, healthy-looking skin and several suspected subclinical AKs. In the other six patients digital and hyperspectral images were taken of the naive AKs followed by one biopsy per patient. RESULTS: HIS detected all clinically visible 52 AKs and numerous subclinical lesions. The histopathology of the 33 biopsied lesions were concordant with the HIS results showing either AK (n = 28) or photodamage (n = 5). Of the 28 histopathologically confirmed AKs, 16 were subclinical. A specific diffuse reflectance spectrum of an AK and healthy skin was defined. CONCLUSION: The hyperspectral imaging system offers a new, non-invasive method for early detection of field cancerization. Lasers Surg. Med. 45:410-417, 2013. © 2013 Wiley Periodicals, Inc.