Deep learning for AI-based diagnosis of skin-related neglected tropical diseases: A pilot study.

BACKGROUND: Deep learning, which is a part of a broader concept of artificial intelligence (AI) and/or machine learning has achieved remarkable success in vision tasks. While there is growing interest in the use of this technology in diagnostic support for skin-related neglected tropical diseases (skin NTDs), there have been limited studies in this area and fewer focused on dark skin. In this study, we aimed to develop deep learning based AI models with clinical images we collected for five skin NTDs, namely, Buruli ulcer, leprosy, mycetoma, scabies, and yaws, to understand how diagnostic accuracy can or cannot be improved using different models and training patterns. METHODOLOGY: This study used photographs collected prospectively in Côte d'Ivoire and Ghana through our ongoing studies with use of digital health tools for clinical data documentation and for teledermatology. Our dataset included a total of 1,709 images from 506 patients. Two convolutional neural networks, ResNet-50 and VGG-16 models were adopted to examine the performance of different deep learning architectures and validate their feasibility in diagnosis of the targeted skin NTDs. PRINCIPAL FINDINGS: The two models were able to correctly predict over 70% of the diagnoses, and there was a consistent performance improvement with more training samples. The ResNet-50 model performed better than the VGG-16 model. A model trained with PCR confirmed cases of Buruli ulcer yielded 1-3% increase in prediction accuracy across all diseases, except, for mycetoma, over a model which training sets included unconfirmed cases. CONCLUSIONS: Our approach was to have the deep learning model distinguish between multiple pathologies simultaneously-which is close to real-world practice. The more images used for training, the more accurate the diagnosis became. The percentages of correct diagnosis increased with PCR-positive cases of Buruli ulcer. This demonstrated that it may be better to input images from the more accurately diagnosed cases in the training models also for achieving better accuracy in the generated AI models. However, the increase was marginal which may be an indication that the accuracy of clinical diagnosis alone is reliable to an extent for Buruli ulcer. Diagnostic tests also have their flaws, and they are not always reliable. One hope for AI is that it will objectively resolve this gap between diagnostic tests and clinical diagnoses with the addition of another tool. While there are still challenges to be overcome, there is a potential for AI to address the unmet needs where access to medical care is limited, like for those affected by skin NTDs.

Marginalized Augmented Few-Shot Domain Adaptation.

Domain adaptation (DA) has recently drawn a lot of attention, as it facilitates unlabeled target learning by borrowing knowledge from an external source domain. Most existing DA solutions seek to align feature representations between the labeled source and unlabeled target data. However, the scarcity of target data easily results in negative transfer, as it misleads the cross DA to the dominance of the source. To address the challenging few-shot domain adaptation (FSDA) problem, in this article, we propose a novel marginalized augmented FSDA (MAF) approach to address the cross-domain distribution disparity and insufficiency of target data simultaneously. On the one hand, cross-domain continuity augmentation (CCA) synthesizes abundant intermediate patterns across domains leading to a continuous domain-invariant latent space. On the other hand, sufficient source-supervised semantic augmentation (SSA) is explored to progressively diversify the conditional distribution within and across domains. Moreover, the proposed augmentation strategies are implemented efficiently via an expected transferable cross-entropy (CE) loss over the augmented distribution instead of explicit data synthesis, and minimizing the upper bound of the expected loss introduces negligible extra computing cost. Experimentally, our method outperforms the state of the art in various FSDA benchmarks, which demonstrates the effectiveness and contribution of our work. Our source code is provided at https://github.com/scottjingtt/MAF.git.

Deep learning for AI-based diagnosis of skin-related neglected tropical diseases: a pilot study.

Background: Deep learning, which is a part of a broader concept of artificial intelligence (AI) and/or machine learning has achieved remarkable success in vision tasks. While there is growing interest in the use of this technology in diagnostic support for skin-related neglected tropical diseases (skin NTDs), there have been limited studies in this area and fewer focused on dark skin. In this study, we aimed to develop deep learning based AI models with clinical images we collected for five skin NTDs, namely, Buruli ulcer, leprosy, mycetoma, scabies, and yaws, to understand how diagnostic accuracy can or cannot be improved using different models and training patterns. Methodology: This study used photographs collected prospectively in Côte d'Ivoire and Ghana through our ongoing studies with use of digital health tools for clinical data documentation and for teledermatology. Our dataset included a total of 1,709 images from 506 patients. Two convolutional neural networks, ResNet-50 and VGG-16 models were adopted to examine the performance of different deep learning architectures and validate their feasibility in diagnosis of the targeted skin NTDs. Principal findings: The two models were able to correctly predict over 70% of the diagnoses, and there was a consistent performance improvement with more training samples. The ResNet-50 model performed better than the VGG-16 model. Model trained with PCR confirmed cases of Buruli ulcer yielded 1-3% increase in prediction accuracy over training sets including unconfirmed cases. Conclusions: Our approach was to have the deep learning model distinguish between multiple pathologies simultaneously - which is close to real-world practice. The more images used for training, the more accurate the diagnosis became. The percentages of correct diagnosis increased with PCR-positive cases of Buruli ulcer. This demonstrated that it may be better to input images from the more accurately diagnosed cases in the training models also for achieving better accuracy in the generated AI models. However, the increase was marginal which may be an indication that the accuracy of clinical diagnosis alone is reliable to an extent for Buruli ulcer. Diagnostic tests also have its flaws, and they are not always reliable. One hope for AI is that it will objectively resolve this gap between diagnostic tests and clinical diagnoses with addition of another tool. While there are still challenges to be overcome, there is a potential for AI to address the unmet needs where access to medical care is limited, like for those affected by skin NTDs. AUTHOR SUMMARY: The diagnosis of skin diseases depends in large part, though not exclusively on visual inspection. The diagnosis and management of these diseases is thus particularly amenable to teledermatology approaches. The widespread availability of cell phone technology and electronic information transfer provides new potential for access to health care in low-income countries, yet there are limited efforts targeting these neglected populations with dark skin and consequently limited availability of tools. In this study, we leveraged a collection of skin images gathered through a system of teledermatology in the West African countries of Côte d'Ivoire and Ghana, and applied deep learning, a form of artificial intelligence (AI) - to see if deep learning models can distinguish between different diseases and support their diagnosis. Skin-related neglected tropical diseases, or skin NTDs, prevail in these regions and were our target conditions: Buruli ulcer, leprosy, mycetoma, scabies, and yaws. The accuracy of prediction depended on the number of images that were fed into the model for training with marginal improvement using laboratory confirmed cases in training. Using more images and greater efforts in this area, it is possible that AI can help address the unmet needs where access to medical care is limited.

Exploiting Missing Value Patterns for a Backdoor Attack on Machine Learning Models of Electronic Health Records: Development and Validation Study.

BACKGROUND: A backdoor attack controls the output of a machine learning model in 2 stages. First, the attacker poisons the training data set, introducing a back door into the victim's trained model. Second, during test time, the attacker adds an imperceptible pattern called a trigger to the input values, which forces the victim's model to output the attacker's intended values instead of true predictions or decisions. While backdoor attacks pose a serious threat to the reliability of machine learning-based medical diagnostics, existing backdoor attacks that directly change the input values are detectable relatively easily. OBJECTIVE: The goal of this study was to propose and study a robust backdoor attack on mortality-prediction machine learning models that use electronic health records. We showed that our backdoor attack grants attackers full control over classification outcomes for safety-critical tasks such as mortality prediction, highlighting the importance of undertaking safe artificial intelligence research in the medical field. METHODS: We present a trigger generation method based on missing patterns in electronic health record data. Compared to existing approaches, which introduce noise into the medical record, the proposed backdoor attack makes it simple to construct backdoor triggers without prior knowledge. To effectively avoid detection by manual inspectors, we employ variational autoencoders to learn the missing patterns in normal electronic health record data and produce trigger data that appears similar to this data. RESULTS: We experimented with the proposed backdoor attack on 4 machine learning models (linear regression, multilayer perceptron, long short-term memory, and gated recurrent units) that predict in-hospital mortality using a public electronic health record data set. The results showed that the proposed technique achieved a significant drop in the victim's discrimination performance (reducing the area under the precision-recall curve by at most 0.45), with a low poisoning rate (2%) in the training data set. In addition, the impact of the attack on general classification performance was negligible (it reduced the area under the precision-recall curve by an average of 0.01025), which makes it difficult to detect the presence of poison. CONCLUSIONS: To the best of our knowledge, this is the first study to propose a backdoor attack that uses missing information from tabular data as a trigger. Through extensive experiments, we demonstrated that our backdoor attack can inflict severe damage on medical machine learning classifiers in practice.

Augmented Multimodality Fusion for Generalized Zero-Shot Sketch-Based Visual Retrieval.

Zero-shot sketch-based image retrieval (ZS-SBIR) has attracted great attention recently, due to the potential application of sketch-based retrieval under zero-shot scenarios, where the categories of query sketches and gallery photos are not observed in the training stage. However, it is still under insufficient exploration for the general and practical scenario when the query sketches and gallery photos contain both seen and unseen categories. Such a problem is defined as generalized zero-shot sketch-based image retrieval (GZS-SBIR), which is the focus of this work. To this end, we propose a novel Augmented Multi-modality Fusion (AMF) framework to generalize seen concepts to unobserved ones efficiently. Specifically, a novel knowledge discovery module named cross-domain augmentation is designed in both visual and semantic space to mimic novel knowledge unseen from the training stage, which is the key to handling the GZS-SBIR challenge. Moreover, a triplet domain alignment module is proposed to couple the cross-domain distribution between photo and sketch in visual space. To enhance the robustness of our model, we explore embedding propagation to refine both visual and semantic features by removing undesired noise. Eventually, visual-semantic fusion representations are concatenated for further domain discrimination and task-specific recognition, which tend to trigger the cross-domain alignment in both visual and semantic feature space. Experimental evaluations are conducted on popular ZS-SBIR benchmarks as well as a new evaluation protocol designed for GZS-SBIR from DomainNet dataset with more diverse sub-domains, and the promising results demonstrate the superiority of the proposed solution over other baselines. The source code is available at https://github.com/scottjingtt/AMF_GZS_SBIR.git.

Deep learning 2D and 3D optical sectioning microscopy using cross-modality Pix2Pix cGAN image translation.

Structured illumination microscopy (SIM) reconstructs optically-sectioned images of a sample from multiple spatially-patterned wide-field images, but the traditional single non-patterned wide-field images are more inexpensively obtained since they do not require generation of specialized illumination patterns. In this work, we translated wide-field fluorescence microscopy images to optically-sectioned SIM images by a Pix2Pix conditional generative adversarial network (cGAN). Our model shows the capability of both 2D cross-modality image translation from wide-field images to optical sections, and further demonstrates potential to recover 3D optically-sectioned volumes from wide-field image stacks. The utility of the model was tested on a variety of samples including fluorescent beads and fresh human tissue samples.

Privacy versus open science.

Pervasive internet and sensor technologies promise to revolutionize psychological science. However, the data collected using these technologies are often very personal-indeed, the value of the data is often directly related to how personal they are. At the same time, driven by the replication crisis, there is a sustained push to publish data to open repositories. These movements are in fundamental conflict. In this article, we propose a way to navigate this issue. We argue that there are significant advantages to be gained by ceding the ownership of data to the participants who generate the data. We then provide desiderata for a privacy-preserving platform. In particular, we suggest that researchers should use an interface to perform experiments and run analyses, rather than observing the stimuli themselves. We argue that this method not only improves privacy but will also encourage greater compliance with good research practices than is possible through open repositories.

Fluid Dynamic Models for Bhattacharyya-Based Discriminant Analysis.

Classical discriminant analysis attempts to discover a low-dimensional subspace where class label information is maximally preserved under projection. Canonical methods for estimating the subspace optimize an information-theoretic criterion that measures the separation between the class-conditional distributions. Unfortunately, direct optimization of the information-theoretic criteria is generally non-convex and intractable in high-dimensional spaces. In this work, we propose a novel, tractable algorithm for discriminant analysis that considers the class-conditional densities as interacting fluids in the high-dimensional embedding space. We use the Bhattacharyya criterion as a potential function that generates forces between the interacting fluids, and derive a computationally tractable method for finding the low-dimensional subspace that optimally constrains the resulting fluid flow. We show that this model properly reduces to the optimal solution for homoscedastic data as well as for heteroscedastic Gaussian distributions with equal means. We also extend this model to discover optimal filters for discriminating Gaussian processes and provide experimental results and comparisons on a number of datasets.

Regional manifold learning for disease classification.

While manifold learning from images itself has become widely used in medical image analysis, the accuracy of existing implementations suffers from viewing each image as a single data point. To address this issue, we parcellate images into regions and then separately learn the manifold for each region. We use the regional manifolds as low-dimensional descriptors of high-dimensional morphological image features, which are then fed into a classifier to identify regions affected by disease. We produce a single ensemble decision for each scan by the weighted combination of these regional classification results. Each weight is determined by the regional accuracy of detecting the disease. When applied to cardiac magnetic resonance imaging of 50 normal controls and 50 patients with reconstructive surgery of Tetralogy of Fallot, our method achieves significantly better classification accuracy than approaches learning a single manifold across the entire image domain.

Dimensional information-theoretic measurement of facial emotion expressions in schizophrenia.

Altered facial expressions of emotions are characteristic impairments in schizophrenia. Ratings of affect have traditionally been limited to clinical rating scales and facial muscle movement analysis, which require extensive training and have limitations based on methodology and ecological validity. To improve reliable assessment of dynamic facial expression changes, we have developed automated measurements of facial emotion expressions based on information-theoretic measures of expressivity of ambiguity and distinctiveness of facial expressions. These measures were examined in matched groups of persons with schizophrenia (n = 28) and healthy controls (n = 26) who underwent video acquisition to assess expressivity of basic emotions (happiness, sadness, anger, fear, and disgust) in evoked conditions. Persons with schizophrenia scored higher on ambiguity, the measure of conditional entropy within the expression of a single emotion, and they scored lower on distinctiveness, the measure of mutual information across expressions of different emotions. The automated measures compared favorably with observer-based ratings. This method can be applied for delineating dynamic emotional expressivity in healthy and clinical populations.

FLOOR: fusing locally optimal registrations.

Most registration algorithms, such as Demons, align two scans by iteratively finding the deformation minimizing the image dissimilarity at each location and smoothing this minimum across the image domain. These methods generally get stuck in local minima, are negatively impacted by missing correspondences between the images, and require careful tuning of the smoothing parameters to achieve optimal results. In this paper, we propose to improve on those issues by choosing the minimum from a set of candidates. Our method generates such candidates by running the registration algorithm multiple times varying the setting of the smoothing and the image domain. We iteratively refine those candidates by fusing them with the outcome of alternative approaches and locally adapting the smoothing parameters. We implement our algorithm based on Demons and find alternative minima via manifold learning. Compared to those two methods, our 600 pairwise registrations of cardiac MRIs significantly better handle the large shape variations of the heart and the different field of views captured by scans.

COMBINING REGIONAL METRICS FOR DISEASE-RELATED BRAIN POPULATION ANALYSIS.

In this paper, we present a new metric combining regional measurements to improve image based population studies that use manifold learning techniques. These studies currently rely on a single score over the whole brain image domain. Thus, they require large amount of training data to uncover spatially complex variation in the whole brain impacted by diseases. We reduce the impact of this issue by first computing pairwise measurements in local regions separately and then combining regional measurements into a single pairwise metric. We apply the new metric to learn the manifold of ADNI data and evaluate the resulting morphological representation by fitting multiple linear regression models to the mini-mental state examination (MMSE) score. The regression models show that the morphological representations from the proposed metric achieves higher estimation accuracy of MMSE score compared to those from the conventional global scores.

Regional manifold learning for deformable registration of brain MR images.

We propose a method for deformable registration based on learning the manifolds of individual brain regions. Recent publications on registration of medical images advocate the use of manifold learning in order to confine the search space to anatomically plausible deformations. Existing methods construct manifolds based on a single metric over the entire image domain thus frequently miss regional brain variations. We address this issue by first learning manifolds for specific regions and then computing region-specific deformations from these manifolds. We then determine deformations for the entire image domain by learning the global manifold in such a way that it preserves the region-specific deformations. We evaluate the accuracy of our method by applying it to the LPBA40 dataset and measuring the overlap of the deformed segmentations. The result shows significant improvement in registration accuracy on cortex regions compared to other state of the art methods.

Automated Facial Action Coding System for dynamic analysis of facial expressions in neuropsychiatric disorders.

Facial expression is widely used to evaluate emotional impairment in neuropsychiatric disorders. Ekman and Friesen's Facial Action Coding System (FACS) encodes movements of individual facial muscles from distinct momentary changes in facial appearance. Unlike facial expression ratings based on categorization of expressions into prototypical emotions (happiness, sadness, anger, fear, disgust, etc.), FACS can encode ambiguous and subtle expressions, and therefore is potentially more suitable for analyzing the small differences in facial affect. However, FACS rating requires extensive training, and is time consuming and subjective thus prone to bias. To overcome these limitations, we developed an automated FACS based on advanced computer science technology. The system automatically tracks faces in a video, extracts geometric and texture features, and produces temporal profiles of each facial muscle movement. These profiles are quantified to compute frequencies of single and combined Action Units (AUs) in videos, and they can facilitate a statistical study of large populations in disorders known to impact facial expression. We derived quantitative measures of flat and inappropriate facial affect automatically from temporal AU profiles. Applicability of the automated FACS was illustrated in a pilot study, by applying it to data of videos from eight schizophrenia patients and controls. We created temporal AU profiles that provided rich information on the dynamics of facial muscle movements for each subject. The quantitative measures of flatness and inappropriateness showed clear differences between patients and the controls, highlighting their potential in automatic and objective quantification of symptom severity.

GRAM: A framework for geodesic registration on anatomical manifolds.

Medical image registration is a challenging problem, especially when there is large anatomical variation in the anatomies. Geodesic registration methods have been proposed to solve the large deformation registration problem. However, analytically defined geodesic paths may not coincide with biologically plausible paths of registration, since the manifold of diffeomorphisms is immensely broader than the manifold spanned by diffeomorphisms between real anatomies. In this paper, we propose a novel framework for large deformation registration using the learned manifold of anatomical variation in the data. In this framework, a large deformation between two images is decomposed into a series of small deformations along the shortest path on an empirical manifold that represents anatomical variation. Using a manifold learning technique, the major variation of the data can be visualized by a low-dimensional embedding, and the optimal group template is chosen as the geodesic mean on the manifold. We demonstrate the advantages of the proposed framework over direct registration with both simulated and real databases of brain images.

Efficient large deformation registration via geodesics on a learned manifold of images.

Geodesic registration methods have been used to solve the large deformation registration problems, which are hard to solve with conventional registration methods. However, analytically defined geodesics may not coincide with anatomically optimal paths of registration. In this paper we propose a novel and efficient method for large deformation registration by learning the underlying structure of the data using a manifold learning technique. In this method a large deformation between two images is decomposed into a series of small deformations along the shortest path on the graph that approximates the metric structure of data. Furthermore, the graph representation allows us to estimate the optimal group template by minimizing geodesic distances. We demonstrate the advantages of the proposed method with synthetic 2D images and real 3D mice brain volumes.