Knowledge Distillation via Constrained Variational Inference.

Knowledge distillation has been used to capture the knowledge of a teacher model and distill it into a student model with some desirable characteristics such as being smaller, more efficient, or more generalizable. In this paper, we propose a framework for distilling the knowledge of a powerful discriminative model such as a neural network into commonly used graphical models known to be more interpretable (e.g., topic models, autoregressive Hidden Markov Models). Posterior of latent variables in these graphical models (e.g., topic proportions in topic models) is often used as feature representation for predictive tasks. However, these posterior-derived features are known to have poor predictive performance compared to the features learned via purely discriminative approaches. Our framework constrains variational inference for posterior variables in graphical models with a similarity preserving constraint. This constraint distills the knowledge of the discriminative model into the graphical model by ensuring that input pairs with (dis)similar representation in the teacher model also have (dis)similar representation in the student model. By adding this constraint to the variational inference scheme, we guide the graphical model to be a reasonable density model for the data while having predictive features which are as close as possible to those of a discriminative model. To make our framework applicable to a wide range of graphical models, we build upon the Automatic Differentiation Variational Inference (ADVI), a black-box inference framework for graphical models. We demonstrate the effectiveness of our framework on two real-world tasks of disease subtyping and disease trajectory modeling.

Incorporating External Information in Tissue Subtyping: A Topic Modeling Approach.

Probabilistic topic models, have been widely deployed for various applications such as learning disease or tissue subtypes. Yet, learning the parameters of such models is usually an ill-posed problem and may result in losing valuable information about disease severity. A common approach is to add a discriminative loss term to the generative model's loss in order to learn a representation that is also predictive of disease severity. However, finding a balance between these two losses is not straightforward. We propose an alternative way in this paper. We develop a framework which allows for incorporating external covariates into the generative model's approximate posterior. These covariates can have more discriminative power for disease severity compared to the representation that we extract from the posterior distribution. For instance, they can be features extracted from a neural network which predicts disease severity from CT images. Effectively, we enforce the generative model's approximate posterior to reside in the subspace of these discriminative covariates. We illustrate our method's application on a large-scale lung CT study of Chronic Obstructive Pulmonary Disease (COPD), a highly heterogeneous disease. We aim at identifying tissue subtypes by using a variant of topic model as a generative model. We quantitatively evaluate the predictive performance of the inferred subtypes and demonstrate that our method outperforms or performs on par with some reasonable baselines. We also show that some of the discovered subtypes are correlated with genetic measurements, suggesting that the identified subtypes may characterize the disease's underlying etiology.

Nonparametric Spherical Topic Modeling with Word Embeddings.

Traditional topic models do not account for semantic regularities in language. Recent distributional representations of words exhibit semantic consistency over directional metrics such as cosine similarity. However, neither categorical nor Gaussian observational distributions used in existing topic models are appropriate to leverage such correlations. In this paper, we propose to use the von Mises-Fisher distribution to model the density of words over a unit sphere. Such a representation is well-suited for directional data. We use a Hierarchical Dirichlet Process for our base topic model and propose an efficient inference algorithm based on Stochastic Variational Inference. This model enables us to naturally exploit the semantic structures of word embeddings while flexibly discovering the number of topics. Experiments demonstrate that our method outperforms competitive approaches in terms of topic coherence on two different text corpora while offering efficient inference.

Generative Method to Discover Genetically Driven Image Biomarkers.

We present a generative probabilistic approach to discovery of disease subtypes determined by the genetic variants. In many diseases, multiple types of pathology may present simultaneously in a patient, making quantification of the disease challenging. Our method seeks common co-occurring image and genetic patterns in a population as a way to model these two different data types jointly. We assume that each patient is a mixture of multiple disease subtypes and use the joint generative model of image and genetic markers to identify disease subtypes guided by known genetic influences. Our model is based on a variant of the so-called topic models that uncover the latent structure in a collection of data. We derive an efficient variational inference algorithm to extract patterns of co-occurrence and to quantify the presence of heterogeneous disease processes in each patient. We evaluate the method on simulated data and illustrate its use in the context of Chronic Obstructive Pulmonary Disease (COPD) to characterize the relationship between image and genetic signatures of COPD subtypes in a large patient cohort.

The prognostic value of BRAF mutation in colorectal cancer and melanoma: a systematic review and meta-analysis.

BACKGROUND: Mutation of BRAF is a predominant event in cancers with poor prognosis such as melanoma and colorectal cancer. BRAF mutation leads to a constitutive activation of mitogen activated protein kinase pathway which is essential for cell proliferation and tumor progression. Despite tremendous efforts made to target BRAF for cancer treatment, the correlation between BRAF mutation and patient survival is still a matter of controversy. METHODS/PRINCIPAL FINDINGS: Clinical studies on the correlation between BRAF mutation and patient survival were retrieved from MEDLINE and EMBASE databases between June 2002 and December 2011. One hundred twenty relevant full text studies were categorized based on study design and cancer type. Publication bias was evaluated for each category and pooled hazard ratio (HR) with 95% confidence interval (CI) was calculated using random or fixed effect meta-analysis based on the percentage of heterogeneity. Twenty six studies on colorectal cancer (11,773 patients) and four studies on melanoma (674 patients) were included in our final meta-analysis. The average prevalence of BRAF mutation was 9.6% in colorectal cancer, and 47.8% in melanoma reports. We found that BRAF mutation increases the risk of mortality in colorectal cancer patients for more than two times; HR = 2.25 (95% CI, 1.82-2.83). In addition, we revealed that BRAF mutation also increases the risk of mortality in melanoma patients by 1.7 times (95% CI, 1.37-2.12). CONCLUSIONS: We revealed that BRAF mutation is an absolute risk factor for patient survival in colorectal cancer and melanoma.