Lead/Lag directionality is not generally equivalent to causality in nonlinear systems: Comparison of phase slope index and conditional mutual information.

Applications of causal techniques to neural time series have increased extensively over last decades, including a wide and diverse family of methods focusing on electroencephalogram (EEG) analysis. Besides connectivity inferred in defined frequency bands, there is a growing interest in the analysis of cross-frequency interactions, in particular phase and amplitude coupling and directionality. Some studies show contradicting results of coupling directionality from high frequency to low frequency signal components, in spite of generally considered modulation of a high-frequency amplitude by a low-frequency phase. We have compared two widely used methods to estimate the directionality in cross frequency coupling: conditional mutual information (CMI) and phase slope index (PSI). The latter, applied to infer cross-frequency phase-amplitude directionality from animal intracranial recordings, gives opposite results when comparing to CMI. Both metrics were tested in a numerically simulated example of unidirectionally coupled Rössler systems, which helped to find the explanation of the contradictory results: PSI correctly estimates the lead/lag relationship which, however, is not generally equivalent to causality in the sense of directionality of coupling in nonlinear systems, correctly inferred by using CMI with surrogate data testing.

A Conditional Mutual Information Estimator for Mixed Data and an Associated Conditional Independence Test.

In this study, we focus on mixed data which are either observations of univariate random variables which can be quantitative or qualitative, or observations of multivariate random variables such that each variable can include both quantitative and qualitative components. We first propose a novel method, called CMIh, to estimate conditional mutual information taking advantages of the previously proposed approaches for qualitative and quantitative data. We then introduce a new local permutation test, called LocAT for local adaptive test, which is well adapted to mixed data. Our experiments illustrate the good behaviour of CMIh and LocAT, and show their respective abilities to accurately estimate conditional mutual information and to detect conditional (in)dependence for mixed data.

Adaptation of Partial Mutual Information from Mixed Embedding to Discrete-Valued Time Series.

A causality analysis aims at estimating the interactions of the observed variables and subsequently the connectivity structure of the observed dynamical system or stochastic process. The partial mutual information from mixed embedding (PMIME) is found appropriate for the causality analysis of continuous-valued time series, even of high dimension, as it applies a dimension reduction by selecting the most relevant lag variables of all the observed variables to the response, using conditional mutual information (CMI). The presence of lag components of the driving variable in this vector implies a direct causal (driving-response) effect. In this study, the PMIME is appropriately adapted to discrete-valued multivariate time series, called the discrete PMIME (DPMIME). An appropriate estimation of the discrete probability distributions and CMI for discrete variables is implemented in the DPMIME. Further, the asymptotic distribution of the estimated CMI is derived, allowing for a parametric significance test for the CMI in the DPMIME, whereas for the PMIME, there is no parametric test for the CMI and the test is performed using resampling. Monte Carlo simulations are performed using different generating systems of discrete-valued time series. The simulation suggests that the parametric significance test for the CMI in the progressive algorithm of the DPMIME is compared favorably to the corresponding resampling significance test, and the accuracy of the DPMIME in the estimation of direct causality converges with the time-series length to the accuracy of the PMIME. Further, the DPMIME is used to investigate whether the global financial crisis has an effect on the causality network of the financial world market.

Information Bottleneck Analysis by a Conditional Mutual Information Bound.

Task-nuisance decomposition describes why the information bottleneck loss I(z;x)-ßI(z;y) is a suitable objective for supervised learning. The true category y is predicted for input x using latent variables z. When n is a nuisance independent from y, I(z;n) can be decreased by reducing I(z;x) since the latter upper bounds the former. We extend this framework by demonstrating that conditional mutual information I(z;x|y) provides an alternative upper bound for I(z;n). This bound is applicable even if z is not a sufficient representation of x, that is, I(z;y)≠I(x;y). We used mutual information neural estimation (MINE) to estimate I(z;x|y). Experiments demonstrated that I(z;x|y) is smaller than I(z;x) for layers closer to the input, matching the claim that the former is a tighter bound than the latter. Because of this difference, the information plane differs when I(z;x|y) is used instead of I(z;x).

Categorical Nature of Major Factor Selection via Information Theoretic Measurements.

Without assuming any functional or distributional structure, we select collections of major factors embedded within response-versus-covariate (Re-Co) dynamics via selection criteria [C1: confirmable] and [C2: irrepaceable], which are based on information theoretic measurements. The two criteria are constructed based on the computing paradigm called Categorical Exploratory Data Analysis (CEDA) and linked to Wiener-Granger causality. All the information theoretical measurements, including conditional mutual information and entropy, are evaluated through the contingency table platform, which primarily rests on the categorical nature within all involved features of any data types: quantitative or qualitative. Our selection task identifies one chief collection, together with several secondary collections of major factors of various orders underlying the targeted Re-Co dynamics. Each selected collection is checked with algorithmically computed reliability against the finite sample phenomenon, and so is each member's major factor individually. The developments of our selection protocol are illustrated in detail through two experimental examples: a simple one and a complex one. We then apply this protocol on two data sets pertaining to two somewhat related but distinct pitching dynamics of two pitch types: slider and fastball. In particular, we refer to a specific Major League Baseball (MLB) pitcher and we consider data of multiple seasons.

Neural Estimator of Information for Time-Series Data with Dependency.

Novel approaches to estimate information measures using neural networks are well-celebrated in recent years both in the information theory and machine learning communities. These neural-based estimators are shown to converge to the true values when estimating mutual information and conditional mutual information using independent samples. However, if the samples in the dataset are not independent, the consistency of these estimators requires further investigation. This is of particular interest for a more complex measure such as the directed information, which is pivotal in characterizing causality and is meaningful over time-dependent variables. The extension of the convergence proof for such cases is not trivial and demands further assumptions on the data. In this paper, we show that our neural estimator for conditional mutual information is consistent when the dataset is generated with samples of a stationary and ergodic source. In other words, we show that our information estimator using neural networks converges asymptotically to the true value with probability one. Besides universal functional approximation of neural networks, a core lemma to show the convergence is Birkhoff's ergodic theorem. Additionally, we use the technique to estimate directed information and demonstrate the effectiveness of our approach in simulations.

Vine copula selection using mutual information for hydrological dependence modeling.

Hydrological risk analysis and management entails multivariate modeling which requires modeling the structure of dependence among different variables. Vine copulas have been increasing applied in multivariate modeling wherein the selection of vine copula structure plays a critical role. Inspired by the relationship between Mutual information (MI) and copula entropy (CE), this study discussed the connection between conditional mutual information (CMI) and CE and developed a mutual information-based sequential approach to select a vine structure which was based on original observations, and model-independent. Then, to reduce the complexity of R-vine copulas, a statistical method-based truncation procedure was applied. Finally, an MI-based approach for hydrological dependence modeling was developed. Two types of hydrological processes with different dependence structures were utilized to show the performance of the proposed approach: (i) drought characterization: showing a D-vine structure; and (ii) multi-site streamflow dependence: showing a C-vine structure. Results indicated that the MI-based approach satisfactorily modeled different kinds of dependence structure and yielded more information on variables in comparison with traditional tau-based approach.

Analysis of Information-Based Nonparametric Variable Selection Criteria.

We consider a nonparametric Generative Tree Model and discuss a problem of selecting active predictors for the response in such scenario. We investigated two popular information-based selection criteria: Conditional Infomax Feature Extraction (CIFE) and Joint Mutual information (JMI), which are both derived as approximations of Conditional Mutual Information (CMI) criterion. We show that both criteria CIFE and JMI may exhibit different behavior from CMI, resulting in different orders in which predictors are chosen in variable selection process. Explicit formulae for CMI and its two approximations in the generative tree model are obtained. As a byproduct, we establish expressions for an entropy of a multivariate gaussian mixture and its mutual information with mixing distribution.

Part mutual information for quantifying direct associations in networks.

Quantitatively identifying direct dependencies between variables is an important task in data analysis, in particular for reconstructing various types of networks and causal relations in science and engineering. One of the most widely used criteria is partial correlation, but it can only measure linearly direct association and miss nonlinear associations. However, based on conditional independence, conditional mutual information (CMI) is able to quantify nonlinearly direct relationships among variables from the observed data, superior to linear measures, but suffers from a serious problem of underestimation, in particular for those variables with tight associations in a network, which severely limits its applications. In this work, we propose a new concept, "partial independence," with a new measure, "part mutual information" (PMI), which not only can overcome the problem of CMI but also retains the quantification properties of both mutual information (MI) and CMI. Specifically, we first defined PMI to measure nonlinearly direct dependencies between variables and then derived its relations with MI and CMI. Finally, we used a number of simulated data as benchmark examples to numerically demonstrate PMI features and further real gene expression data from Escherichia coli and yeast to reconstruct gene regulatory networks, which all validated the advantages of PMI for accurately quantifying nonlinearly direct associations in networks.

Structure Learning of Bayesian Network Based on Adaptive Thresholding.

Direct dependencies and conditional dependencies in restricted Bayesian network classifiers (BNCs) are two basic kinds of dependencies. Traditional approaches, such as filter and wrapper, have proved to be beneficial to identify non-significant dependencies one by one, whereas the high computational overheads make them inefficient especially for those BNCs with high structural complexity. Study of the distributions of information-theoretic measures provides a feasible approach to identifying non-significant dependencies in batch that may help increase the structure reliability and avoid overfitting. In this paper, we investigate two extensions to the k-dependence Bayesian classifier, MI-based feature selection, and CMI-based dependence selection. These two techniques apply a novel adaptive thresholding method to filter out redundancy and can work jointly. Experimental results on 30 datasets from the UCI machine learning repository demonstrate that adaptive thresholds can help distinguish between dependencies and independencies and the proposed algorithm achieves competitive classification performance compared to several state-of-the-art BNCs in terms of 0-1 loss, root mean squared error, bias, and variance.

Simple Stopping Criteria for Information Theoretic Feature Selection.

Feature selection aims to select the smallest feature subset that yields the minimum generalization error. In the rich literature in feature selection, information theory-based approaches seek a subset of features such that the mutual information between the selected features and the class labels is maximized. Despite the simplicity of this objective, there still remain several open problems in optimization. These include, for example, the automatic determination of the optimal subset size (i.e., the number of features) or a stopping criterion if the greedy searching strategy is adopted. In this paper, we suggest two stopping criteria by just monitoring the conditional mutual information (CMI) among groups of variables. Using the recently developed multivariate matrix-based Rényi's α-entropy functional, which can be directly estimated from data samples, we showed that the CMI among groups of variables can be easily computed without any decomposition or approximation, hence making our criteria easy to implement and seamlessly integrated into any existing information theoretic feature selection methods with a greedy search strategy.

Applications of Information Theory in Solar and Space Physics.

Characterizing and modeling processes at the sun and space plasma in our solar system are difficult because the underlying physics is often complex, nonlinear, and not well understood. The drivers of a system are often nonlinearly correlated with one another, which makes it a challenge to understand the relative effects caused by each driver. However, entropy-based information theory can be a valuable tool that can be used to determine the information flow among various parameters, causalities, untangle the drivers, and provide observational constraints that can help guide the development of the theories and physics-based models. We review two examples of the applications of the information theoretic tools at the Sun and near-Earth space environment. In the first example, the solar wind drivers of radiation belt electrons are investigated using mutual information (MI), conditional mutual information (CMI), and transfer entropy (TE). As previously reported, radiation belt electron flux (Je) is anticorrelated with solar wind density (nsw) with a lag of 1 day. However, this lag time and anticorrelation can be attributed mainly to the Je(t + 2 days) correlation with solar wind velocity (Vsw)(t) and nsw(t + 1 day) anticorrelation with Vsw(t). Analyses of solar wind driving of the magnetosphere need to consider the large lag times, up to 3 days, in the (Vsw, nsw) anticorrelation. Using CMI to remove the effects of Vsw, the response of Je to nsw is 30% smaller and has a lag time <24 h, suggesting that the loss mechanism due to nsw or solar wind dynamic pressure has to start operating in <24 h. Nonstationarity in the system dynamics is investigated using windowed TE. The triangle distribution in Je(t + 2 days) vs. Vsw(t) can be better understood with TE. In the second example, the previously identified causal parameters of the solar cycle in the Babcock-Leighton type model such as the solar polar field, meridional flow, polar faculae (proxy for polar field), and flux emergence are investigated using TE. The transfer of information from the polar field to the sunspot number (SSN) peaks at lag times of 3-4 years. Both the flux emergence and the meridional flow contribute to the polar field, but at different time scales. The polar fields from at least the last 3 cycles contain information about SSN.

A statistical framework for neuroimaging data analysis based on mutual information estimated via a gaussian copula.

We begin by reviewing the statistical framework of information theory as applicable to neuroimaging data analysis. A major factor hindering wider adoption of this framework in neuroimaging is the difficulty of estimating information theoretic quantities in practice. We present a novel estimation technique that combines the statistical theory of copulas with the closed form solution for the entropy of Gaussian variables. This results in a general, computationally efficient, flexible, and robust multivariate statistical framework that provides effect sizes on a common meaningful scale, allows for unified treatment of discrete, continuous, unidimensional and multidimensional variables, and enables direct comparisons of representations from behavioral and brain responses across any recording modality. We validate the use of this estimate as a statistical test within a neuroimaging context, considering both discrete stimulus classes and continuous stimulus features. We also present examples of analyses facilitated by these developments, including application of multivariate analyses to MEG planar magnetic field gradients, and pairwise temporal interactions in evoked EEG responses. We show the benefit of considering the instantaneous temporal derivative together with the raw values of M/EEG signals as a multivariate response, how we can separately quantify modulations of amplitude and direction for vector quantities, and how we can measure the emergence of novel information over time in evoked responses. Open-source Matlab and Python code implementing the new methods accompanies this article. Hum Brain Mapp 38:1541-1573, 2017. © 2016 Wiley Periodicals, Inc.

Feature selection by optimizing a lower bound of conditional mutual information.

A unified framework is proposed to select features by optimizing computationally feasible approximations of high-dimensional conditional mutual information (CMI) between features and their associated class label under different assumptions. Under this unified framework, state-of-the-art information theory based feature selection algorithms are rederived, and a new algorithm is proposed to select features by optimizing a lower bound of the CMI with a weaker assumption than those adopted by existing methods. The new feature selection method integrates a plug-in component to distinguish redundant features from irrelevant ones for improving the feature selection robustness. Furthermore, a novel metric is proposed to evaluate feature selection methods based on simulated data. The proposed method has been compared with state-of-the-art feature selection methods based on the new evaluation metric and classification performance of classifiers built upon the selected features. The experiment results have demonstrated that the proposed method could achieve promising performance in a variety of feature selection problems.

Gene selection and cancer classification using interaction-based feature clustering and improved-binary Bat algorithm.

In high-dimensional gene expression data, selecting an optimal subset of genes is crucial for achieving high classification accuracy and reliable diagnosis of diseases. This paper proposes a two-stage hybrid model for gene selection based on clustering and a swarm intelligence algorithm to identify the most informative genes with high accuracy. First, a clustering-based multivariate filter approach is performed to explore the interactions between the features and eliminate any redundant or irrelevant ones. Then, by controlling for the problem of premature convergence in the binary Bat algorithm, the optimal gene subset is determined using different classifiers with the Monte Carlo cross-validation data partitioning model. The effectiveness of our proposed framework is evaluated using eight gene expression datasets, by comparison with other recently published algorithms in the literature. Experiments confirm that in seven out of eight datasets, the proposed method can achieve superior results in terms of classification accuracy and gene subset size. In particular, it achieves a classification accuracy of 100% in Lymphoma and Ovarian datasets and above 97.4% in the rest with a minimum number of genes. The results demonstrate that our proposed algorithm has the potential to solve the feature selection problem in different applications with high-dimensional datasets.

Permutation conditional mutual information to quantify TMS-evoked cortical connectivity in disorders of consciousness.

Objective.To improve the understanding and diagnostic accuracy of disorders of consciousness (DOC) by quantifying transcranial magnetic stimulation (TMS) evoked electroencephalography connectivity using permutation conditional mutual information (PCMI).Approach.PCMI can characterize the functional connectivity between different brain regions. This study employed PCMI to analyze TMS-evoked cortical connectivity (TEC) in 154 DOC patients and 16 normal controls, focusing on optimizing parameter selection for PCMI (Data length, Order length, Time delay). We compared short-range and long-range PCMI values across different consciousness states-unresponsive wakefulness syndrome (UWS), minimally conscious state (MCS), and normal (NOR)-and assessed various feature selection and classification techniques to distinguish these states.Main results.(1) PCMI can quantify TEC. We found optimal parameters to be Data length: 500 ms; Order: 3; Time delay: 6 ms. (2) TMS evoked potentials (TEPs) for NOR showed a rich response, while MCS patients showed only a few components, and UWS patients had almost no significant components. The values of PCMI connectivity metrics demonstrated its usefulness for measuring cortical connectivity evoked by TMS. From NOR to MCS to UWS, the number and strength of TEC decreased. Quantitative analysis revealed significant differences in the strength and number of TEC in the entire brain, local regions and inter-regions among different consciousness states. (3) A decision tree with feature selection by mutual information performed the best (balanced accuracy: 87.0% and accuracy: 83.5%). This model could accurately identify NOR (100.0%), but had lower identification accuracy for UWS (86.5%) and MCS (74.1%).Significance.The application of PCMI in measuring TMS-evoked connectivity provides a robust metric that enhances our ability to differentiate between various states of consciousness in DOC patients. This approach not only aids in clinical diagnosis but also contributes to the broader understanding of cortical connectivity and consciousness.

CNVs in 8q24.3 do not influence gene co-expression in breast cancer subtypes.

Gene co-expression networks are a useful tool in the study of interactions that have allowed the visualization and quantification of diverse phenomena, including the loss of co-expression over long distances in cancerous samples. This characteristic, which could be considered fundamental to cancer, has been widely reported in various types of tumors. Since copy number variations (CNVs) have previously been identified as causing multiple genetic diseases, and gene expression is linked to them, they have often been mentioned as a probable cause of loss of co-expression in cancerous networks. In order to carry out a comparative study of the validity of this statement, we took 477 protein-coding genes from chromosome 8, and the CNVs of 101 genes, also protein-coding, belonging to the 8q24.3 region, a cytoband that is particularly active in the appearance of breast cancer. We created CNVS-conditioned co-expression networks of each of the 101 genes in the 8q24.3 region using conditional mutual information. The study was carried out using the four molecular subtypes of breast cancer (Luminal A, Luminal B, Her2, and Basal), as well as a case corresponding to healthy samples. We observed that in all cancer cases, the measurement of the Kolmogorov-Smirnov statistic shows that there are no significant differences between one and other values of the CNVs for any case. Furthermore, the co-expression interactions are stronger in all cancer subtypes than in the control networks. However, the control network presents a homogeneously distributed set of co-expression interactions, while for cancer networks, the highest interactions are more confined to specific cytobands, in particular 8q24.3 and 8p21.3. With this approach, we demonstrate that despite copy number alterations in the 8q24 region being a common trait in breast cancer, the loss of long-distance co-expression in breast cancer is not determined by CNVs.

The Role of Copy Number Variants in Gene Co-Expression Patterns for Luminal B Breast Tumors.

Gene co-expression networks have become a usual approach to integrate the vast amounts of information coming from gene expression studies in cancer cohorts. The reprogramming of the gene regulatory control and the molecular pathways depending on such control are central to the characterization of the disease, aiming to unveil the consequences for cancer prognosis and therapeutics. There is, however, a multitude of factors which have been associated with anomalous control of gene expression in cancer. In the particular case of co-expression patterns, we have previously documented a phenomenon of loss of long distance co-expression in several cancer types, including breast cancer. Of the many potential factors that may contribute to this phenomenology, copy number variants (CNVs) have been often discussed. However, no systematic assessment of the role that CNVs may play in shaping gene co-expression patterns in breast cancer has been performed to date. For this reason we have decided to develop such analysis. In this study, we focus on using probabilistic modeling techniques to evaluate to what extent CNVs affect the phenomenon of long/short range co-expression in Luminal B breast tumors. We analyzed the co-expression patterns in chromosome 8, since it is known to be affected by amplifications/deletions during cancer development. We found that the CNVs pattern in chromosome 8 of Luminal B network does not alter the co-expression patterns significantly, which means that the co-expression program in this cancer phenotype is not determined by CNV structure. Additionally, we found that region 8q24.3 is highly dense in interactions, as well as region p21.3. The most connected genes in this network belong to those cytobands and are associated with several manifestations of cancer in different tissues. Interestingly, among the most connected genes, we found MAF1 and POLR3D, which may constitute an axis of regulation of gene transcription, in particular for non-coding RNA species. We believe that by advancing on our knowledge of the molecular mechanisms behind gene regulation in cancer, we will be better equipped, not only to understand tumor biology, but also to broaden the scope of diagnostic, prognostic and therapeutic interventions to ultimately benefit oncologic patients.

Multi-dimensional conditional mutual information with application on the EEG signal analysis for spatial cognitive ability evaluation.

This study aims to explore an effective method to evaluate spatial cognitive ability, which can effectively extract and classify the feature of EEG signals collected from subjects participating in the virtual reality (VR) environment; and evaluate the training effect objectively and quantitatively to ensure the objectivity and accuracy of spatial cognition evaluation, according to the classification results. Therefore, a multi-dimensional conditional mutual information (MCMI) method is proposed, which could calculate the coupling strength of two channels considering the influence of other channels. The coupled characteristics of the multi-frequency combination were transformed into multi-spectral images, and the image data were classified employing the convolutional neural networks (CNN) model. The experimental results showed that the multi-spectral image transform features based on MCMI are better in classification than other methods, and among the classification results of six band combinations, the best classification accuracy of Beta1-Beta2-Gamma combination is 98.3%. The MCMI characteristics on the Beta1-Beta2-Gamma band combination can be a biological marker for the evaluation of spatial cognition. The proposed feature extraction method based on MCMI provides a new perspective for spatial cognitive ability assessment and analysis.

Deep-layer motif method for estimating information flow between EEG signals.

Accurate identification for the information flow between epileptic seizure signals is the key to construct the directional epileptic brain network which can be used to localize epileptic focus. In this paper, our concern is on how to improve the direction identification of information flow and also investigate how it can be cut off or weakened. In view of this, we propose the deep-layer motif method. Based on the directional index (DI) estimation using permutation conditional mutual information, the effectiveness of the proposed deep-layer motif method is numerically assessed with the coupled mass neural model. Furthermore, we investigate the robustness of this method in considering the interference of autaptic coupling, time delay and short-term plasticity. Results show that compared to the simple 1-layer motif method, the 2nd- and 3rd-layer motif methods have the dominant enhancement effects for the direction identification. In particular, deep-layer motif method possesses good anti-jamming performance and good robustness in calculating DI. In addition, we investigate the effect of deep brain stimulation (DBS) on the information flow. It is found that this deep-layer motif method is still superior to the single-layer motif method in direction identification and is robust to weak DBS. However, the high-frequency strong DBS can effectively decrease the DI suggesting the weakened information flow. These results may give new insights into the seizure detection and control.