Data-Driven Identification of Stroke through Machine Learning Applied to Complexity Metrics in Multimodal Electromyography and Kinematics.

A stroke represents a significant medical condition characterized by the sudden interruption of blood flow to the brain, leading to cellular damage or death. The impact of stroke on individuals can vary from mild impairments to severe disability. Treatment for stroke often focuses on gait rehabilitation. Notably, assessing muscle activation and kinematics patterns using electromyography (EMG) and stereophotogrammetry, respectively, during walking can provide information regarding pathological gait conditions. The concurrent measurement of EMG and kinematics can help in understanding disfunction in the contribution of specific muscles to different phases of gait. To this aim, complexity metrics (e.g., sample entropy; approximate entropy; spectral entropy) applied to EMG and kinematics have been demonstrated to be effective in identifying abnormal conditions. Moreover, the conditional entropy between EMG and kinematics can identify the relationship between gait data and muscle activation patterns. This study aims to utilize several machine learning classifiers to distinguish individuals with stroke from healthy controls based on kinematics and EMG complexity measures. The cubic support vector machine applied to EMG metrics delivered the best classification results reaching 99.85% of accuracy. This method could assist clinicians in monitoring the recovery of motor impairments for stroke patients.

Cervical lesion image enhancement based on conditional entropy generative adversarial network framework.

Early screening and diagnosis of cervical precancerous lesions are very important to prevent cervical cancer. High-quality colposcopy images will help doctors make faster and more accurate diagnoses. To tackle the problem of low image quality caused by complex interference during colposcopy operation, this paper proposed a conditional entropy generative adversarial networks framework for image enhancement. A decomposition network based on Retinex theory is constructed to obtain the reflection images of the low-quality images, then the conditional generative adversarial network is used as the enhancement network. The low-quality images and the decomposed reflection images are both input the enhancement network for training, and the conditional entropy distance is used as a part of the loss of the conditional generative adversarial network to alleviate the over-fitting problem during the training process. The test results show that compared with published methods, the proposed method of this paper can significantly improve the image quality, and can enhance the colposcopy image while retaining image details.

Unified model-free interaction screening via CV-entropy filter.

For many practical high-dimensional problems, interactions have been increasingly found to play important roles beyond main effects. A representative example is gene-gene interaction. Joint analysis, which analyzes all interactions and main effects in a single model, can be seriously challenged by high dimensionality. For high-dimensional data analysis in general, marginal screening has been established as effective for reducing computational cost, increasing stability, and improving estimation/selection performance. Most of the existing marginal screening methods are designed for the analysis of main effects only. The existing screening methods for interaction analysis are often limited by making stringent model assumptions, lacking robustness, and/or requiring predictors to be continuous (and hence lacking flexibility). A unified marginal screening approach tailored to interaction analysis is developed, which can be applied to regression, classification, and survival analysis. Predictors are allowed to be continuous and discrete. The proposed approach is built on Coefficient of Variation (CV) filters based on information entropy. Statistical properties are rigorously established. It is shown that the CV filters are almost insensitive to the distribution tails of predictors, correlation structure among predictors, and sparsity level of signals. An efficient two-stage algorithm is developed to make the proposed approach scalable to ultrahigh-dimensional data. Simulations and the analysis of TCGA LUAD data further establish the practical superiority of the proposed approach.

Conditional Entropic Approach to Nonequilibrium Complex Systems with Weak Fluctuation Correlation.

A conditional entropic approach is discussed for nonequilibrium complex systems with a weak correlation between spatiotemporally fluctuating quantities on a large time scale. The weak correlation is found to constitute the fluctuation distribution that maximizes the entropy associated with the conditional fluctuations. The approach is illustrated in diffusion phenomenon of proteins inside bacteria. A further possible illustration is also presented for membraneless organelles in embryos and beads in cell extracts, which share common natures of fluctuations in their diffusion.

Ordinal Conditional Entropy Displays Reveal Intrinsic Characteristics of the Rosenberg Self-Esteem Scale.

Individual subjects' ratings neither are metric nor have homogeneous meanings, consequently digital- labeled collections of subjects' ratings are intrinsically ordinal and categorical. However, in these situations, the literature privileges the use of measures conceived for numerical data. In this paper, we discuss the exploratory theme of employing conditional entropy to measure degrees of uncertainty in responding to self-rating questions and that of displaying the computed entropies along the ordinal axis for visible pattern recognition. We apply this theme to the study of an online dataset, which contains responses to the Rosenberg Self-Esteem Scale. We report three major findings. First, at the fine scale level, the resultant multiple ordinal-display of response-vs-covariate entropy measures reveals that the subjects on both extreme labels (high self-esteem and low self-esteem) show distinct degrees of uncertainty. Secondly, at the global scale level, in responding to positively posed questions, the degree of uncertainty decreases for increasing levels of self-esteem, while, in responding to negative questions, the degree of uncertainty increases. Thirdly, such entropy-based computed patterns are preserved across age groups. We provide a set of tools developed in R that are ready to implement for the analysis of rating data and for exploring pattern-based knowledge in related research.

Feasibility of Ultra-Short-Term Analysis of Heart Rate and Systolic Arterial Pressure Variability at Rest and during Stress via Time-Domain and Entropy-Based Measures.

Heart Rate Variability (HRV) and Blood Pressure Variability (BPV) are widely employed tools for characterizing the complex behavior of cardiovascular dynamics. Usually, HRV and BPV analyses are carried out through short-term (ST) measurements, which exploit ~five-minute-long recordings. Recent research efforts are focused on reducing the time series length, assessing whether and to what extent Ultra-Short-Term (UST) analysis is capable of extracting information about cardiovascular variability from very short recordings. In this work, we compare ST and UST measures computed on electrocardiographic R-R intervals and systolic arterial pressure time series obtained at rest and during both postural and mental stress. Standard time-domain indices are computed, together with entropy-based measures able to assess the regularity and complexity of cardiovascular dynamics, on time series lasting down to 60 samples, employing either a faster linear parametric estimator or a more reliable but time-consuming model-free method based on nearest neighbor estimates. Our results are evidence that shorter time series down to 120 samples still exhibit an acceptable agreement with the ST reference and can also be exploited to discriminate between stress and rest. Moreover, despite neglecting nonlinearities inherent to short-term cardiovascular dynamics, the faster linear estimator is still capable of detecting differences among the conditions, thus resulting in its suitability to be implemented on wearable devices.

Unraveling Hidden Major Factors by Breaking Heterogeneity into Homogeneous Parts within Many-System Problems.

For a large ensemble of complex systems, a Many-System Problem (MSP) studies how heterogeneity constrains and hides structural mechanisms, and how to uncover and reveal hidden major factors from homogeneous parts. All member systems in an MSP share common governing principles of dynamics, but differ in idiosyncratic characteristics. A typical dynamic is found underlying response features with respect to covariate features of quantitative or qualitative data types. Neither all-system-as-one-whole nor individual system-specific functional structures are assumed in such response-vs-covariate (Re-Co) dynamics. We developed a computational protocol for identifying various collections of major factors of various orders underlying Re-Co dynamics. We first demonstrate the immanent effects of heterogeneity among member systems, which constrain compositions of major factors and even hide essential ones. Secondly, we show that fuller collections of major factors are discovered by breaking heterogeneity into many homogeneous parts. This process further realizes Anderson's "More is Different" phenomenon. We employ the categorical nature of all features and develop a Categorical Exploratory Data Analysis (CEDA)-based major factor selection protocol. Information theoretical measurements-conditional mutual information and entropy-are heavily used in two selection criteria: C1-confirmable and C2-irreplaceable. All conditional entropies are evaluated through contingency tables with algorithmically computed reliability against the finite sample phenomenon. We study one artificially designed MSP and then two real collectives of Major League Baseball (MLB) pitching dynamics with 62 slider pitchers and 199 fastball pitchers, respectively. Finally, our MSP data analyzing techniques are applied to resolve a scientific issue related to the Rosenberg Self-Esteem Scale.

Belavkin-Staszewski Relative Entropy, Conditional Entropy, and Mutual Information.

Belavkin-Staszewski relative entropy can naturally characterize the effects of the possible noncommutativity of quantum states. In this paper, two new conditional entropy terms and four new mutual information terms are first defined by replacing quantum relative entropy with Belavkin-Staszewski relative entropy. Next, their basic properties are investigated, especially in classical-quantum settings. In particular, we show the weak concavity of the Belavkin-Staszewski conditional entropy and obtain the chain rule for the Belavkin-Staszewski mutual information. Finally, the subadditivity of the Belavkin-Staszewski relative entropy is established, i.e., the Belavkin-Staszewski relative entropy of a joint system is less than the sum of that of its corresponding subsystems with the help of some multiplicative and additive factors. Meanwhile, we also provide a certain subadditivity of the geometric Rényi relative entropy.

Life Worth Mentioning: Complexity in Life-Like Cellular Automata.

Cellular automata (CA) have been lauded for their ability to generate complex global patterns from simple local rules. The late English mathematician, John Horton Conway, developed his illustrious Game of Life (Life) CA in 1970, which has since remained one of the most quintessential CA constructions-capable of producing a myriad of complex dynamic patterns and computational universality. Life and several other Life-like rules have been classified in the same group of aesthetically and dynamically interesting CA rules characterized by their complex behaviors. However, a rigorous quantitative comparison among similarly classified Life-like rules has not yet been fully established. Here we show that Life is capable of maintaining as much complexity as similar rules while remaining the most parsimonious. In other words, Life contains a consistent amount of complexity throughout its evolution, with the least number of rule conditions compared to other Life-like rules. We also found that the complexity of higher density Life-like rules, which themselves contain the Life rule as a subset, form a distinct concave density-complexity relationship whereby an optimal complexity candidate is proposed. Our results also support the notion that Life functions as the basic ingredient for cultivating the balance between structure and randomness to maintain complexity in 2D CA for low- and high-density regimes, especially over many iterations. This work highlights the genius of John Horton Conway and serves as a testament to his timeless marvel, which is referred to simply as: Life.

Conditional Entropy: A Potential Digital Marker for Stress.

Recent decades have witnessed a substantial progress in the utilization of brain activity for the identification of stress digital markers. In particular, the success of entropic measures for this purpose is very appealing, considering (1) their suitability for capturing both linear and non-linear characteristics of brain activity recordings and (2) their direct association with the brain signal variability. These findings rely on external stimuli to induce the brain stress response. On the other hand, research suggests that the use of different types of experimentally induced psychological and physical stressors could potentially yield differential impacts on the brain response to stress and therefore should be dissociated from more general patterns. The present study takes a step toward addressing this issue by introducing conditional entropy (CE) as a potential electroencephalography (EEG)-based resting-state digital marker of stress. For this purpose, we use the resting-state multi-channel EEG recordings of 20 individuals whose responses to stress-related questionnaires show significantly higher and lower level of stress. Through the application of representational similarity analysis (RSA) and K-nearest-neighbor (KNN) classification, we verify the potential that the use of CE can offer to the solution concept of finding an effective digital marker for stress.

Categorical Nature of Major Factor Selection via Information Theoretic Measurements.

Without assuming any functional or distributional structure, we select collections of major factors embedded within response-versus-covariate (Re-Co) dynamics via selection criteria [C1: confirmable] and [C2: irrepaceable], which are based on information theoretic measurements. The two criteria are constructed based on the computing paradigm called Categorical Exploratory Data Analysis (CEDA) and linked to Wiener-Granger causality. All the information theoretical measurements, including conditional mutual information and entropy, are evaluated through the contingency table platform, which primarily rests on the categorical nature within all involved features of any data types: quantitative or qualitative. Our selection task identifies one chief collection, together with several secondary collections of major factors of various orders underlying the targeted Re-Co dynamics. Each selected collection is checked with algorithmically computed reliability against the finite sample phenomenon, and so is each member's major factor individually. The developments of our selection protocol are illustrated in detail through two experimental examples: a simple one and a complex one. We then apply this protocol on two data sets pertaining to two somewhat related but distinct pitching dynamics of two pitch types: slider and fastball. In particular, we refer to a specific Major League Baseball (MLB) pitcher and we consider data of multiple seasons.

Mimicking Complexity of Structured Data Matrix's Information Content: Categorical Exploratory Data Analysis.

We develop Categorical Exploratory Data Analysis (CEDA) with mimicking to explore and exhibit the complexity of information content that is contained within any data matrix: categorical, discrete, or continuous. Such complexity is shown through visible and explainable serial multiscale structural dependency with heterogeneity. CEDA is developed upon all features' categorical nature via histogram and it is guided by all features' associative patterns (order-2 dependence) in a mutual conditional entropy matrix. Higher-order structural dependency of k(≥3) features is exhibited through block patterns within heatmaps that are constructed by permuting contingency-kD-lattices of counts. By growing k, the resultant heatmap series contains global and large scales of structural dependency that constitute the data matrix's information content. When involving continuous features, the principal component analysis (PCA) extracts fine-scale information content from each block in the final heatmap. Our mimicking protocol coherently simulates this heatmap series by preserving global-to-fine scales structural dependency. Upon every step of mimicking process, each accepted simulated heatmap is subject to constraints with respect to all of the reliable observed categorical patterns. For reliability and robustness in sciences, CEDA with mimicking enhances data visualization by revealing deterministic and stochastic structures within each scale-specific structural dependency. For inferences in Machine Learning (ML) and Statistics, it clarifies, upon which scales, which covariate feature-groups have major-vs.-minor predictive powers on response features. For the social justice of Artificial Intelligence (AI) products, it checks whether a data matrix incompletely prescribes the targeted system.

Complexity analysis of heart rate variability in chronic obstructive pulmonary disease: relationship with severity and symptoms.

PURPOSE: Chronic obstructive pulmonary disease (COPD) negatively impacts autonomic control of the heart rate, as assessed by time and frequency domains of heart rate variability (HRV) analysis. However, it is unknown whether symbolic dynamic analysis may identify cardiac autonomic impairment, and whether such nonlinear indices may be associated with disease severity, prognostic markers, perceived dyspnea and functional capacity in patients with COPD. The current study assessed cardiac autonomic modulation by symbolic analysis of HRV in patients with COPD compared with healthy controls. METHODS: We recruited 54 COPD patients and 20 healthy controls. The interval between two successive R-wave peaks was calculated in the resting supine position. HRV was analyzed using symbolic markers and Shannon entropy (SE). The six-minute walk test (6MWT) was applied in a 30-m corridor. RESULTS: We found a lower 6MWT distance in patients with COPD compared with healthy controls (p < 0.05). We found increased SE and decreased percentage of no variation patterns (0V%) in COPD patients compared with the control group (p = 0.001). Significant correlations were found between the percentage of one variation pattern (1V%) and the Medical Research Council dyspnea scale (r = 0.38, p = 0.01), BODE index (r = 0.38, p = 0.01), forced expiratory volume in the first second (FEV1) [L] (r = -0.44, p = 0.003) and FEV1 [%] (r = -0.35, p = 0.02). It was found that SE was inversely associated with 0V% (r = -0.87, p < 0.0001). CONCLUSION: COPD patients present with depressed sympathetic modulation of HR and higher SE compared with healthy controls. This increased irregularity was inversely associated with 0V%. These results suggested that COPD patients seem to have a cardiac control shifted towards a parasympathetic predominance compared with controls. Symbolic dynamic and complexity index of HRV are related to disease severity, symptoms and functional impairment in these patients.

New Service Virtualisation Approach to Generate the Categorical Fields in the Service Response.

Software services communicate with different requisite services over the computer network to accomplish their tasks. The requisite services may not be readily available to test a specific service. Thus, service virtualisation has been proposed as an industry solution to ensure availability of the interactive behaviour of the requisite services. However, the existing techniques of virtualisation cannot satisfy the required accuracy or time constraints to keep up with the competitive business world. These constraints sacrifices quality and testing coverage, thereby delaying the delivery of software. We proposed a novel technique to improve the accuracy of the existing service virtualisation solutions without sacrificing time. This method generates the service response and predicts categorical fields in virtualised responses, extending existing research with lower complexity and higher accuracy. The proposed service virtualisation approach uses conditional entropy to identify the fields that can be used to drive the value of each categorical field based on the historical messages. Then, it uses joint probability distribution to find the best values for the categorical fields. The experimental evaluation illustrates that the proposed approach can generate responses with the required fields and accurate values for categorical fields over four data sets with stateful nature.

Ordinal Pattern Based Entropies and the Kolmogorov-Sinai Entropy: An Update.

Different authors have shown strong relationships between ordinal pattern based entropies and the Kolmogorov-Sinai entropy, including equality of the latter one and the permutation entropy, the whole picture is however far from being complete. This paper is updating the picture by summarizing some results and discussing some mainly combinatorial aspects behind the dependence of Kolmogorov-Sinai entropy from ordinal pattern distributions on a theoretical level. The paper is more than a review paper. A new statement concerning the conditional permutation entropy will be given as well as a new proof for the fact that the permutation entropy is an upper bound for the Kolmogorov-Sinai entropy. As a main result, general conditions for the permutation entropy being a lower bound for the Kolmogorov-Sinai entropy will be stated. Additionally, a previously introduced method to analyze the relationship between permutation and Kolmogorov-Sinai entropies as well as its limitations will be investigated.

Weighted Mutual Information for Aggregated Kernel Clustering.

Background: A common task in machine learning is clustering data into different groups based on similarities. Clustering methods can be divided in two groups: linear and nonlinear. A commonly used linear clustering method is K-means. Its extension, kernel K-means, is a non-linear technique that utilizes a kernel function to project the data to a higher dimensional space. The projected data will then be clustered in different groups. Different kernels do not perform similarly when they are applied to different datasets. Methods: A kernel function might be relevant for one application but perform poorly to project data for another application. In turn choosing the right kernel for an arbitrary dataset is a challenging task. To address this challenge, a potential approach is aggregating the clustering results to obtain an impartial clustering result regardless of the selected kernel function. To this end, the main challenge is how to aggregate the clustering results. A potential solution is to combine the clustering results using a weight function. In this work, we introduce Weighted Mutual Information (WMI) for calculating the weights for different clustering methods based on their performance to combine the results. The performance of each method is evaluated using a training set with known labels. Results: We applied the proposed Weighted Mutual Information to four data sets that cannot be linearly separated. We also tested the method in different noise conditions. Conclusions: Our results show that the proposed Weighted Mutual Information method is impartial, does not rely on a single kernel, and performs better than each individual kernel specially in high noise.

Multi-Harmonic Source Localization Based on Sparse Component Analysis and Minimum Conditional Entropy.

Aiming at the fact that the independent component analysis algorithm requires more measurement points and cannot solve the problem of harmonic source location under underdetermined conditions, a new method based on sparse component analysis and minimum conditional entropy for identifying multiple harmonic source locations in a distribution system is proposed. Under the condition that the network impedance is unknown and the number of harmonic sources is undetermined, the measurement node configuration algorithm selects the node position to make the separated harmonic current more accurate. Then, using the harmonic voltage data of the selected node as the input, the sparse component analysis is used to solve the harmonic current waveform under underdetermination. Finally, the conditional entropy between the harmonic current and the system node is calculated, and the node corresponding to the minimum condition entropy is the location of the harmonic source. In order to verify the effectiveness and accuracy of the proposed method, the simulation was performed in an IEEE 14-node system. Moreover, compared with the results of independent component analysis algorithms. Simulation results verify the correctness and effectiveness of the proposed algorithm.

A Novel Recognition Strategy for Epilepsy EEG Signals Based on Conditional Entropy of Ordinal Patterns.

Epilepsy is one of the most ordinary neuropathic illnesses, and electroencephalogram (EEG) is the essential method for recording various brain rhythm activities due to its high temporal resolution. The conditional entropy of ordinal patterns (CEOP) is known to be fast and easy to implement, which can effectively measure the irregularity of the physiological signals. The present work aims to apply the CEOP to analyze the complexity characteristics of the EEG signals and recognize the epilepsy EEG signals. We discuss the parameter selection and the performance analysis of the CEOP based on the neural mass model. The CEOP is applied to the real EEG database of Bonn epilepsy for identification. The results show that the CEOP is an excellent metrics for the analysis and recognition of epileptic EEG signals. The differences of the CEOP in normal and epileptic brain states suggest that the CEOP could be a judgment tool for the diagnosis of the epileptic seizure.

An Efficient, Parallelized Algorithm for Optimal Conditional Entropy-Based Feature Selection.

In Machine Learning, feature selection is an important step in classifier design. It consists of finding a subset of features that is optimum for a given cost function. One possibility to solve feature selection is to organize all possible feature subsets into a Boolean lattice and to exploit the fact that the costs of chains in that lattice describe U-shaped curves. Minimization of such cost function is known as the U-curve problem. Recently, a study proposed U-Curve Search (UCS), an optimal algorithm for that problem, which was successfully used for feature selection. However, despite of the algorithm optimality, the UCS required time in computational assays was exponential on the number of features. Here, we report that such scalability issue arises due to the fact that the U-curve problem is NP-hard. In the sequence, we introduce the Parallel U-Curve Search (PUCS), a new algorithm for the U-curve problem. In PUCS, we present a novel way to partition the search space into smaller Boolean lattices, thus rendering the algorithm highly parallelizable. We also provide computational assays with both synthetic data and Machine Learning datasets, where the PUCS performance was assessed against UCS and other golden standard algorithms in feature selection.

Sharp Second-Order Pointwise Asymptotics for Lossless Compression with Side Information.

The problem of determining the best achievable performance of arbitrary lossless compression algorithms is examined, when correlated side information is available at both the encoder and decoder. For arbitrary source-side information pairs, the conditional information density is shown to provide a sharp asymptotic lower bound for the description lengths achieved by an arbitrary sequence of compressors. This implies that for ergodic source-side information pairs, the conditional entropy rate is the best achievable asymptotic lower bound to the rate, not just in expectation but with probability one. Under appropriate mixing conditions, a central limit theorem and a law of the iterated logarithm are proved, describing the inevitable fluctuations of the second-order asymptotically best possible rate. An idealised version of Lempel-Ziv coding with side information is shown to be universally first- and second-order asymptotically optimal, under the same conditions. These results are in part based on a new almost-sure invariance principle for the conditional information density, which may be of independent interest.