Maximum-entropy-based metrics for quantifying critical dynamics in spiking neuron data.

An important working hypothesis to investigate brain activity is whether it operates in a critical regime. Recently, maximum-entropy phenomenological models have emerged as an alternative way of identifying critical behavior in neuronal data sets. In the present paper, we investigate the signatures of criticality from a firing rate-based maximum-entropy approach on data sets generated by computational models, and we compare them to experimental results. We found that the maximum entropy approach consistently identifies critical behavior around the phase transition in models and rules out criticality in models without phase transition. The maximum-entropy-model results are compatible with results for cortical data from urethane-anesthetized rats data, providing further support for criticality in the brain.

Symbolic information approach applied to human intracranial data to characterize and distinguish different congnitive processes.

How the human brain processes information during different cognitive tasks is one of the greatest questions in contemporary neuroscience. Understanding the statistical properties of brain signals during specific activities is one promising way to address this question. Here we analyze freely available data from implanted electrocorticography (ECoG) in five human subjects during two different cognitive tasks in the light of information theory quantifiers ideas. We employ a symbolic information approach to determine the probability distribution function associated with the time series from different cortical areas. Then we utilize these probabilities to calculate the associated Shannon entropy and a statistical complexity measure based on the disequilibrium between the actual time series and one with a uniform probability distribution function. We show that an Euclidian distance in the complexity-entropy plane and an asymmetry index for complexity are useful for comparing the two conditions. We show that our method can distinguish visual search epochs from blank screen intervals in different electrodes and patients. By using a multiscale approach and embedding time delays to downsample the data, we find important timescales in which the relevant information is being processed. We also determine cortical regions and time intervals along the 2-s-long trials that present more pronounced differences between the two cognitive tasks. Finally, we show that the method is useful to distinguish cognitive processes using brain activity on a trial-by-trial basis.

Changes in physiological signal entropy in patients with obstructive sleep apnoea: a systematic review.

Background and Objective.Obstructive sleep apnoea (OSA) affects an estimated 936 million people worldwide, yet only 15% receive a definitive diagnosis. Diagnosis of OSA poses challenges due to the dynamic nature of physiological signals such as oxygen saturation (SpO2) and heart rate variability (HRV). Linear analysis methods may not fully capture the irregularities present in these signals. The application of entropy of routine physiological signals offers a promising method to better measure variabilities in dynamic biological data. This review aims to explore entropy changes in physiological signals among individuals with OSA.Approach.Keyword and title searches were performed on Medline, Embase, Scopus, and CINAHL databases. Studies had to analyse physiological signals in OSA using entropy. Quality assessment used the Newcastle-Ottawa Scale. Evidence was qualitatively synthesised, considering entropy signals, entropy type, and time-series length.Main results.Twenty-two studies were included. Multiple physiological signals related to OSA, including SpO2, HRV, and the oxygen desaturation index (ODI), have been investigated using entropy. Results revealed a significant decrease in HRV entropy in those with OSA compared to control groups. Conversely, SpO2and ODI entropy values were increased in OSA. Despite variations in entropy types, time scales, and data extraction devices, studies using receiver operating characteristic curves demonstrated a high discriminative accuracy (>80% AUC) in distinguishing OSA patients from control groups.Significance. This review highlights the potential of SpO2entropy analysis in developing new diagnostic indices for patients with OSA. Further investigation is needed before applying this technique clinically.

Elucidating the link between binding statistics and Shannon information in biological networks.

The response of a biological network to ligand binding is of crucial importance for regulatory control in various cellular biophysical processes that is achieved with information transmission through the different ligand-bound states of such networks. In this work, we address a vital issue regarding the link between the information content of such network states and the experimentally measurable binding statistics. Several fundamental networks of cooperative ligand binding, with the bound states being adjacent in time only and in both space and time, are considered for this purpose using the chemical master equation approach. To express the binding characteristics in the language of information, a quantity denoted as differential information index is employed based on the Shannon information. The index, determined for the whole network, follows a linear relationship with (logarithmic) ligand concentration with a slope equal to the size of the system. On the other hand, the variation of Shannon information associated with the individual network states and the logarithmic sensitivity of its slope are shown to have generic forms related to the average binding number and variance, respectively, the latter yielding the Hill slope, the phenomenological measure of cooperativity. Furthermore, the variation of Shannon information entropy, the average of Shannon information, is also shown to be related to the average binding.

Space-time cluster analysis and maximum entropy modeling of Peste des petits ruminants (PPR) in Türkiye.

Peste des petits ruminants (PPR) is an economically important highly serious transboundary disease that mainly occurs in small ruminants such as sheep and goats. The aim of this study was to identify the probability of risk and and space-time clusters of Peste des Petits Ruminants (PPR) in Türkiye. The occurrence of PPR in Türkiye from 2017 to 2019 was investigated in this study using spatial analysis based on geographic information system (GIS). Between these dates, it was determined that 337 outbreaks and 18,467 cases. The highest number of outbreaks were detected in the Central Anatolia region. It was determined that PPR is seen more intensely in sheep compared to goats in Türkiye. In this study, 34 environmental variables (19 bioclimatic, 12 precipitation, altitude and small livestock density variables) were used to explore the environmental influences on PPR outbreak by maximum entropy modeling (Maxent). The clusters of PPR in Türkiye were identified using the retrospective space-time scan data that were computed using the space-time permutation model. A PPR prediction model was created using data on PPR outbreaks combination with environmental variables. Nineteen significant (p < 0.001) space-time clusters were determined. It was discovered that the variables altitude, sheep density, precipitation in june, and average temperature in the warmest season made important contributions to the model and the PPR outbreak may be strongly related with these variables. In this study, PPR in Türkiye has been characterized significantly spatio-temporal and enviromental factors. In this context, the disease pattern and obtained these findings will contribute to policymakers in the prevention and control of the disease.

Assessment of alternative metrics in the application of infrared thermography to detect muscle damage in sports.

Objective.The association between muscle damage and skin temperature is controversial. We hypothesize that including metrics that are more sensitive to individual responses by considering variability and regions representative of higher temperature could influence skin temperature outcomes. Here, the objective of the study was to determine whether using alternative metrics (TMAX, entropy, and pixelgraphy) leads to different results than mean, maximum, minimum, and standard deviation (SD) skin temperature when addressing muscle damage using infrared thermography.Approach.Thermal images from four previous investigations measuring skin temperature before and after muscle damage in the anterior thigh and the posterior lower leg were used. The TMAX, entropy, and pixelgraphy (percentage of pixels above 33 °C) metrics were applied.Main results.On 48 h after running a marathon or half-marathon, no differences were found in skin temperature when applying any metric. Mean, minimum, maximum, TMAX, and pixelgraphy were lower 48 h after than at basal condition following quadriceps muscle damage (p< 0.05). Maximum skin temperature and pixelgraphy were lower 48 h after than the basal condition following muscle damage to the triceps sural (p< 0.05). Overall, TMAX strongly correlated with mean (r= 0.85) and maximum temperatures (r= 0.99) and moderately with minimum (r= 0.66) and pixelgraphy parameter (r= 0.64). Entropy strongly correlates with SD (r= 0.94) and inversely moderately with minimum temperature (r= -0.53). The pixelgraphy moderately correlated with mean (r= 0.68), maximum (r= 0.62), minimum (r= 0.58), and TMAX (r= 0.64).Significance.Using alternative metrics does not change skin temperature outcomes following muscle damage of lower extremity muscle groups.

Using mindfulness-based intervention to promote executive function in young children: a multivariable and multiscale sample entropy study.

Early childhood marks a pivotal period in the maturation of executive function, the cognitive ability to consciously regulate actions and thoughts. Mindfulness-based interventions have shown promise in bolstering executive function in children. This study used the functional near-infrared spectroscopy technique to explore the impact of mindfulness-based training on young children. Brain imaging data were collected from 68 children (41 boys, aged 61.8 ± 10.7 months) who were randomly assigned to either an intervention group (N = 37, aged 60.03 ± 11.14 months) or a control group (N = 31, aged 59.99 ± 10.89 months). Multivariate and multiscale sample entropy analyses were used. The results showed that: (1) brain complexity was reduced in the intervention group after receiving the mindfulness-based intervention in all three executive function tasks (ps < 0.05), indicating a more efficient neural processing mechanism after the intervention; (2) difference comparisons between the intervention and control groups showed significant differences in relevant brain regions during cognitive shifting (left dorsolateral prefrontal cortex and medial prefrontal cortex) and working memory tasks (left dorsolateral prefrontal cortex), which corroborates with improved behavioral results in the intervention group (Z = -3.674, P < 0.001 for cognitive shifting; Z = 2.594, P < 0.01 for working memory). These findings improve our understanding of early brain development in young children and highlight the neural mechanisms by which mindfulness-based interventions affect executive function. Implications for early intervention to promote young children's brain development are also addressed.

The impact of controlled breathing on autonomic nervous system modulation: analysis using phase-rectified signal averaging, entropy and heart rate variability.

Objective.The present study investigated how breathing stimuli affect both non-linear and linear metrics of the autonomic nervous system (ANS).Approach.The analysed dataset consisted of 70 young, healthy volunteers, in whom arterial blood pressure (ABP) was measured noninvasively during 5 min sessions of controlled breathing at three different frequencies: 6, 10 and 15 breaths min-1. CO2concentration and respiratory rate were continuously monitored throughout the controlled breathing sessions. The ANS was characterized using non-linear methods, including phase-rectified signal averaging (PRSA) for estimating heart acceleration and deceleration capacity (AC, DC), multiscale entropy, approximate entropy, sample entropy, and fuzzy entropy, as well as time and frequency-domain measures (low frequency, LF; high-frequency, HF; total power, TP) of heart rate variability (HRV).Main results.Higher breathing rates resulted in a significant decrease in end-tidal CO2concentration (p< 0.001), accompanied by increases in both ABP (p <0.001) and heart rate (HR,p <0.001). A strong, linear decline in AC and DC (p <0.001 for both) was observed with increasing breathing rate. All entropy metrics increased with breathing frequency (p <0.001). In the time-domain, HRV metrics significantly decreased with breathing frequency (p <0.01 for all). In the frequency-domain, HRV LF and HRV HF decreased (p= 0.038 andp= 0.040, respectively), although these changes were modest. There was no significant change in HRV TP with breathing frequencies.Significance.Alterations in CO2levels, a potent chemoreceptor trigger, and changes in HR most likely modulate ANS metrics. Non-linear PRSA and entropy appear to be more sensitive to breathing stimuli compared to frequency-dependent HRV metrics. Further research involving a larger cohort of healthy subjects is needed to validate our observations.

Thermodynamics-inspired explanations of artificial intelligence.

In recent years, predictive machine learning models have gained prominence across various scientific domains. However, their black-box nature necessitates establishing trust in them before accepting their predictions as accurate. One promising strategy involves employing explanation techniques that elucidate the rationale behind a model's predictions in a way that humans can understand. However, assessing the degree of human interpretability of these explanations is a nontrivial challenge. In this work, we introduce interpretation entropy as a universal solution for evaluating the human interpretability of any linear model. Using this concept and drawing inspiration from classical thermodynamics, we present Thermodynamics-inspired Explainable Representations of AI and other black-box Paradigms, a method for generating optimally human-interpretable explanations in a model-agnostic manner. We demonstrate the wide-ranging applicability of this method by explaining predictions from various black-box model architectures across diverse domains, including molecular simulations, text, and image classification.

Dynamics of gas exchange and heart rate signal entropy in standard cardiopulmonary exercise testing during critical periods of growth and development.

Standard cardiopulmonary exercise testing (CPET) produces a rich dataset but its current analysis is often limited to a few derived variables such as maximal or peak oxygen uptake (V̇O2). We tested whether breath-by-breath CPET data could be used to determine sample entropy (SampEn) in 81 healthy children and adolescents (age 7-18 years old, equal sex distribution). To overcome challenges of the relatively small time-series CPET data size and its nonstationarity, we developed a Python algorithm for short-duration physiological signals. Comparing pre- and post-ventilatory threshold (VT1) CPET phases, we found: (1) SampEn decreased by 9.46% for V̇O2 and 5.01% for V̇CO2 (p < 0.05), in the younger, early-pubertal participants; and (2) HR SampEn fell substantially by 70.8% in the younger and 77.5% in the older participants (p < 0.001). Across all ages, females exhibited greater HR SampEn than males during both pre- and post VT1 CPET phases by 14.10% and 23.79%, respectively, p < 0.01. In females, late-pubertal had 17.6% lower HR SampEn compared to early-pubertal participants (p < 0.05). Breath-by-breath gas exchange and HR data from CPET are amenable to SampEn analysis that leads to novel insight into physiological responses to work intensity, and sex and maturational effects.

On the Stabilizing Effect of Aspartate and Glutamate and Its Counteraction by Common Denaturants.

By performing differential scanning calorimetry(DSC) measurements on RNase A, we studied the stabilization provided by the addition of potassium aspartate(KAsp) or potassium glutamate (KGlu) and found that it leads to a significant increase in the denaturation temperature of the protein. The stabilization proves to be mainly entropic in origin. A counteraction of the stabilization provided by KAsp or KGlu is obtained by adding common denaturants such as urea, guanidinium chloride, or guanidinium thiocyanate. A rationalization of the experimental data is devised on the basis of a theoretical approach developed by one of the authors. The main contribution to the conformational stability of globular proteins comes from the gain in translational entropy of water and co-solute ions and/or molecules for the decrease in solvent-excluded volume associated with polypeptide folding (i.e., there is a large decrease in solvent-accessible surface area). The magnitude of this entropic contribution increases with the number density and volume packing density of the solution. The two destabilizing contributions come from the conformational entropy of the chain, which should not depend significantly on the presence of co-solutes, and from the direct energetic interactions between co-solutes and the protein surface in both the native and denatured states. It is the magnitude of the latter that discriminates between stabilizing and destabilizing agents.

Machine learning of the prime distribution.

In the present work we use maximum entropy methods to derive several theorems in probabilistic number theory, including a version of the Hardy-Ramanujan Theorem. We also provide a theoretical argument explaining the experimental observations of Y.-H. He about the learnability of primes, and posit that the Erdos-Kac law would very unlikely be discovered by current machine learning techniques. Numerical experiments that we perform corroborate our theoretical findings.

Ultrasound normalized cumulative residual entropy imaging: Theory, methodology, and application.

BACKGROUND AND OBJECTIVE: Ultrasound information entropy imaging is an emerging quantitative ultrasound technique for characterizing local tissue scatterer concentrations and arrangements. However, the commonly used ultrasound Shannon entropy imaging based on histogram-derived discrete probability estimation suffers from the drawbacks of histogram settings dependence and unknown estimator performance. In this paper, we introduced the information-theoretic cumulative residual entropy (CRE) defined in a continuous distribution of cumulative distribution functions as a new entropy measure of ultrasound backscatter envelope uncertainty or complexity, and proposed ultrasound CRE imaging for tissue characterization. METHODS: We theoretically analyzed the CRE for Rayleigh and Nakagami distributions and proposed a normalized CRE for characterizing scatterer distribution patterns. We proposed a method based on an empirical cumulative distribution function estimator and a trapezoidal numerical integration for estimating the normalized CRE from ultrasound backscatter envelope signals. We presented an ultrasound normalized CRE imaging scheme based on the normalized CRE estimator and the parallel computation technique. We also conducted theoretical analysis of the differential entropy which is an extension of the Shannon entropy to a continuous distribution, and introduced a method for ultrasound differential entropy estimation and imaging. Monte-Carlo simulation experiments were performed to evaluate the estimation accuracy of the normalized CRE and differential entropy estimators. Phantom simulation and clinical experiments were conducted to evaluate the performance of the proposed normalized CRE imaging in characterizing scatterer concentrations and hepatic steatosis (n = 204), respectively. RESULTS: The theoretical normalized CRE for the Rayleigh distribution was π/4, corresponding to the case where there were ≥10 randomly distributed scatterers within the resolution cell of an ultrasound transducer. The theoretical normalized CRE for the Nakagami distribution decreased as the Nakagami parameter m increased, corresponding to that the ultrasound backscattered statistics varied from pre-Rayleigh to Rayleigh and to post-Rayleigh distributions. Monte-Carlo simulation experiments showed that the proposed normalized CRE and differential entropy estimators can produce a satisfying estimation accuracy even when the size of the test samples is small. Phantom simulation experiments showed that the proposed normalized CRE and differential entropy imaging can characterize scatterer concentrations. Clinical experiments showed that the proposed ultrasound normalized CRE imaging is capable to quantitatively characterize hepatic steatosis, outperforming ultrasound differential entropy imaging and being comparable to ultrasound Shannon entropy and Nakagami imaging. CONCLUSION: This study sheds light on the theory and methodology of ultrasound normalized CRE. The proposed ultrasound normalized CRE can serve as a new, flexible quantitative ultrasound envelope statistics parameter. The proposed ultrasound normalized CRE imaging may find applications in quantified characterization of biological tissues. Our code will be made available publicly at https://github.com/zhouzhuhuang.

An information-theoretic perspective of physical adversarial patches.

Real-world adversarial patches were shown to be successful in compromising state-of-the-art models in various computer vision applications. Most existing defenses rely on analyzing input or feature level gradients to detect the patch. However, these methods have been compromised by recent GAN-based attacks that generate naturalistic patches. In this paper, we propose a new perspective to defend against adversarial patches based on the entropy carried by the input, rather than on its saliency. We present Jedi, a new defense against adversarial patches that tackles the patch localization problem from an information theory perspective; leveraging the high entropy of adversarial patches to identify potential patch zones, and using an autoencoder to complete patch regions from high entropy kernels. Jedi achieves high-precision adversarial patch localization and removal, detecting on average 90% of adversarial patches across different benchmarks, and recovering up to 94% of successful patch attacks. Since Jedi relies on an input entropy analysis, it is model-agnostic, and can be applied to off-the-shelf models without changes to the training or inference of the models. Moreover, we propose a comprehensive qualitative analysis that investigates the cases where Jedi fails, comparatively with related methods. Interestingly, we find a significant core failure cases among the different defenses share one common property: high entropy. We think that this work offers a new perspective to understand the adversarial effect under physical-world settings. We also leverage these findings to enhance Jedi's handling of entropy outliers by introducing Adaptive Jedi, which boosts performance by up to 9% in challenging images.

Entropy-Driven Catalytic G-Quadruple Cycle Amplification Integrated with Ligases for Label-Free Detection of Single Nucleotide Polymorphisms.

G-Quadruplex/thioflavin (G4/THT) has become a very promising label-free fluorescent luminescent element for nucleic acid detection due to its good programmability and compatibility. However, the weak fluorescence efficiency of single-molecule G4/THT limits its potential applications. Here, we developed an entropy-driven catalytic (EDC) G4 (EDC-G4) cycle amplification technology as a universal label-free signal amplification and output system by properly programming classical EDC and G4 backbone sequences, preintegrated ligase chain reaction (LCR) for label-free sensitive detection of single nucleotide polymorphisms (SNPs). First, the positive strand LCR enabled specific transduction and preliminary signal amplification from single-base mutation information to single-strand information. Subsequently, the EDC-G4 cycle amplification reaction was activated, accompanied by the production of a large number of G4/THT luminophores to output fluorescent signals. The EDC-G4 system was proposed to address the weak fluorescence of G4/THT and obtain a label-free fluorescence signal amplification. The dual-signal amplification effect enabled the LCR-EDC-G4 detection system to accurately detect mutant target (MT) at concentrations as low as 22.39 fM and specifically identify 0.01% MT in a mixed detection pool. Moreover, the LCR-EDC-G4 system was further demonstrated for its potential application in real biological samples. Therefore, this study not only contributes ideas for the development of label-free fluorescent biosensing strategies but also provides a high-performance and practical SNP detection tool in parallel.

The ribosome lowers the entropic penalty of protein folding.

Most proteins fold during biosynthesis on the ribosome1, and co-translational folding energetics, pathways and outcomes of many proteins have been found to differ considerably from those in refolding studies2-10. The origin of this folding modulation by the ribosome has remained unknown. Here we have determined atomistic structures of the unfolded state of a model protein on and off the ribosome, which reveal that the ribosome structurally expands the unfolded nascent chain and increases its solvation, resulting in its entropic destabilization relative to the peptide chain in isolation. Quantitative 19F NMR experiments confirm that this destabilization reduces the entropic penalty of folding by up to 30 kcal mol-1 and promotes formation of partially folded intermediates on the ribosome, an observation that extends to other protein domains and is obligate for some proteins to acquire their active conformation. The thermodynamic effects also contribute to the ribosome protecting the nascent chain from mutation-induced unfolding, which suggests a crucial role of the ribosome in supporting protein evolution. By correlating nascent chain structure and dynamics to their folding energetics and post-translational outcomes, our findings establish the physical basis of the distinct thermodynamics of co-translational protein folding.

Green missing spots: Information entropy on greenhouse gas emission disclosure by Brazilian companies.

This study aims to address a critical gap in the literature by examining the incorporation of uncertainty in measuring carbon emissions using the greenhouse gas (GHG) Protocol methodology across all three scopes. By comprehensively considering the various dimensions of CO2 emissions within the context of organizational activities, our research contributes significantly to the existing body of knowledge. We address challenges such as data quality issues and a high prevalence of missing values by using information entropy, techniques for order preference by similarity to ideal solution (TOPSIS), and an artificial neural network (ANN) to analyze the contextual variables. Our findings, derived from the data sample of 56 companies across 18 sectors and 13 Brazilian states between 2017 and 2019, reveal that Scope 3 emissions exhibit the highest levels of information entropy. Additionally, we highlight the pivotal role of public policies in enhancing the availability of GHG emissions data, which, in turn, positively impacts policy-making practices. By demonstrating the potential for a virtuous cycle between improved information availability and enhanced policy outcomes, our research underscores the importance of addressing uncertainty in carbon emissions measurement for advancing effective climate change mitigation strategies.

Assessing Consciousness in Patients With Disorders of Consciousness Using a Musical Stimulation Paradigm and Verifiable Criteria.

Numerous studies have shown that musical stimulation can activate corresponding functional brain areas. Electroencephalogram (EEG) activity during musical stimulation can be used to assess the consciousness states of patients with disorders of consciousness (DOC). In this study, a musical stimulation paradigm and verifiable criteria were used for consciousness assessment. Twenty-nine participants (13 healthy subjects, 6 patients in a minimally conscious state (MCS) and 10 patients in a vegetative state (VS)) were recruited, and EEG signals were collected while participants listened to preferred and relaxing music. Fusion features based on differential entropy (DE), common spatial pattern (CSP), and EEG-based network pattern (ENP) features were extracted from EEG signals, and a convolutional neural network-long short-term memory (CNN-LSTM) model was employed to classify preferred and relaxing music.The results showed that the average classification accuracy for healthy subjects reached 85.58%. For two of the patients in the MCS group, the classification accuracies reached 78.18% and 66.14%, and they were diagnosed with emergence from MCS (EMCS) two months later. The accuracies of three patients in the VS group were 58.18%, 64.32% and 62.05%, with two patients showing slight increases in scale scores. Our study suggests that musical stimulation could be an effective method for consciousness detection, with significant diagnostic implications for patients with DOC.

Factors Associated With the Spatial Distribution of Severe Fever With Thrombocytopenia Syndrome in Zhejiang Province, China: Risk Analysis Based on Maximum Entropy.

Background: Severe fever with thrombocytopenia syndrome (SFTS) is an emerging infectious disease that was first identified in mainland China in 2009 and has been reported in Zhejiang Province, China, since 2011. However, few studies have focused on the association between ticks, host animals, and SFTS. Objective: In this study, we analyzed the influence of meteorological and environmental factors as well as the influence of ticks and host animals on SFTS. This can serve as a foundational basis for the development of strategic policies aimed at the prevention and control of SFTS. Methods: Data on SFTS incidence, tick density, cattle density, and meteorological and environmental factors were collected and analyzed using a maximum entropy-based model. Results: As of December 2019, 463 laboratory-confirmed SFTS cases were reported in Zhejiang Province. We found that the density of ticks, precipitation in the wettest month, average temperature, elevation, and the normalized difference vegetation index were significantly associated with SFTS spatial distribution. The niche model fitted accurately with good performance in predicting the potential risk areas of SFTS (the average test area under the receiver operating characteristic curve for the replicate runs was 0.803 and the SD was 0.013). The risk of SFTS occurrence increased with an increase in tick density, and the response curve indicated that the risk was greater than 0.5 when tick density exceeded 1.4. The risk of SFTS occurrence decreased with increased precipitation in the wettest month, and the risk was less than 0.5 when precipitation exceeded 224.4 mm. The relationship between elevation and SFTS occurrence showed a reverse V shape, and the risk peaked at approximately 400 m. Conclusions: Tick density, precipitation, and elevation were dominant influencing factors for SFTS, and comprehensive intervention measures should be adjusted according to these factors to reduce SFTS incidence in Zhejiang Province.

Intrinsic Rewards for Exploration Without Harm From Observational Noise: A Simulation Study Based on the Free Energy Principle.

In reinforcement learning (RL), artificial agents are trained to maximize numerical rewards by performing tasks. Exploration is essential in RL because agents must discover information before exploiting it. Two rewards encouraging efficient exploration are the entropy of action policy and curiosity for information gain. Entropy is well established in the literature, promoting randomized action selection. Curiosity is defined in a broad variety of ways in literature, promoting discovery of novel experiences. One example, prediction error curiosity, rewards agents for discovering observations they cannot accurately predict. However, such agents may be distracted by unpredictable observational noises known as curiosity traps. Based on the free energy principle (FEP), this letter proposes hidden state curiosity, which rewards agents by the KL divergence between the predictive prior and posterior probabilities of latent variables. We trained six types of agents to navigate mazes: baseline agents without rewards for entropy or curiosity and agents rewarded for entropy and/or either prediction error curiosity or hidden state curiosity. We find that entropy and curiosity result in efficient exploration, especially both employed together. Notably, agents with hidden state curiosity demonstrate resilience against curiosity traps, which hinder agents with prediction error curiosity. This suggests implementing the FEP that may enhance the robustness and generalization of RL models, potentially aligning the learning processes of artificial and biological agents.