A Minimum Free Energy Model of Motor Learning.

Even highly trained behaviors demonstrate variability, which is correlated with performance on current and future tasks. An objective of motor learning that is general enough to explain these phenomena has not been precisely formulated. In this six-week longitudinal learning study, participants practiced a set of motor sequences each day, and neuroimaging data were collected on days 1, 14, 28, and 42 to capture the neural correlates of the learning process. In our analysis, we first modeled the underlying neural and behavioral dynamics during learning. Our results demonstrate that the densities of whole-brain response, task-active regional response, and behavioral performance evolve according to a Fokker-Planck equation during the acquisition of a motor skill. We show that this implies that the brain concurrently optimizes the entropy of a joint density over neural response and behavior (as measured by sampling over multiple trials and subjects) and the expected performance under this density; we call this formulation of learning minimum free energy learning (MFEL). This model provides an explanation as to how behavioral variability can be tuned while simultaneously improving performance during learning. We then develop a novel variant of inverse reinforcement learning to retrieve the cost function optimized by the brain during the learning process, as well as the parameter used to tune variability. We show that this population-level analysis can be used to derive a learning objective that each subject optimizes during his or her study. In this way, MFEL effectively acts as a unifying principle, allowing users to precisely formulate learning objectives and infer their structure.

Control of neural systems at multiple scales using model-free, deep reinforcement learning.

Recent improvements in hardware and data collection have lowered the barrier to practical neural control. Most of the current contributions to the field have focus on model-based control, however, models of neural systems are quite complex and difficult to design. To circumvent these issues, we adapt a model-free method from the reinforcement learning literature, Deep Deterministic Policy Gradients (DDPG). Model-free reinforcement learning presents an attractive framework because of the flexibility it offers, allowing the user to avoid modeling system dynamics. We make use of this feature by applying DDPG to models of low-level and high-level neural dynamics. We show that while model-free, DDPG is able to solve more difficult problems than can be solved by current methods. These problems include the induction of global synchrony by entrainment of weakly coupled oscillators and the control of trajectories through a latent phase space of an underactuated network of neurons. While this work has been performed on simulated systems, it suggests that advances in modern reinforcement learning may enable the solution of fundamental problems in neural control and movement towards more complex objectives in real systems.

Whole-genome DNA methylation status associated with clinical PTSD measures of OIF/OEF veterans.

Emerging knowledge suggests that post-traumatic stress disorder (PTSD) pathophysiology is linked to the patients' epigenetic changes, but comprehensive studies examining genome-wide methylation have not been performed. In this study, we examined genome-wide DNA methylation in peripheral whole blood in combat veterans with and without PTSD to ascertain differentially methylated probes. Discovery was initially made in a training sample comprising 48 male Operation Enduring Freedom (OEF)/Operation Iraqi Freedom (OIF) veterans with PTSD and 51 age/ethnicity/gender-matched combat-exposed PTSD-negative controls. Agilent whole-genome array detected ~5600 differentially methylated CpG islands (CpGI) annotated to ~2800 differently methylated genes (DMGs). The majority (84.5%) of these CpGIs were hypermethylated in the PTSD cases. Functional analysis was performed using the DMGs encoding the promoter-bound CpGIs to identify networks related to PTSD. The identified networks were further validated by an independent test set comprising 31 PTSD+/29 PTSD- veterans. Targeted bisulfite sequencing was also used to confirm the methylation status of 20 DMGs shown to be highly perturbed in the training set. To improve the statistical power and mitigate the assay bias and batch effects, a union set combining both training and test set was assayed using a different platform from Illumina. The pathways curated from this analysis confirmed 65% of the pool of pathways mined from training and test sets. The results highlight the importance of assay methodology and use of independent samples for discovery and validation of differentially methylated genes mined from whole blood. Nonetheless, the current study demonstrates that several important epigenetically altered networks may distinguish combat-exposed veterans with and without PTSD.

Linear noise approximation is valid over limited times for any chemical system that is sufficiently large.

The linear noise approximation (LNA) is a way of approximating the stochastic time evolution of a well-stirred chemically reacting system. It can be obtained either as the lowest order correction to the deterministic chemical reaction rate equation (RRE) in van Kampen's system-size expansion of the chemical master equation (CME), or by linearising the two-term-truncated chemical Kramers-Moyal equation. However, neither of those derivations sheds much light on the validity of the LNA. The problematic character of the system-size expansion of the CME for some chemical systems, the arbitrariness of truncating the chemical Kramers-Moyal equation at two terms, and the sometimes poor agreement of the LNA with the solution of the CME, have all raised concerns about the validity and usefulness of the LNA. Here, the authors argue that these concerns can be resolved by viewing the LNA as an approximation of the chemical Langevin equation (CLE). This view is already implicit in Gardiner's derivation of the LNA from the truncated Kramers-Moyal equation, as that equation is mathematically equivalent to the CLE. However, the CLE can be more convincingly derived in a way that does not involve either the truncated Kramers-Moyal equation or the system-size expansion. This derivation shows that the CLE will be valid, at least for a limited span of time, for any system that is sufficiently close to the thermodynamic (large-system) limit. The relatively easy derivation of the LNA from the CLE shows that the LNA shares the CLE's conditions of validity, and it also suggests that what the LNA really gives us is a description of the initial departure of the CLE from the RRE as we back away from the thermodynamic limit to a large but finite system. The authors show that this approach to the LNA simplifies its derivation, clarifies its limitations, and affords an easier path to its solution.

Spatial localisation of chaperone distribution in the endoplasmic reticulum of yeast.

In eukaryotes, the endoplasmic reticulum (ER) serves as the first membrane-enclosed organelle in the secretory pathway, with functions including protein folding, maturation and transport. Molecular chaperones, of the Hsp70 family of proteins, participate in assisting these processes and are essential to cellular function and survival. BiP is a resident Hsp70 chaperone in the ER of Saccharomyces cerevisiae. In this study the authors have created a partial differential equation model to examine how BiP interacts with the membrane-bound co-chaperone Sec63 in translocation, a process in which BiP assists in guiding a nascent protein into the ER lumen. It has been found that when Sec63 participates in translocation through localisation at the membrane, the spatial distribution of BiP is inhomogeneous, with more BiP at the surface. When translocation is inhibited through a disabling of Sec63's membrane tether, the concentration of BiP throughout the ER becomes homogeneous. The computational simulations suggest that Sec63's localisation and the resulting binding to BiP near the membrane surface of the ER enable a heterogeneous distribution of BiP within the ER, and may facilitate BiP's role in translocation. [Includes supplementary material].

Legitimacy of the stochastic Michaelis-Menten approximation.

Michaelis-Menten kinetics are commonly used to represent enzyme-catalysed reactions in biochemical models. The Michaelis-Menten approximation has been thoroughly studied in the context of traditional differential equation models. The presence of small concentrations in biochemical systems, however, encourages the conversion to a discrete stochastic representation. It is shown that the Michaelis-Menten approximation is applicable in discrete stochastic models and that the validity conditions are the same as in the deterministic regime. The authors then compare the Michaelis-Menten approximation to a procedure called the slow-scale stochastic simulation algorithm (ssSSA). The theory underlying the ssSSA implies a formula that seems in some cases to be different from the well-known Michaelis-Menten formula. Here those differences are examined, and some special cases of the stochastic formulas are confirmed using a first-passage time analysis. This exercise serves to place the conventional Michaelis-Menten formula in a broader rigorous theoretical framework.

[Bone changes in hemoblastoses and malignant lymphomas]. / Knochenveränderungen bei Hämoblastosen und malignen Lymphomen.

In haematological systemic diseases such as acute and chronic leukoses, malignant lymphomas, lymphogranulomatoses, osteomyelofibroses, polycythaemias and aplastic anaemias with a different proportion changes of the bones in form of osteoporoses, osteolytic processes and deformations of the vertebral bodies are to be found. The proof may be performed radiologically and histologically. In 461 patients with different haematological diseases absorption measurings of monoenergetic rays of a J-125-source were performed at the distal third of radius and ulna. It was shown that the bone mineral content of patients with proved bone destructions did not significantly differ from the normal group. The too peripherally located place of the measuring and also the late inclusion of the compacta into the changes is regarded as cause for the negative result.