Forward-Backward Sweep Method for the System of HJB-FP Equations in Memory-Limited Partially Observable Stochastic Control.

Memory-limited partially observable stochastic control (ML-POSC) is the stochastic optimal control problem under incomplete information and memory limitation. To obtain the optimal control function of ML-POSC, a system of the forward Fokker-Planck (FP) equation and the backward Hamilton-Jacobi-Bellman (HJB) equation needs to be solved. In this work, we first show that the system of HJB-FP equations can be interpreted via Pontryagin's minimum principle on the probability density function space. Based on this interpretation, we then propose the forward-backward sweep method (FBSM) for ML-POSC. FBSM is one of the most basic algorithms for Pontryagin's minimum principle, which alternately computes the forward FP equation and the backward HJB equation in ML-POSC. Although the convergence of FBSM is generally not guaranteed in deterministic control and mean-field stochastic control, it is guaranteed in ML-POSC because the coupling of the HJB-FP equations is limited to the optimal control function in ML-POSC.

Decentralized Stochastic Control with Finite-Dimensional Memories: A Memory Limitation Approach.

Decentralized stochastic control (DSC) is a stochastic optimal control problem consisting of multiple controllers. DSC assumes that each controller is unable to accurately observe the target system and the other controllers. This setup results in two difficulties in DSC; one is that each controller has to memorize the infinite-dimensional observation history, which is not practical, because the memory of the actual controllers is limited. The other is that the reduction of infinite-dimensional sequential Bayesian estimation to finite-dimensional Kalman filter is impossible in general DSC, even for linear-quadratic-Gaussian (LQG) problems. In order to address these issues, we propose an alternative theoretical framework to DSC-memory-limited DSC (ML-DSC). ML-DSC explicitly formulates the finite-dimensional memories of the controllers. Each controller is jointly optimized to compress the infinite-dimensional observation history into the prescribed finite-dimensional memory and to determine the control based on it. Therefore, ML-DSC can be a practical formulation for actual memory-limited controllers. We demonstrate how ML-DSC works in the LQG problem. The conventional DSC cannot be solved except in the special LQG problems where the information the controllers have is independent or partially nested. We show that ML-DSC can be solved in more general LQG problems where the interaction among the controllers is not restricted.

Memory-Limited Partially Observable Stochastic Control and Its Mean-Field Control Approach.

Control problems with incomplete information and memory limitation appear in many practical situations. Although partially observable stochastic control (POSC) is a conventional theoretical framework that considers the optimal control problem with incomplete information, it cannot consider memory limitation. Furthermore, POSC cannot be solved in practice except in special cases. In order to address these issues, we propose an alternative theoretical framework, memory-limited POSC (ML-POSC). ML-POSC directly considers memory limitation as well as incomplete information, and it can be solved in practice by employing the technique of mean-field control theory. ML-POSC can generalize the linear-quadratic-Gaussian (LQG) problem to include memory limitation. Because estimation and control are not clearly separated in the LQG problem with memory limitation, the Riccati equation is modified to the partially observable Riccati equation, which improves estimation as well as control. Furthermore, we demonstrate the effectiveness of ML-POSC for a non-LQG problem by comparing it with the local LQG approximation.

Forward and Backward Bellman Equations Improve the Efficiency of the EM Algorithm for DEC-POMDP.

Decentralized partially observable Markov decision process (DEC-POMDP) models sequential decision making problems by a team of agents. Since the planning of DEC-POMDP can be interpreted as the maximum likelihood estimation for the latent variable model, DEC-POMDP can be solved by the EM algorithm. However, in EM for DEC-POMDP, the forward-backward algorithm needs to be calculated up to the infinite horizon, which impairs the computational efficiency. In this paper, we propose the Bellman EM algorithm (BEM) and the modified Bellman EM algorithm (MBEM) by introducing the forward and backward Bellman equations into EM. BEM can be more efficient than EM because BEM calculates the forward and backward Bellman equations instead of the forward-backward algorithm up to the infinite horizon. However, BEM cannot always be more efficient than EM when the size of problems is large because BEM calculates an inverse matrix. We circumvent this shortcoming in MBEM by calculating the forward and backward Bellman equations without the inverse matrix. Our numerical experiments demonstrate that the convergence of MBEM is faster than that of EM.

NMDAR-Mediated Ca2+ Increase Shows Robust Information Transfer in Dendritic Spines.

A dendritic spine is a small structure on the dendrites of a neuron that processes input timing information from other neurons. Tens of thousands of spines are present on a neuron. Why are spines so small and many? We addressed this issue by using the stochastic simulation model of N-methyl-D-aspartate receptor (NMDAR)-mediated Ca2+ increase. NMDAR-mediated Ca2+ increase codes the input timing information between prespiking and postspiking. We examined how much the input timing information is encoded by Ca2+ increase against presynaptic fluctuation. We found that the input timing information encoded in the cell volume (103µm3) largely decreases against the presynaptic fluctuation, whereas that in the spine volume (10-1µm3) slightly decreases. Therefore, the input timing information encoded in the spine volume is more robust against presynaptic fluctuation than that in the cell volume. We further examined the mechanism of the robust information transfer in the spine volume. We demonstrated that the condition for the robustness is that the stochastic NMDAR-mediated Ca2+ increase (intrinsic noise) becomes much larger than the presynaptic fluctuation (extrinsic noise). When the presynaptic fluctuation is large, the condition is satisfied in the spine volume but not in the cell volume. Moreover, we compared the input timing information encoded in many small spines with that encoded in a single large spine. We found that the input timing information encoded in many small spines are larger than that in a single large spine when presynaptic fluctuation is large because of their robustness. Thus, robustness is a functional reason why dendritic spines are so small and many.

Robustness against additional noise in cellular information transmission.

Fluctuations in intracellular reactions (intrinsic noise) reduce the information transmitted from an extracellular input to a cellular response. However, recent studies have demonstrated that the decrease in the transmitted information with respect to extracellular input fluctuations (extrinsic noise) is smaller when the intrinsic noise is larger. Therefore, it has been suggested that robustness against extrinsic noise increases with the level of the intrinsic noise. We call this phenomenon intrinsic noise-induced robustness (INIR). As previous studies on this phenomenon have focused on complex biochemical reactions, the relation between INIR and the input-output of a system is unclear. Moreover, the mechanism of INIR remains elusive. In this paper, we address these questions by analyzing simple models. We first analyze a model in which the input-output relation is linear. We show that the robustness against extrinsic noise increases with the intrinsic noise, confirming the INIR phenomenon. Moreover, the robustness against the extrinsic noise is more strongly dependent on the intrinsic noise when the variance of the intrinsic noise is larger than that of the input distribution. Next, we analyze a threshold model in which the output depends on whether the input exceeds the threshold. When the threshold is equal to the mean of the input, INIR is realized, but when the threshold is much larger than the mean, the threshold model exhibits stochastic resonance, and INIR is not always apparent. The robustness against extrinsic noise and the transmitted information can be traded off against one another in the linear model and the threshold model without stochastic resonance, whereas they can be simultaneously increased in the threshold model with stochastic resonance.