Existence of reservoir with finite-dimensional output for universal reservoir computing.

In this paper, we prove the existence of a reservoir that has a finite-dimensional output and makes the reservoir computing model universal. Reservoir computing is a method for dynamical system approximation that trains the static part of a model but fixes the dynamical part called the reservoir. Hence, reservoir computing has the advantage of training models with a low computational cost. Moreover, fixed reservoirs can be implemented as physical systems. Such reservoirs have attracted attention in terms of computation speed and energy consumption. The universality of a reservoir computing model is its ability to approximate an arbitrary system with arbitrary accuracy. Two sufficient reservoir conditions to make the model universal have been proposed. The first is the combination of fading memory and the separation property. The second is the neighborhood separation property, which we proposed recently. To date, it has been unknown whether a reservoir with a finite-dimensional output can satisfy these conditions. In this study, we prove that no reservoir with a finite-dimensional output satisfies the former condition. By contrast, we propose a single output reservoir that satisfies the latter condition. This implies that, for any dimension, a reservoir making the model universal exists with the output of that specified dimension. These results clarify the practical importance of our proposed conditions.

Automatic Temperature Parameter Tuning for Reinforcement Learning Using Path Integral Policy Improvement.

In this article, we propose a novel variant of path integral policy improvement with covariance matrix adaptation ( [Formula: see text] - [Formula: see text] ), which is a reinforcement learning (RL) algorithm that aims to optimize a parameterized policy for the continuous behavior of robots. [Formula: see text] - [Formula: see text] has a hyperparameter called the temperature parameter, and its value is critical for performance; however, little research has been conducted on it and the existing method still contains a tunable parameter, which can be critical to performance. Therefore, tuning by trial and error is necessary in the existing method. Moreover, we show that there is a problem setting that cannot be learned by the existing method. The proposed method solves both problems by automatically adjusting the temperature parameter for each update. We confirmed the effectiveness of the proposed method using numerical tests.

Nonessentiality of Reservoir's Fading Memory for Universality of Reservoir Computing.

This article describes a novel sufficient condition concerning approximations with reservoir computing (RC). Recently, RC using a physical system as the reservoir has attracted attention. Because many physical systems are modeled as state-space systems, it is necessary to guarantee the approximations given by reservoirs represented as nonlinear state-space systems. There are two problems with existing approaches: a reservoir must have a property called fading memory and must be represented as a set of maps between input and output signals on the bi-infinite-time (BIT) interval. These two conditions are too strict for reservoirs represented as nonlinear state-space systems as they require the reservoir to have a unique equilibrium state for the zero input. This article proposes an approach that employs operators from right-infinite-time (RIT) inputs to RIT outputs. Furthermore, we develop a novel extension of the Stone-Weierstrass theorem to handle discontinuous functions. To apply the extended theorem, we define functionals corresponding to operators and introduce a metric on the domain of the functionals. The resulting sufficient condition does not require the reservoir to have fading memory or continuity with respect to inputs and time. Therefore, our result guarantees the approximations with very common reservoirs and provides a rationale for physical RC. We present an example of a physical reservoir without fading memory. With the example reservoir, the RC model successfully approximates NARMA10, a benchmark task for time series predictions.

Simultaneous Optimization of Discrete and Continuous Parameters Defining a Robot Morphology and Controller.

The morphology and controller design of robots is often a labor-intensive task performed by experienced and intuitive engineers. Automatic robot design using machine learning is attracting increasing attention in the hope that it will reduce the design workload and result in better-performing robots. Most robots are created by joining several rigid parts and then mounting actuators and their controllers. Many studies limit the possible types of rigid parts to a finite set to reduce the computational burden. However, this not only limits the search space, but also prohibits the use of powerful optimization techniques. To find a robot closer to the global optimal design, a method that explores a richer set of robots is desirable. In this article, we propose a novel method to efficiently search for various robot designs. The method combines three different optimization methods with different characteristics. We apply proximal policy optimization (PPO) or soft actor-critic (SAC) as the controller, the REINFORCE algorithm to determine the lengths and other numerical parameters of the rigid parts, and a newly proposed method to determine the number and layout of the rigid parts and joints. Experiments with physical simulations confirm that when this method is used to handle two types of tasks-walking and manipulation-it performs better than simple combinations of existing methods. The source code and videos of our experiments are available online (https://github.com/r-koike/eagent).

Step Climbing Control of Snake Robot with Prismatic Joints.

The ultimate goal of this research study is to perform continuous rather than sequential movements of prismatic joints for effective motion of a snake robot with prismatic joints in a complex terrain. We present herein a control method for robotic step climbing. This method is composed of two parts: the first involves the shift reference generator that generates the joint motion for climbing a step, and the other is use of the trajectory tracking controller, which generates the joint motion for the head to track the target trajectory. In this method, prismatic joints are divided into those that are directly controlled for climbing a step and those that are represented as redundancies. By directly controlling the link length, it is possible to prevent the trailing part from back motion when climbing a step, and to avoid a singular configuration in the parts represented as redundancies. A snake robot that has rotational and prismatic joints and can move in three-dimensions was developed, and the effectiveness of the proposed method was demonstrated by experiments using this robot. In the experiment, it was confirmed that the proposed method realizes the step climbing, and the link length limitation using the sigmoid function works effectively.

Path Integral Policy Improvement With Population Adaptation.

Path integral policy improvement (PI2) is known to be an efficient reinforcement learning algorithm, particularly, if the target system is a high-dimensional dynamical system. However, PI2, and its existing extensions, have adjustable parameters, on which the efficiency depends significantly. This article proposes an extension of PI2 that adjusts all of the critical parameters automatically. Motion acquisition tasks for three different types of simulated legged robots were performed to test the efficacy of the proposed algorithm. The results show that the proposed method cannot only eliminate the burden on the user to set the parameters appropriately but also improve the optimization performance significantly. For one of the acquired motions, a real robot experiment was conducted to show the validity of the motion.