State-space Model Based Inverse Reinforcement Learning for Reward Function Estimation in Brain-machine Interfaces.

The use of reinforcement learning (RL) in brain machine interfaces (BMIs) is considered to be a promising method for neural decoding. One key component of RL-based BMIs is the reward signal, which is used to guide decoders to update the parameters. However, designing effective and efficient rewards can be challenging, especially for complex tasks. Inverse reinforcement learning (IRL) is a method that has been proposed to estimate the internal reward function from subjects' neural activity. However, multi-channel neural activity, which may encode many sources of information, builds a large dimensions of state-action space, making it difficult to directly apply IRL methods in BMI systems. In this paper, we propose a state-space model based inverse Q-learning (SSM-IQL) method to improve the performance of the existing IRL method. The state-space model is designed to extract hidden brain state from high-dimensional neural activity. We tested the proposed method on real data collected from rats during a two-lever discrimination task. Preliminary results show that SSM-IQL provides a more accurate and stable estimation of the internal reward function than the traditional IQL algorithm. This suggests that the use of state-space model in IRL method has potential to improve the design of RL-based BMIs.

Audio-induced medial prefrontal cortical dynamics enhances coadaptive learning in brain-machine interfaces.

Objectives. Coadaptive brain-machine interfaces (BMIs) allow subjects and external devices to adapt to each other during the closed-loop control, which provides a promising solution for paralyzed individuals. Previous studies have focused on either improving sensory feedback to facilitate subject learning or developing adaptive algorithms to maintain stable decoder performance. In this work, we aim to design an efficient coadaptive BMI framework which not only facilitates the learning of subjects on new tasks with designed sensory feedback, but also improves decoders' learning ability by extracting sensory feedback-induced evaluation information.Approach. We designed dynamic audio feedback during the trial according to the subjects' performance when they were trained to learn a new behavioral task. We compared the learning performance of two groups of Sprague Dawley rats, one with and the other without the designed audio feedback to show whether this audio feedback could facilitate the subjects' learning. Compared with the traditional closed-loop in BMI systems, an additional closed-loop involving medial prefrontal cortex (mPFC) activity was introduced into the coadaptive framework. The neural dynamics of audio-induced mPFC activity was analyzed to investigate whether a significant neural response could be triggered. This audio-induced response was then translated into reward expectation information to guide the learning of decoders on a new task. The multiday decoding performance of the decoders with and without audio-induced reward expectation was compared to investigate whether the extracted information could accelerate decoders to learn a new task.Main results. The behavior performance comparison showed that the average days for rats to achieve 80% well-trained behavioral performance was improved by 26.4% after introducing the designed audio feedback sequence. The analysis of neural dynamics showed that a significant neural response of mPFC activity could be elicited by the audio feedback and the visualization of audio-induced neural patterns was emerged and accompanied by the behavioral improvement of subjects. The multiday decoding performance comparison showed that the decoder taking the reward expectation information could achieve faster task learning by 33.8% on average across subjects.Significance. This study demonstrates that the designed audio feedback could improve the learning of subjects and the mPFC activity induced by audio feedback can be utilized to improve the decoder's learning efficiency on new tasks. The coadaptive framework involving mPFC dynamics in the closed-loop interaction can advance the BMIs into a more adaptive and efficient system with learning ability on new tasks.

Estimating Reward Function from Medial Prefrontal Cortex Cortical Activity using Inverse Reinforcement Learning.

Reinforcement learning (RL)-based brain-machine interfaces (BMIs) learn the mapping from neural signals to subjects' intention using a reward signal. External rewards (water or food) or internal rewards extracted from neural activity are leveraged to update the parameters of decoders in the existing RL-based BMI framework. However, for complex tasks, the design of external reward could be difficult, which may not fully reflect the subject's own evaluation internally. It is important to obtain an internal reward model from neural activity to access subject's internal evaluation when the subject is performing the task through trial and error. In this paper, we propose to use an inverse reinforcement learning (IRL) method to estimate the internal reward function interpreted from the brain to assist the update of the decoders. Specifically, the inverse Q-learning (IQL) algorithm is applied to extract internal reward information from real data collected from medial prefrontal cortex (mPFC) when a rat was learning a two-lever-press discrimination task. Such an internal reward information is validated by checking whether it can guide the training of the RL decoder to complete movement task. Compared with the RL decoder trained with the external reward, our approach achieves a similar decoding performance. This preliminary result validates the effectiveness of using IRL to obtain the internal reward model. It reveals the potential of estimating internal reward model to improve the design of autonomous learning BMIs.

Multivariate Encoding Analysis of Medial Prefrontal Cortex Cortical Activity during Task Learning.

Studies have shown that medial prefrontal cortex (mPFC) is responsible for outcome evaluation. Some recent studies also suggest that mPFC may play an important role in goal planning and action execution when performing a task. If the information encoded in mPFC can be accurately extracted and identified, it can improve the design of brain-machine interfaces by better reconstructing subjects' motion intention guided by reward information. In this paper, we investigate whether mPFC neural signals simultaneously encode information of goal planning, action execution and outcome evaluation. Linear-nonlinear-Poisson (LNP) model is applied for encoding analysis on mPFC neural spike data when a rat is learning a two-lever-press discrimination task. We use the L2-norm of tuning parameter in LNP model to indicate the importance of the encoded information and compare the spike train prediction performance of LNP model using all information, the most significant information and reward information only. The preliminary results indicate that mPFC activity can encode simultaneously the information of goal planning, action execution and outcome evaluation and that all the relevant information could be reconstructed from mPFC spike trains on a single trial basis.

Chirality induced asymmetric spin splitting of light beams reflected from an air-chiral interface.

The spin Hall effect (SHE) of light beams reflected from an air-chiral interface are investigated systematically. Due to the intrinsic chiral asymmetry of the medium, a horizontally polarized incident Gaussian beam will undergo asymmetric spin splitting, i.e., both the displacements and energies of two spin components of the reflected beam are different. One spin component can undergo large displacement near points of |rpp| = |rsp| (rpp and rsp are the Fresnel reflection coefficients), where the reflected beams are almost in circular polarization states. Moreover, for an incident beam carrying orbital angular momentum (OAM), the two spin components acquire additional OAM dependent shifts, which attribute to the asymmetric spin splitting. Thus, the asymmetric spin splitting of the reflected beam will vary with the incident OAM. These findings provide a deeper insight into the SHE of light, and they may have potential application in precision metrology.

Giant spin splitting induced by orbital angular momentum in an epsilon-near-zero metamaterial slab.

An orbital angular momentum (OAM)-induced spin splitting is theoretically predicted when a higher-order Laguerre-Gaussian beam is transmitted through a metamaterial slab. The upper bound of this spin splitting is found to be |ℓ|w0/(|ℓ|+1)1/2, where ℓ and w0 are the incident OAM and beam waist, respectively. By optimizing the structure parameter of the metamaterial, as well as the incident angle, the OAM-induced spin splitting can reach more than 0.99 of the upper bound in the cases of both the horizontal and vertical incident polarization states, and the transmitted light fields turn out to be full Poincaré beams. These findings provide a deeper insight into the spin-orbit interaction, and, thereby, facilitate the development of spin-based applications.