Improving Contractors' Participation of Resource Utilization in Construction and Demolition Waste through Government Incentives and Punishments.

This paper develops a simulation model for analyzing how government incentives and punishments improve contractors' participation in resource utilization of construction and demolition waste (RUCDW) based on system dynamics theory. The construction industry's long-term objective is to become more sustainable and resource-effective, and as part of this objective, generated construction and demolition waste should be recycled and resource utilized. However, most contractors have little willingness to engage in RUCDW because it increases their costs. The government thus plays a vital role in improving their participation in RUCDW through a range of educational tools such as advertisements, professional training, incentives, and punishments. Among these approaches, incentives and punishments are considered the most effective because they directly change project costs. We use the Vensim software package for numerical simulation and data collected from Suzhou, China are used to demonstrate and validate the developed model. Simulation results show that the government can improve contractors' participation in RUCDW through three kinds of incentives and punishments: (1) subsidizing RUCDW; (2) increasing landfill fees; and 3) issuing fines for illegal dumping. Comprehensive application of multiple policies has a stronger effect than single policies. The established model is therefore a valuable tool for assessing the dynamic effects of government incentives and punishments on RUCDW ahead of implementation, which can provide guidance for policymakers.

Hierarchical Dynamical Model for Multiple Cortical Neural Decoding.

Motor brain machine interfaces (BMIs) interpret neural activities from motor-related cortical areas in the brain into movement commands to control a prosthesis. As the subject adapts to control the neural prosthesis, the medial prefrontal cortex (mPFC), upstream of the primary motor cortex (M1), is heavily involved in reward-guided motor learning. Thus, considering mPFC and M1 functionality within a hierarchical structure could potentially improve the effectiveness of BMI decoding while subjects are learning. The commonly used Kalman decoding method with only one simple state model may not be able to represent the multiple brain states that evolve over time as well as along the neural pathway. In addition, the performance of Kalman decoders degenerates in heavy-tailed nongaussian noise, which is usually generated due to the nonlinear neural system or influences of movement-related noise in online neural recording. In this letter, we propose a hierarchical model to represent the brain states from multiple cortical areas that evolve along the neural pathway. We then introduce correntropy theory into the hierarchical structure to address the heavy-tailed noise existing in neural recordings. We test the proposed algorithm on in vivo recordings collected from the mPFC and M1 of two rats when the subjects were learning to perform a lever-pressing task. Compared with the classic Kalman filter, our results demonstrate better movement decoding performance due to the hierarchical structure that integrates the past failed trial information over multisite recording and the combination with correntropy criterion to deal with noisy heavy-tailed neural recordings.

A Nonlinear Maximum Correntropy Information Filter for High-Dimensional Neural Decoding.

Neural signal decoding is a critical technology in brain machine interface (BMI) to interpret movement intention from multi-neural activity collected from paralyzed patients. As a commonly-used decoding algorithm, the Kalman filter is often applied to derive the movement states from high-dimensional neural firing observation. However, its performance is limited and less effective for noisy nonlinear neural systems with high-dimensional measurements. In this paper, we propose a nonlinear maximum correntropy information filter, aiming at better state estimation in the filtering process for a noisy high-dimensional measurement system. We reconstruct the measurement model between the high-dimensional measurements and low-dimensional states using the neural network, and derive the state estimation using the correntropy criterion to cope with the non-Gaussian noise and eliminate large initial uncertainty. Moreover, analyses of convergence and robustness are given. The effectiveness of the proposed algorithm is evaluated by applying it on multiple segments of neural spiking data from two rats to interpret the movement states when the subjects perform a two-lever discrimination task. Our results demonstrate better and more robust state estimation performance when compared with other filters.

Online Estimating Pairwise Neuronal Functional Connectivity in Brain-Machine Interface.

Neurons respond to external stimuli and form functional networks through pairwise interactions. A neural encoding model can describe a single neuron's behavior, and brain-machine interfaces (BMIs) provide a platform to investigate how neurons adapt, functionally connect, and encode movement. Movement modulation and pairwise functional connectivity are modeled as high-dimensional tuning states, estimated from neural spike train observations. However, accurate estimation of this neural state vector can be challenging as pairwise neural interactions are highly dimensional, change in different temporal scales from movement, and could be non-stationary. We propose an Adam-based gradient descent method to online estimate high-dimensional pairwise neuronal functional connectivity and single neuronal tuning adaptation simultaneously. By minimizing negative log-likelihood based on point process observation, the proposed method adaptively adjusts the learning rate for each dimension of the neural state vectors by employing momentum and regularizer. We test the method on real recordings of two rats performing the brain control mode of a two-lever discrimination task. Our results show that our method outperforms existing methods, especially when the state is sparse. Our method is more stable and faster for an online scenario regardless of the parameter initializations. Our method provides a promising tool to track and build the time-variant functional neural connectivity, which dynamically forms the functional network and results in better brain control.

The accuracy of screening tools for sarcopenia in older Chinese adults: a systematic review and meta-analysis.

Objective: This review aimed to analyze and compare the accuracy of eight screening tools for sarcopenia in older Chinese adults according to different diagnostic criteria. Methods: This systematic review was conducted in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines. The PubMed, Embase, Web of Science, China National Knowledge Infrastructure (CNKI), and Wanfang databases were searched between the publication of the first expert consensus on sarcopenia in 2010 and April 2023 using relevant MeSH terms. We evaluated the risk bias of the included studies using the Quality Assessment of Diagnostic Accuracy Studies-2 (QUADAS-2) tool. The pooled result of sensitivity, specificity, positive likelihood ratio (PLR), negative likelihood ratio (NLR), diagnostic odds ratio (DOR), and plot the summary receiver operating characteristic curve (SROC) were calculated by using a bivariate random-effects model. The accuracies of sensitivity and specificity of the screening tools were compared using the Z-test. Results: A total of 30 studies (23,193 participants) were included, except for calf circumference (CC), Ishii, and Finger-ring Test; Screening tools for sarcopenia in older Chinese adults have consistently shown low to moderate sensitivity and moderate to high specificity. Regional and sex differences affect the accuracy of the screening tools. In terms of sensitivity and specificity, the CC, Ishii, and Finger-ring Test were superior to the other screening tools. Conclusion: The Asian Working Group on Sarcopenia (AWGS) 2019 criteria are more appropriate for the diagnosis of sarcopenia in older Chinese adults. According to the AWGS 2019, CC and Ishii are recommended for sarcopenia screening in older Chinese adults.

Giant nonlinear Hall and wireless rectification effects at room temperature in the elemental semiconductor tellurium.

The second-order nonlinear Hall effect (NLHE) in non-centrosymmetric materials has recently drawn intense interest, since its inherent rectification could enable various device applications such as energy harvesting and wireless charging. However, previously reported NLHE systems normally suffer from relatively small Hall voltage outputs and/or low working temperatures. In this study, we report the observation of a pronounced NLHE in tellurium (Te) thin flakes at room temperature. Benefiting from the semiconductor nature of Te, the obtained nonlinear response can be readily enhanced through electrostatic gating, leading to a second-harmonic output at 300 K up to 2.8 mV. By utilizing such a giant NLHE, we further demonstrate the potential of Te as a wireless Hall rectifier within the radiofrequency range, which is manifested by the remarkable and tunable rectification effect also at room temperature. Extrinsic scattering is then revealed to be the dominant mechanism for the NLHE in Te, with symmetry breaking on the surface playing a key role. As a simple elemental semiconductor, Te provides an appealing platform to advance our understanding of nonlinear transport in solids and to develop NLHE-based electronic devices.

A Bayesian Decoder Representing Single-Directional Connectivity between Neurons in Brain-Machine Interface.

Directional neural connectivity is essential to understanding how neurons encode and transmit information in the neural network. The previous studies on single neuronal encoding models illustrate how the neurons modulate the stimulus, underlying movement, and interactions with other neurons. And these encoding models have been used in the Bayesian decoders of the brain-machine interface (BMI) to explain how the neural population represents the movement intentions. However, the existing methods only consider rough correlations between neurons without directional connections, while the synapses between real neurons have explicit directions. Therefore, in these models, we cannot specify the proper functional neural connectivity and how the neurons cooperate to represent the movement intentions in truth. Therefore, we propose representing the directional neural connectivity in the Bayesian decoder in BMI. Our method derives a chain-likelihood based on Bayes' rule to form the single-directional influence between neurons. According to the derived structure, the prior causality relationship can be used to build more precise neural encoding models. Therefore, our method can represent the functional neural circuit more precisely and benefit the decoding in the BMI. We validate the proposed method in synthetic data simulating the rat's two-lever discrimination task. The results demonstrate that our method outperforms the existing methods by representing directional-neural connectivity. Besides, our method is more efficient in training because it employs fewer parameters. Consequently, our method can be used to evaluate the causality between neurons at the behavior level.Clinical Relevance-This paper proposes a decoder that can represent single-directional neural connectivity, which is potential to validate the causality relationship between neurons at behavior level.

Kernel Reinforcement Learning-Assisted Adaptive Decoder Facilitates Stable and Continuous Brain Control Tasks.

Brain-Machine Interfaces (BMIs) assist paralyzed people to brain control (BC) the neuro-prosthesis continuously moving in space. During the BC process, the subject imagines the movement of the real limb and adapts the brain activity according to the sensory feedback. The neural adaptation in the closed-loop control results in complex and changing brain signals. Simultaneously, the decoder interprets the time-varying functional mapping between neural activity and continuous trajectory. It is crucial and challenging to accurately and adaptively track the mapping to help the subject accomplish the BC task with a stable performance. Existing Kalman Filter (KF) based decoders achieve continuous trajectory control by linearly interpreting neural firing observations into self-evolving prosthetic states. However, the linear neural-state mapping might not accurately reflect the movement intention of the subject. In this paper, we propose a novel method that allows subjects to achieve continuous brain control efficiently and stably. The proposed method incorporates a kernel reinforcement learning method into a state-observation model to decode the nonlinearly neural observation into a continuous trajectory state. The state transition function ensures the continuity of the prosthetic state. The kernel reinforcement learning allows the quick adaptation of the nonlinear neural-movement mapping during the BC process. The proposed method is tested in an online brain control reaching task for rats. Compared with KF, our method achieved more successful trials, faster response time, shorter inter-trial time, and remained stable over days. These results demonstrate that the proposed method is an efficient tool to assist subjects in brain control tasks.

Tracking the Dynamic Neural Connectivity via Conjugate Gradient Optimization.

Neural connectivity describes how neuron populations coordinate and create cognitive and behavioral functions. Neural connectivity performs dynamics where its population spiking responses to stimuli or intention change over time. Brain-machine interface (BMI) provides a framework for studying dynamical neural connectivity. In BMI, point process is a powerful technique in analyzing the single neuronal tuning. And generalized linear mode (GLM) as an encoding model can incorporate the tuning in kinematics and the neural connectivity. Quantification and tracking of dynamic neural connectivity can contribute to the elucidation of the generation of brain functions in a computational way. However, most of the previous work focused on single neuronal adaptation to kinematics. When a neuron is significantly modulated by some other neurons in some tasks, the shape of the log likelihood function for single neuronal observations can be narrowed in some dimensions. And the existing gradient-based methods are not able to reach the optimum in a fast and adaptive searching way. In this work, to maximize the likelihood of observations and obtain the dynamic neural connectivity tuning parameters, we proposed a conjugate gradient-based encoding model (CGE). We illustrate CGE for likelihood function using the real experimental data under manual control and brain control. The results show that the proposed CGE has better performance in tracking the dynamic neural connectivity tuning parameters and modeling neural encoding.Clinical Relevance- Not directly related.

Audio-induced medial prefrontal cortical dynamics enhances coadaptive learning in brain-machine interfaces.

Objectives. Coadaptive brain-machine interfaces (BMIs) allow subjects and external devices to adapt to each other during the closed-loop control, which provides a promising solution for paralyzed individuals. Previous studies have focused on either improving sensory feedback to facilitate subject learning or developing adaptive algorithms to maintain stable decoder performance. In this work, we aim to design an efficient coadaptive BMI framework which not only facilitates the learning of subjects on new tasks with designed sensory feedback, but also improves decoders' learning ability by extracting sensory feedback-induced evaluation information.Approach. We designed dynamic audio feedback during the trial according to the subjects' performance when they were trained to learn a new behavioral task. We compared the learning performance of two groups of Sprague Dawley rats, one with and the other without the designed audio feedback to show whether this audio feedback could facilitate the subjects' learning. Compared with the traditional closed-loop in BMI systems, an additional closed-loop involving medial prefrontal cortex (mPFC) activity was introduced into the coadaptive framework. The neural dynamics of audio-induced mPFC activity was analyzed to investigate whether a significant neural response could be triggered. This audio-induced response was then translated into reward expectation information to guide the learning of decoders on a new task. The multiday decoding performance of the decoders with and without audio-induced reward expectation was compared to investigate whether the extracted information could accelerate decoders to learn a new task.Main results. The behavior performance comparison showed that the average days for rats to achieve 80% well-trained behavioral performance was improved by 26.4% after introducing the designed audio feedback sequence. The analysis of neural dynamics showed that a significant neural response of mPFC activity could be elicited by the audio feedback and the visualization of audio-induced neural patterns was emerged and accompanied by the behavioral improvement of subjects. The multiday decoding performance comparison showed that the decoder taking the reward expectation information could achieve faster task learning by 33.8% on average across subjects.Significance. This study demonstrates that the designed audio feedback could improve the learning of subjects and the mPFC activity induced by audio feedback can be utilized to improve the decoder's learning efficiency on new tasks. The coadaptive framework involving mPFC dynamics in the closed-loop interaction can advance the BMIs into a more adaptive and efficient system with learning ability on new tasks.

Modeling Neural Connectivity in a Point-Process Analogue of Kalman Filter.

A neural encoding model describes how single neuron tunes to external stimuli as well as its connectivity with other neurons. The connectivity illustrates the neuronal interaction within populations in response to the shared latent brain states. Understanding these interactions is crucial to computationally predict the neural activity, which elucidates the neural encoding mechanism A computational analysis on the neural connectivity also facilitates developing more point process decoding model to interpret movement state from neural spike observations for brain machine interfaces (BMI). Most of the previous point process models only consider single neural tuning property and assumes conditional independence among multiple neurons. The connectivity among neurons is not considered in such a Bayesian approach to derive the state. In this work, we propose a point-process analogue of Kalman Filter to model the neural connectivity in a closed-form Bayesian filter. Neural connectivity corrects the posterior of the state given the multi-dimension observation, and a Gaussian distribution is used to approximate the updated posterior distribution. We validate the proposed method on simulation data and compared with traditional point process filtering with conditional independent assumption. The result shows that our method models the neural connectivity information and the single neuronal tuning property at the same time and achieves a better performance of the state decoding. Clinical Relevance - This paper proposes a closed-form derivation of a point process filter based on Gaussian approximations. It can model both single neuronal tuning property and the neural connectivity, which is potential to understanding the neural connectivity computationally.

Intermediate Sensory Feedback Assisted Multi-Step Neural Decoding for Reinforcement Learning Based Brain-Machine Interfaces.

Reinforcement-learning (RL)-based brain-machine interfaces (BMIs) interpret dynamic neural activity into movement intention without patients' real limb movements, which is promising for clinical applications. A movement task generally requires the subjects to reach the target within one step and rewards the subjects instantaneously. However, a real BMI scenario involves tasks that require multiple steps, during which sensory feedback is provided to indicate the status of the prosthesis, and the reward is only given at the end of the trial. Actually, subjects internally evaluate the sensory feedback to adjust motor activity. Existing RL-BMI tasks have not fully utilized the internal evaluation from the brain upon the sensory feedback to guide the decoder training, and there lacks an effective tool to assign credit for the multi-step decoding task. We propose first to extract intermediate guidance from the medial prefrontal cortex (mPFC) to assist the learning of multi-step decoding in an RL framework. To effectively explore the neural-action mapping in a large state-action space, a temporal difference (TD) method is incorporated into quantized attention-gated kernel reinforcement learning (QAGKRL) to assign the credit over the temporal sequence of movement, but also discriminate spatially in the Reproducing Kernel Hilbert Space (RKHS). We test our approach on the data collected from the primary motor cortex (M1) and the mPFC of rats when they brain control the cursor to reach the target within multiple steps. Compared with the models which only utilize the final reward, the intermediate evaluation interpreted from the mPFC can help improve the prediction accuracy by 10.9% on average across subjects, with faster convergence and more stability. Moreover, our proposed algorithm further increases 18.2% decoding accuracy compared with existing TD-RL methods. The results reveal the possibility of achieving better multi-step decoding performance for more complicated BMI tasks.

Extracting synchronized neuronal activity from local field potentials based on a marked point process framework.

Objective.Brain-machine interfaces (BMIs) translate neural activity into motor commands to restore motor functions for people with paralysis. Local field potentials (LFPs) are promising for long-term BMIs, since the quality of the recording lasts longer than single neuronal spikes. Inferring neuronal spike activity from population activities such as LFPs is challenging, because LFPs stem from synaptic currents flowing in the neural tissue produced by various neuronal ensembles and reflect neural synchronization. Existing studies that combine LFPs with spikes leverage the spectrogram of the former, which can neither detect the transient characteristics of LFP features (here, neuromodulation in a specific frequency band) with high accuracy, nor correlate them with relevant neuronal activity with a sufficient time resolution.Approach.We propose a feature extraction and validation framework to directly extract LFP neuromodulations related to synchronized spike activity using recordings from the primary motor cortex of six Sprague Dawley rats during a lever-press task. We first select important LFP frequency bands relevant to behavior, and then implement a marked point process (MPP) methodology to extract transient LFP neuromodulations. We validate the LFP feature extraction by examining the correlation with the pairwise synchronized firing probability of important neurons, which are selected according to their contribution to behavioral decoding. The highly correlated synchronized firings identified by the LFP neuromodulations are fed into a decoder to check whether they can serve as a reliable neural data source for movement decoding.Main results.We find that the gamma band (30-80 Hz) LFP neuromodulations demonstrate significant correlation with synchronized firings. Compared with traditional spectrogram-based method, the higher-temporal resolution MPP method captures the synchronized firing patterns with fewer false alarms, and demonstrates significantly higher correlation than single neuron spikes. The decoding performance using the synchronized neuronal firings identified by the LFP neuromodulations can reach 90% compared to the full recorded neuronal ensembles.Significance.Our proposed framework successfully extracts the sparse LFP neuromodulations that can identify temporal synchronized neuronal spikes with high correlation. The identified neuronal spike pattern demonstrates high decoding performance, which suggest LFP can be used as an effective modality for long-term BMI decoding.

Spike prediction on primary motor cortex from medial prefrontal cortex during task learning.

Objectives. Brain-machine interfaces (BMIs) aim to help people with motor disabilities by interpreting brain signals into motor intentions using advanced signal processing methods. Currently, BMI users require intensive training to perform a pre-defined task, not to mention learning a new task. Thus, it is essential to understand neural information pathways among the cortical areas in task learning to provide principles for designing BMIs with learning abilities. We propose to investigate the relationship between the medial prefrontal cortex (mPFC) and primary motor cortex (M1), which are actively involved in motor control and task learning, and show how information is conveyed in spikes between the two regions on a single-trial basis by computational models.Approach. We are interested in modeling the functional relationship between mPFC and M1 activities during task learning. Six Sprague Dawley rats were trained to learn a new behavioral task. Neural spike data was recorded from mPFC and M1 during learning. We then implement the generalized linear model, the second-order generalized Laguerre-Volterra model, and the staged point-process model to predict M1 spikes from mPFC spikes across multiple days during task learning. The prediction performance is compared across different models or learning stages to reveal the relationship between mPFC and M1 spike activities.Main results. We find that M1 neural spikes can be well predicted from mPFC spikes on the single-trial level, which indicates a highly correlated relationship between mPFC and M1 activities during task learning. By comparing the performance across models, we find that models with higher nonlinear capacity perform significantly better than linear models. This indicates that predicting M1 activity from mPFC activity requires the model to consider higher-order nonlinear interactions beyond pairwise interactions. We also find that the correlation coefficient between the mPFC and M1 spikes increases during task learning. The spike prediction models perform the best when the subjects become well trained on the new task compared with the early and middle stages. The results suggest that the co-activation between mPFC and M1 activities evolves during task learning, and becomes stronger as subjects become well trained.Significance. This study demonstrates that the dynamic patterns of M1 spikes can be predicted from mPFC spikes during task learning, and this will further help in the design of adaptive BMI decoders for task learning.

Considering Neural Connectivity in Point Process Decoder for Brain-Machine Interface.

Brain machine interface (BMI) can translate neural activity into digital commands to control prostheses. The decoder in BMI models the mechanism relating to neural activity and intents in brain. In our brain, single neuronal tuning property and neural connectivity contribute to encoding the intents together. These properties may change, a phenomenon which is named neural adaptation during using BMIs. Neural adaptation requires the decoder to consider the two factors at the same time and has the potential to follow their changes. However, in the previous work, the class of neural network and clustering decoder can consider the neural connectivity regardless of the single neuronal tuning property. On the other hand, point process methods can model the single neuronal tuning property but fail to address the neural connectivity. In this paper, we propose a new point process decoder with the information of neural connectivity named NCPP. We derive the neural connectivity component from the point process method by Bayes' rule and use a clustering decoder to represent the neural connectivity. This method can consider the neural connectivity and the single neuronal tuning property at the same time. We validate the method on simulation data where the point process method cannot achieve a good decoding performance and compare it with sequential Monte Carlo point process method (SMCPP). The results show our method outperforms the pure point process method which indicates our method can model the neural connectivity and single neuronal tuning property at the same time.Clinical Relevance-This paper proposes a decoder that can model the neural connectivity and the single neuronal tuning property at the same time, which is potential to explain the neural adaptation computationally.

Corrections to "Task Learning Over Multi-Day Recording via Internally Rewarded Reinforcement Learning Based Brain Machine Interfaces".

In the above article [1], in the second paragraph of Section II-A, the randomly selected low-pitch audio cue was 1.5 kHz (not 4 kHz). Thus, the article should state "the two-lever-press discrimination task trials were initialized by a high-pitch (10 kHz) or low-pitch (1.5 kHz) audio cue, which was randomly generated."

Tracking Fast Neural Adaptation by Globally Adaptive Point Process Estimation for Brain-Machine Interface.

Brain-machine interfaces (BMIs) help the disabled restore body functions by translating neural activity into digital commands to control external devices. Neural adaptation, where the brain signals change in response to external stimuli or movements, plays an important role in BMIs. When subjects purely use neural activity to brain-control a prosthesis, some neurons will actively explore a new tuning property to accomplish the movement task. The prediction of this neural tuning property can help subjects adapt more efficiently to brain control and maintain a good decoding performance. Existing prediction methods track the slow change of the tuning property in the manual control, which is not suitable for the fast neural adaptation in brain control. In order to identify the active neurons in brain control and track their tuning property changes, we propose a globally adaptive point process method (GaPP) to estimate the neural modulation state from spike trains, decompose the states into the hyper preferred direction and reconstruct the kinematics in a dual-model framework. We implement the method on real data from rats performing a two-lever discrimination task under manual control and brain control. The results show our method successfully predicts the neural modulation state and identifies the neurons that become active in brain control. Compared to the existing method, ours tracks the fast changes of the hyper preferred direction from manual control to brain control more accurately and efficiently and reconstructs the kinematics better and faster.

Estimating Neural Modulation via Adaptive Point Process Method in Brain-machine Interface.

Brain-machine interfaces (BMIs) translate neural signals into digital commands to control external devices. During the use of BMI, neurons may change their activity corresponding to the same stimuli or movement. The changes are represented by the neural tuning parameters which may change gradually and abruptly. Adaptive algorithms were proposed to estimate the time-varying parameters in order to keep decoding performance stable. The existing methods only searched new parameters locally which failed to detect the abrupt changes. Global search helps but requires the known boundary of estimated parameter which is hard to be defined in many cases. We propose to estimate the neural modulation parameter by the global search using adaptive point process estimation. This neural modulation parameter represents the similarity between the kinematics and the neural preferred hyper tuning direction with finite range [0,1]. The preferred hyper tuning direction is then decoupled from the neural modulation parameter by gradient descent method. We apply the proposed method on real data to detect the abrupt change of the neural tuning parameter when the subject switched from manual control to brain control mode. The proposed method demonstrates better tracking on the neural hyper tuning parameters than local searching method and validated by KS statistical test.

Reinforcement Learning based Decoding Using Internal Reward for Time Delayed Task in Brain Machine Interfaces.

Reinforcement learning (RL) algorithm interprets neural signals into movement intentions with the guidance of the reward in Brain-machine interfaces (BMIs). Current RL algorithms generally work for the tasks with immediate rewards delivery, and lack of efficiency in delayed reward task. Prefrontal cortex, including medial prefrontal cortex(mPFC), has been demonstrated to assign credit to intermediate steps, which reinforces preceding action more efficiently. In this paper, we propose to simulate the functionality of mPFC activities as intermediate rewards to train a RL based decoder in a two-step movement task. A support vector machine (SVM) is adopted to verify if the subject expects a reward due to the accomplishment of a subtask from mPFC activity. Then this discrimination result will be utilized to guide the training of the RL decoder for each step respectively. Here, we apply the Sarsa-style attention-gated reinforcement learning (SAGREL) as the decoder to interpret motor cortex(M1) activity to action states. We test on in vivo primary motor cortex (M1) and mPFC data collected from rats, where the rats need to first trigger the start and then press lever for rewards using M1 signals. SAGREL using intermediate rewards from mPFC activities achieves a prediction accuracy of 66.8% ± 2.0.% (mean ± std) %, which is significantly better than the one using the reward by the end of trial (45.9.% ± 1.2%). This reveals the potentials of modelling mPFC activities as intermediate rewards for the delayed reward tasks.

Task Learning Over Multi-Day Recording via Internally Rewarded Reinforcement Learning Based Brain Machine Interfaces.

Autonomous brain machine interfaces (BMIs) aim to enable paralyzed people to self-evaluate their movement intention to control external devices. Previous reinforcement learning (RL)-based decoders interpret the mapping between neural activity and movements using the external reward for well-trained subjects, and have not investigated the task learning procedure. The brain has developed a learning mechanism to identify the correct actions that lead to rewards in the new task. This internal guidance can be utilized to replace the external reference to advance BMIs as an autonomous system. In this study, we propose to build an internally rewarded reinforcement learning-based BMI framework using the multi-site recording to demonstrate the autonomous learning ability of the BMI decoder on the new task. We test the model on the neural data collected over multiple days while the rats were learning a new lever discrimination task. The primary motor cortex (M1) and medial prefrontal cortex (mPFC) spikes are interpreted by the proposed RL framework into the discrete lever press actions. The neural activity of the mPFC post the action duration is interpreted as the internal reward information, where a support vector machine is implemented to classify the reward vs. non-reward trials with a high accuracy of 87.5% across subjects. This internal reward is used to replace the external water reward to update the decoder, which is able to adapt to the nonstationary neural activity during subject learning. The multi-cortical recording allows us to take in more cortical recordings as input and uses internal critics to guide the decoder learning. Comparing with the classic decoder using M1 activity as the only input and external guidance, the proposed system with multi-cortical recordings shows a better decoding accuracy. More importantly, our internally rewarded decoder demonstrates the autonomous learning ability on the new task as the decoder successfully addresses the time-variant neural patterns while subjects are learning, and works asymptotically as the subjects' behavioral learning progresses. It reveals the potential of endowing BMIs with autonomous task learning ability in the RL framework.