Neuro-Inspired Reinforcement Learning to Improve Trajectory Prediction in Reward-Guided Behavior.

Hippocampal pyramidal cells and interneurons play a key role in spatial navigation. In goal-directed behavior associated with rewards, the spatial firing pattern of pyramidal cells is modulated by the animal's moving direction toward a reward, with a dependence on auditory, olfactory, and somatosensory stimuli for head orientation. Additionally, interneurons in the CA1 region of the hippocampus monosynaptically connected to CA1 pyramidal cells are modulated by a complex set of interacting brain regions related to reward and recall. The computational method of reinforcement learning (RL) has been widely used to investigate spatial navigation, which in turn has been increasingly used to study rodent learning associated with the reward. The rewards in RL are used for discovering a desired behavior through the integration of two streams of neural activity: trial-and-error interactions with the external environment to achieve a goal, and the intrinsic motivation primarily driven by brain reward system to accelerate learning. Recognizing the potential benefit of the neural representation of this reward design for novel RL architectures, we propose a RL algorithm based on [Formula: see text]-learning with a perspective on biomimetics (neuro-inspired RL) to decode rodent movement trajectories. The reward function, inspired by the neuronal information processing uncovered in the hippocampus, combines the preferred direction of pyramidal cell firing as the extrinsic reward signal with the coupling between pyramidal cell-interneuron pairs as the intrinsic reward signal. Our experimental results demonstrate that the neuro-inspired RL, with a combined use of extrinsic and intrinsic rewards, outperforms other spatial decoding algorithms, including RL methods that use a single reward function. The new RL algorithm could help accelerate learning convergence rates and improve the prediction accuracy for moving trajectories.

Quantification Analysis of Sleep Based on Smartwatch Sensors for Parkinson's Disease.

Rapid eye movement (REM) sleep behavior disorder (RBD) is associated with Parkinson's disease (PD). In this study, a smartwatch-based sensor is utilized as a convenient tool to detect the abnormal RBD phenomenon in PD patients. Instead, a questionnaire with sleep quality assessment and sleep physiological indices, such as sleep stage, activity level, and heart rate, were measured in the smartwatch sensors. Therefore, this device can record comprehensive sleep physiological data, offering several advantages such as ubiquity, long-term monitoring, and wearable convenience. In addition, it can provide the clinical doctor with sufficient information on the patient's sleeping patterns with individualized treatment. In this study, a three-stage sleep staging method (i.e., comprising sleep/awake detection, sleep-stage detection, and REM-stage detection) based on an accelerometer and heart-rate data is implemented using machine learning (ML) techniques. The ML-based algorithms used here for sleep/awake detection, sleep-stage detection, and REM-stage detection were a Cole-Kripke algorithm, a stepwise clustering algorithm, and a k-means clustering algorithm with predefined criteria, respectively. The sleep staging method was validated in a clinical trial. The results showed a statistically significant difference in the percentage of abnormal REM between the control group (1.6 ± 1.3; n = 18) and the PD group (3.8 ± 5.0; n = 20) (p = 0.04). The percentage of deep sleep stage in our results presented a significant difference between the control group (38.1 ± 24.3; n = 18) and PD group (22.0 ± 15.0, n = 20) (p = 0.011) as well. Further, our results suggested that the smartwatch-based sensor was able to detect the difference of an abnormal REM percentage in the control group (1.6 ± 1.3; n = 18), PD patient with clonazepam (2.0 ± 1.7; n = 10), and without clonazepam (5.7 ± 7.1; n = 10) (p = 0.007). Our results confirmed the effectiveness of our sensor in investigating the sleep stage in PD patients. The sensor also successfully determined the effect of clonazepam on reducing abnormal REM in PD patients. In conclusion, our smartwatch sensor is a convenient and effective tool for sleep quantification analysis in PD patients.