Guided Cooperation in Hierarchical Reinforcement Learning via Model-Based Rollout.

Goal-conditioned hierarchical reinforcement learning (HRL) presents a promising approach for enabling effective exploration in complex, long-horizon reinforcement learning (RL) tasks through temporal abstraction. Empirically, heightened interlevel communication and coordination can induce more stable and robust policy improvement in hierarchical systems. Yet, most existing goal-conditioned HRL algorithms have primarily focused on the subgoal discovery, neglecting interlevel cooperation. Here, we propose a novel goal-conditioned HRL framework named Guided Cooperation via Model-Based Rollout (GCMR; code is available at https://github.com/HaoranWang-TJ/GCMR_ACLG_official), aiming to bridge interlayer information synchronization and cooperation by exploiting forward dynamics. First, the GCMR mitigates the state-transition error within off-policy correction via model-based rollout, thereby enhancing sample efficiency. Second, to prevent disruption by the unseen subgoals and states, lower level Q -function gradients are constrained using a gradient penalty with a model-inferred upper bound, leading to a more stable behavioral policy conducive to effective exploration. Third, we propose a one-step rollout-based planning, using higher level critics to guide the lower level policy. Specifically, we estimate the value of future states of the lower level policy using the higher level critic function, thereby transmitting global task information downward to avoid local pitfalls. These three critical components in GCMR are expected to facilitate interlevel cooperation significantly. Experimental results demonstrate that incorporating the proposed GCMR framework with a disentangled variant of hierarchical reinforcement learning guided by landmarks (HIGL), namely, adjacency constraint and landmark-guided planning (ACLG), yields more stable and robust policy improvement compared with various baselines and significantly outperforms previous state-of-the-art (SOTA) algorithms.

Different Neural Information Flows Affected by Activity Patterns for Action and Verb Generation.

Shared brain regions have been found for processing action and language, including the left inferior frontal gyrus (IFG), the premotor cortex (PMC), and the inferior parietal lobule (IPL). However, in the context of action and language generation that shares the same action semantics, it is unclear whether the activity patterns within the overlapping brain regions would be the same. The changes in effective connectivity affected by these activity patterns are also unclear. In this fMRI study, participants were asked to perform hand action and verb generation tasks toward object pictures. We identified shared and specific brain regions for the two tasks in the left PMC, IFG, and IPL. The mean activation level and multi-voxel pattern analysis revealed that the activity patterns in the shared sub-regions were distinct for the two tasks. The dynamic causal modeling results demonstrated that the information flows for the two tasks were different across the shared sub-regions. These results provided the first neuroimaging evidence that the action and verb generation were task context driven in the shared regions, and the distinct patterns of neural information flow across the PMC-IFG-IPL neural network were affected by the polymodal processing in the shared regions.

Cross-Subject Assistance: Inter- and Intra-Subject Maximal Correlation for Enhancing the Performance of SSVEP-Based BCIs.

OBJECTIVE: The current state-of-the-art methods significantly improve the detection performance of the steady-state visual evoked potentials (SSVEPs) by using the individual calibration data. However, the time-consuming calibration sessions limit the number of training trials and may give rise to visual fatigue, which weakens the effectiveness of the individual training data. For addressing this issue, this study proposes a novel inter- and intra-subject maximal correlation (IISMC) method to enhance the robustness of SSVEP recognition via employing the inter- and intra-subject similarity and variability. Through efficient transfer learning, similar experience under the same task is shared across subjects. METHODS: IISMC extracts subject-specific information and similar task-related information from oneself and other subjects performing the same task by maximizing the inter- and intra-subject correlation. Multiple weak classifiers are built from several existing subjects and then integrated to construct the strong classifiers by the average weighting. Finally, a powerful fusion predictor is obtained for target recognition. RESULTS: The proposed framework is validated on a benchmark data set of 35 subjects, and the experimental results demonstrate that IISMC obtains better performance than the state of the art task-related component analysis (TRCA). SIGNIFICANCE: The proposed method has great potential for developing high-speed BCIs.

Inter- and Intra-subject Template-Based Multivariate Synchronization Index Using an Adaptive Threshold for SSVEP-Based BCIs.

The steady-state visually evoked potential (SSVEP) has been widely used in brain-computer interfaces (BCIs). Many studies have proved that the Multivariate synchronization index (MSI) is an efficient method for recognizing the frequency components in SSVEP-based BCIs. Despite its success, the recognition accuracy has not been satisfactory because the simplified pre-constructed sine-cosine waves lack abundant features from the real electroencephalogram (EEG) data. Recent advances in addressing this issue have achieved a significant improvement in recognition accuracy by using individual calibration data. In this study, a new extension based on inter- and intra-subject template signals is introduced to improve the performance of the standard MSI method. Through template transfer, inter-subject similarity and variability are employed to enhance the robustness of SSVEP recognition. Additionally, most existed methods for SSVEP recognition utilize a fixed time window (TW) to perform frequency domain analysis, which limits the information transfer rate (ITR) of BCIs. For addressing this problem, a novel adaptive threshold strategy is integrated into the extension of MSI, which uses a dynamic window to extract the temporal features of SSVEPs and recognizes the stimulus frequency based on a pre-set threshold. The pre-set threshold contributes to obtaining an appropriate and shorter signal length for frequency recognition and filtering ignored-invalid trials. The proposed method is evaluated on a 12-class SSVEP dataset recorded from 10 subjects, and the result shows that this achieves higher recognition accuracy and information transfer rate when compared with the CCA, MSI, Multi-set CCA, and Individual Template-based CCA. This paper demonstrates that the proposed method is a promising approach for developing high-speed BCIs.

The role of the precuneus and posterior cingulate cortex in the neural routes to action.

Neural substrates of action to the object or this specific direct route, however, remain unclear, especially for the connection from the visual pathway to the motor cortex. The study examined this issue by conducting an fMRI experiment, in which two action generation tasks involving pictures of real objects (PA) and the object's nouns (NA) were used, with pictures naming (PN) and covert noun reading (NR) being the control tasks. The result showed that the model predefined for the PCC and precuneus connecting IPL to the posterior-medial frontal cortex dominated over the others (with 0.45 probability), suggesting that the PCC and the precuneus locate at the neural substrates of action to the object. Furthermore, a feasibility study suggests that the neural pathway composed of the V3/MT, precuneus, PCC, and PM (premotor cortex) forms the direct route from perception to action, which also links to the dorsal pathway so that the perception of objects bypasses the semantic ventral pathway and then directly cues actions via the affordance.

Dynamical Motor Control Learned with Deep Deterministic Policy Gradient.

Conventional models of motor control exploit the spatial representation of the controlled system to generate control commands. Typically, the control command is gained with the feedback state of a specific instant in time, which behaves like an optimal regulator or spatial filter to the feedback state. Yet, recent neuroscience studies found that the motor network may constitute an autonomous dynamical system and the temporal patterns of the control command can be contained in the dynamics of the motor network, that is, the dynamical system hypothesis (DSH). Inspired by these findings, here we propose a computational model that incorporates this neural mechanism, in which the control command could be unfolded from a dynamical controller whose initial state is specified with the task parameters. The model is trained in a trial-and-error manner in the framework of deep deterministic policy gradient (DDPG). The experimental results show that the dynamical controller successfully learns the control policy for arm reaching movements, while the analysis of the internal activities of the dynamical controller provides the computational evidence to the DSH of the neural coding in motor cortices.

Representation of action semantics in the motor cortex and Broca's area.

Previous studies have shown that both reading action words and observing actions engage the motor cortex and Broca's area, but it is still controversial whether a somatotopic representation exists for action verbs within the motor cortex and whether Broca's area encodes action-specific semantics for verbs. Here we examined these two issues using a set of functional MRI experiments, including word reading, action observation and a movement localiser task. Results from multi-voxel pattern analysis (MVPA) showed a somatotopic organisation within the motor areas and action-specific activation in Broca's area for observed actions, suggesting the representation of action semantics for observed actions in these neural regions. For action verbs, however, a lack of finding for the somatotopic activation argues against semantic somatotopy within the motor cortex. Furthermore, activation patterns in Broca's area were not separable between action verbs and unrelated verbs, suggesting that Broca's area does not encode action-specific semantics for verbs.

Covert Verb Reading Contributes to Signal Classification of Motor Imagery in BCI.

Motor imagery is widely used in the brain-computer interface (BCI) systems that can help people actively control devices to directly communicate with the external world, but its training and performance effect is usually poor for normal people. To improve operators' BCI performances, here we proposed a novel paradigm, which combined the covert verb reading in the traditional motor imagery paradigm. In our proposed paradigm, participants were asked to covertly read the presented verbs during imagining right hand or foot movements referred by those verbs. EEG signals were recorded with both our proposed paradigm and the traditional paradigm. By the common spatial pattern method, we, respectively, decomposed these signals into spatial patterns and extracted their features used in the following classification of support vector machine. Compared with the traditional paradigm, our proposed paradigm could generate clearer spatial patterns following a somatotopic distribution, which led to more distinguishable features and higher classification accuracies than those in the traditional paradigm. These results suggested that semantic processing of verbs can influence the brain activity of motor imagery and enhance the mu event-related desynchronisation. The combination of semantic processing with motor imagery is therefore a promising method for the improvement of operators' BCI performances.

Potential for false positive results from multi-voxel pattern analysis on functional imaging data.

BACKGROUND: Multi-voxel pattern analysis (MVPA) provides a powerful tool to investigate neural mechanisms for various cognitive processes under functional brain imaging. However, the high sensitivity of the MVPA method could bring about false positive results, which has been overlooked by previous research. OBJECTIVE: We investigated the potential for obtaining false positives from the MVPA method. METHODS: We conducted MVPA on a public functional MRI dataset on the neural encoding of various object categories. Different scenarios for pattern classification were involved by varying the number of voxels for each region of interest (ROI) and the number of object categories. RESULTS: The classification accuracy became higher with more voxels involved, and false positive results emerged for the primary auditory cortex and even a white matter ROI, where object-related neural processing was not supposed to occur. CONCLUSIONS: Our results imply that the classification accuracy obtained from MVPA may be inflated due to the high sensitivity of the method. Therefore, we suggest involving control ROIs in future MVPA studies and comparing the classification accuracy for a target ROI with that for a control ROI, instead of comparing the obtained accuracy with the chance-level accuracy.

Different activity patterns for action and language within their shared neural areas: An fMRI study on action observation and language phonology.

The neural processes for action and language activate shared brain regions including the left inferior frontal, parietal and temporal-occipital cortices. However, it still remains unclear how action and language are related and what neural activity patterns are elicited within these shared cortical regions. In this study we examined the neural activation for action observation and language phonology in their shared cortical regions by conducting three experiments in a single fMRI session: a mixed-task experiment involving both action and language phonological processing, and two independent experiments involving language phonology and action observation respectively. To control for differences in the visual processing and to enable a direct comparison between the tasks, the same visual stimuli were used for the mixed-tasks. Common neural areas for action observation and language phonology were located in the junction of the left inferior frontal/precentral gyrus, the left intraparietal sulcus and the left temporal-occipital cortex. Nevertheless, multi-voxel pattern analysis on the shared neural areas revealed that different patterns of neural activity were elicited for the action and language phonological tasks. Our results provide the first neuroimaging evidence that the common neural structures are engaged differently by action and language phonological processing.

Face Context Influences Local Part Processing: An ERP Study.

Perception of face parts on the basis of features is thought to be different from perception of whole faces, which is more based on configural information. Face context is also suggested to play an important role in face processing. To investigate how face context influences the early-stage perception of facial local parts, we used an oddball paradigm that tested perceptual stages of face processing rather than recognition. We recorded the event-related potentials (ERPs) elicited by whole faces and face parts presented in four conditions (upright-normal, upright-thatcherised, inverted-normal and inverted-thatcherised), as well as the ERPs elicited by non-face objects (whole houses and house parts) with corresponding conditions. The results showed that face context significantly affected the N170 with increased amplitudes and earlier peak latency for upright normal faces. Removing face context delayed the P1 latency but did not affect the P1 amplitude prominently for both upright and inverted normal faces. Across all conditions, neither the N170 nor the P1 was modulated by house context. The significant changes on the N170 and P1 components revealed that face context influences local part processing at the early stage of face processing and this context effect might be specific for face perception. We further suggested that perceptions of whole faces and face parts are functionally distinguished.

Perceiving object affordances through visual and linguistic pathways: A comparative study.

It is known that both perceiving visual objects and reading object names automatically activate associated motor codes and modulate motor responses. We examined the nature of these motor activation effects for different effectors (hands and feet), and for pictures and words, across the time course of responding. The compatibility effects elicited by objects and words were comparable for the mean effect size, both were larger for slow than for fast responses and the effects were positively correlated across the stimulus types. Our results support an embodied cognition account in which the perception of objects and words automatically activates perceptual simulations of the associated actions, suggesting that objects and words share cognitive and neural mechanisms for accessing motor codes. However, the compatibility effects for objects and words carried over across trials differently: the compatibility effect for words was sensitive to a previous response, while the effect for objects was more immune to such influence. This result suggests a stronger link between objects and actions through a visual pathway than through a linguistic pathway.

Automatic correction of hand pointing in stereoscopic depth.

In order to examine whether stereoscopic depth information could drive fast automatic correction of hand pointing, an experiment was designed in a 3D visual environment in which participants were asked to point to a target at different stereoscopic depths as quickly and accurately as possible within a limited time window (≤300 ms). The experiment consisted of two tasks: "depthGO" in which participants were asked to point to the new target position if the target jumped, and "depthSTOP" in which participants were instructed to abort their ongoing movements after the target jumped. The depth jump was designed to occur in 20% of the trials in both tasks. Results showed that fast automatic correction of hand movements could be driven by stereoscopic depth to occur in as early as 190 ms.

Electroencephalogram evidence for the activation of human mirror neuron system during the observation of intransitive shadow and line drawing actions.

Previous studies have demonstrated that hand shadows may activate the motor cortex associated with the mirror neuron system in human brain. However, there is no evidence of activity of the human mirror neuron system during the observation of intransitive movements by shadows and line drawings of hands. This study examined the suppression of electroencephalography mu waves (8-13 Hz) induced by observation of stimuli in 18 healthy students. Three stimuli were used: real hand actions, hand shadow actions and actions made by line drawings of hands. The results showed significant desynchronization of the mu rhythm ("mu suppression") across the sensorimotor cortex (recorded at C3, Cz and C4), the frontal cortex (recorded at F3, Fz and F4) and the central and right posterior parietal cortex (recorded at Pz and P4) under all three conditions. Our experimental findings suggest that the observation of "impoverished hand actions", such as intransitive movements of shadows and line drawings of hands, is able to activate widespread cortical areas related to the putative human mirror neuron system.

Mirror neural training induced by virtual reality in brain-computer interfaces may provide a promising approach for the autism therapy.

Previous studies have suggested that the dysfunction of the human mirror neuron system (hMNS) plays an important role in the autism spectrum disorder (ASD). In this work, we propose a novel training program from our interdisciplinary research to improve mirror neuron functions of autistic individuals by using a BCI system with virtual reality technology. It is a promising approach for the autism to learn and develop social communications in a VR environment. A test method for this hypothesis is also provided.

On-line 'automatic pilot' training for hand and arm motor rehabilitation after stroke.

As stroke being one of the most leading causes of death worldwide, even stroke survivors have to suffer from dysfunctions of limb controls and inabilities of speech or vision. Cognitive neuroscientists have found various forms of automatic behaviours in healthy people, which generally cover motor components of upper limbs and are essential for coordination and mobility relevant activities. Meanwhile, the robot-assisted therapy and functional electrical stimulation have become prominent rehabilitation techniques for patients' rehabilitation after stroke. With the integration of robot-aided therapeutic systems and the functional electrical stimulation, the on-line 'automatic pilot' training of the visual inspired stimulation for upper limbs can offer a feasible treatment for patients after stroke to recover motor performance.

Self-organizing and adaptive peer-to-peer network.

In this paper, an algorithm that forms a dynamic and self-organizing network is demonstrated. The hypothesis of this work is that in order to achieve a resilient and adaptive peer-to-peer (P2P) network, each network node must proactively maintain a minimum number of edges. Specifically, low-level communication protocols are not sufficient by themselves to achieve high-service availability, especially in the case of ad hoc or dynamic networks with a high degree of node addition and deletion. The concept has been evaluated within a P2P agent application in which each agent has a goal to maintain a preferred number of connections to a number of service providing agents. Using this algorithm, the agents update a weight value associated with each connection, based on the perceived utility of the connection to the corresponding agent. This utility function can be a combination of several node or edge parameters, such as degree k of the target node, or frequency of the message response from the node. This weight is updated using a set of Hebbian-style learning rules, such that the network as a whole exhibits adaptive self-organizing behavior. The principal result is the finding that by limiting the connection neighborhood within the overlay topology, the resulting P2P network can be made highly resilient to targeted attacks on high-degree nodes, while maintaining search efficiency.

Structure of peer-to-peer social networks.

This paper presents a statistical analysis of the structure of peer-to-peer (P2P) social networks that captures social associations of distributed peers in resource sharing. Peer social networks appear to be mainly composed of pure resource providers that guarantee high resource availability and reliability of P2P systems. The major peers that both provide and request resources are only a small fraction. The connectivity between peers, including undirected, directed (out and in) and weighted connections, is scale-free and the social networks of all peers and major peers are small world networks. The analysis also confirms that peer social networks show in general disassortative correlations, except that active providers are connected between each other and by active requesters. The study presented in this paper gives a better understanding of peer relationships in resource sharing, which may help a better design of future P2P networks and open the path to the study of transport processes on top of real P2P topologies.