Optogenetic Editing Reveals the Hierarchical Organization of Learned Action Sequences.

The organization of action into sequences underlies complex behaviors that are essential for organismal survival and reproduction. Despite extensive studies of innate sequences in relation to central pattern generators, how learned action sequences are controlled and whether they are organized as a chain or a hierarchy remain largely unknown. By training mice to perform heterogeneous action sequences, we demonstrate that striatal direct and indirect pathways preferentially encode different behavioral levels of sequence structure. State-dependent closed-loop optogenetic stimulation of the striatal direct pathway can selectively insert a single action element into the sequence without disrupting the overall sequence length. Optogenetic manipulation of the striatal indirect pathway completely removes the ongoing subsequence while leaving the following subsequence to be executed with the appropriate timing and length. These results suggest that learned action sequences are not organized in a serial but rather a hierarchical structure that is distinctly controlled by basal ganglia pathways.

Joint modeling of action sequences and action time in computer-based interactive tasks.

Process data refers to data recorded in computer-based assessments that reflect the problem-solving processes of participants and provide greater insight into how they solve problems. Action time, namely the amount of time required to complete a state transition, is also included in such data along with actions. In this study, an action-level joint model of action sequences and action time is proposed, in which the sequential response model (SRM) is used as the measurement model for action sequences, and a new log-normal action time model is proposed as the measurement model for action time. The proposed model can be regarded as an extension of the SRM by incorporating action time within the joint-hierarchical modeling framework and as an extension of the conventional item-level joint models in process data analysis. Results of the empirical and simulation studies demonstrated that the model setup was justified, model parameters could be interpreted, parameter estimates were accurate, and taking into account participants' action time further was beneficial for obtaining a deep understanding of participants' behavioral patterns. Overall, the proposed action-level joint model provides an innovative modeling framework for analyzing process data in computer-based assessments from the latent variable modeling perspective.

This is not who you are: The posterior cerebellum and stereotype-inconsistent action sequences.

Recent research has indicated that the posterior cerebellum plays a crucial role in social cognition by encoding sequences of social actions. This study investigates its role in learning sequences of stereotype-implying actions by group members. We presented a set of five sentences that each described a group member who performed either stereotype-consistent or inconsistent actions. Participants were instructed to memorize the temporal order of the sentences and infer a common stereotype of the group. As a comparison, we included control conditions where participants had to memorize sequences of nonsocial consistent events or simply read stereotype-consistent sentences without memorizing their order. The results showed that the posterior cerebellum was strongly activated when participants were memorizing the order of the social actions, as opposed to simply reading these social actions. More importantly, when the social actions were inconsistent as opposed to consistent with the stereotype of the group, the posterior cerebellum was activated more strongly. This activation occurred together with cortical recruitment of the mentalizing network involving the dorsomedial prefrontal cortex (dmPFC) during social actions, and additionally the conflict monitoring network involving the lateral prefrontal cortex (PFC) and posterior medial frontal cortex (pmFC) during stereotype-inconsistent actions. These findings suggest that the cerebellum supports not only learning of low-level action sequences, but also of their high-level social implications.

Chemogenetic inhibition in the dorsal striatum reveals regional specificity of direct and indirect pathway control of action sequencing.

Animals engage in intricate action sequences that are constructed during instrumental learning. There is broad consensus that the basal ganglia play a crucial role in the formation and fluid performance of action sequences. To investigate the role of the basal ganglia direct and indirect pathways in action sequencing, we virally expressed Cre-dependent Gi-DREADDs in either the dorsomedial (DMS) or dorsolateral (DLS) striatum during and/or after action sequence learning in D1 and D2 Cre rats. Action sequence performance in D1 Cre rats was slowed down early in training when DREADDs were activated in the DMS, but sped up when activated in the DLS. Acquisition of the reinforced sequence was hindered when DREADDs were activated in the DLS of D2 Cre rats. Outcome devaluation tests conducted after training revealed that the goal-directed control of action sequence rates was immune to chemogenetic inhibition-rats suppressed the rate of sequence performance when rewards were devalued. Sequence initiation latencies were generally sensitive to outcome devaluation, except in the case where DREADD activation was removed in D2 Cre rats that previously experienced DREADD activation in the DMS during training. Sequence completion latencies were generally not sensitive to outcome devaluation, except in the case where D1 Cre rats experienced DREADD activation in the DMS during training and test. Collectively, these results suggest that the indirect pathway originating from the DLS is part of a circuit involved in the effective reinforcement of action sequences, while the direct and indirect pathways originating from the DMS contribute to the goal-directed control of sequence completion and initiation, respectively.

Sequential actions: effects of upcoming perceptual and motor tasks on current actions.

Reports of spatial interactions between current and upcoming elements in a movement sequence could be due to intentional planning of a "global" action sequence (i.e., strategic effects), or to unintentional motor planning arising from merely paying attention to upcoming target objects (i.e., interference effects). The purpose of this experiment was to determine whether paying attention to an upcoming target object could induce kinematic changes in a current grasping action when there is no strategic advantage associated with such changes. Specifically, participants grasped a rectangular target object in the presence of a second rectangular object that was either ignored, or was the target of a subsequent grasping or perceptual judgment task. Peak grip aperture during the primary grasping action was assessed in relation to the size of the second rectangle. The results revealed an effect of the second rectangle's size only when it was the target of a subsequent perceptual judgment task. This result calls into question the assumption that interactions between current and subsequent elements of an action sequence are necessarily due to strategic movement planning processes and might instead arise from interference arising from merely paying attention to nontarget objects.

Sequential action-induced invariant representation for reinforcement learning.

How to accurately learn task-relevant state representations from high-dimensional observations with visual distractions is a realistic and challenging problem in visual reinforcement learning. Recently, unsupervised representation learning methods based on bisimulation metrics, contrast, prediction, and reconstruction have shown the ability for task-relevant information extraction. However, due to the lack of appropriate mechanisms for the extraction of task information in the prediction, contrast, and reconstruction-related approaches and the limitations of bisimulation-related methods in domains with sparse rewards, it is still difficult for these methods to be effectively extended to environments with distractions. To alleviate these problems, in the paper, the action sequences, which contain task-intensive signals, are incorporated into representation learning. Specifically, we propose a Sequential Action-induced invariant Representation (SAR) method, which decouples the controlled part (i.e., task-relevant information) and the uncontrolled part (i.e., task-irrelevant information) in noisy observations through sequential actions, thereby extracting effective representations related to decision tasks. To achieve it, the characteristic function of the action sequence's probability distribution is modeled to specifically optimize the state encoder. We conduct extensive experiments on the distracting DeepMind Control suite while achieving the best performance over strong baselines. We also demonstrate the effectiveness of our method at disregarding task-irrelevant information by applying SAR to real-world CARLA-based autonomous driving with natural distractions. Finally, we provide the analysis results of generalization drawn from the generalization decay and t-SNE visualization. Code and demo videos are available at https://github.com/DMU-XMU/SAR.git.

A delayed matching task-based study on action sequence of motor imagery.

The way people imagine greatly affects performance of brain-computer interface (BCI) based on motion imagery (MI). Action sequence is a basic unit of imitation, learning, and memory for motor behavior. Whether it influences the MI-BCI is unknown, and how to manifest this influence is difficult since the MI is a spontaneous brain activity. To investigate the influence of the action sequence, this study proposes a novel paradigm named action sequences observing and delayed matching task to use images and videos to guide people to observe, match and reinforce the memory of sequence. Seven subjects' ERPs and MI performance are analyzed under four different levels of complexities or orders of the sequence. Results demonstrated that the action sequence in terms of complexity and sequence order significantly affects the MI. The complex action in positive order obtains stronger ERD/ERS and more pronounced MI feature distributions, and yields an MI classification accuracy that is 12.3% higher than complex action in negative order (p < 0.05). In addition, the ERP amplitudes derived from the supplementary motor area show a positive correlation to the MI. This study demonstrates a new perspective of improving imagery in the MI-BCI by considering the complexity and order of the action sequences, and provides a novel index for manifesting the MI performance by ERP.

Biclustering of Log Data: Insights from a Computer-Based Complex Problem Solving Assessment.

Computer-based assessments provide the opportunity to collect a new source of behavioral data related to the problem-solving process, known as log file data. To understand the behavioral patterns that can be uncovered from these process data, many studies have employed clustering methods. In contrast to one-mode clustering algorithms, this study utilized biclustering methods, enabling simultaneous classification of test takers and features extracted from log files. By applying the biclustering algorithms to the "Ticket" task in the PISA 2012 CPS assessment, we evaluated the potential of biclustering algorithms in identifying and interpreting homogeneous biclusters from the process data. Compared with one-mode clustering algorithms, the biclustering methods could uncover clusters of individuals who are homogeneous on a subset of feature variables, holding promise for gaining fine-grained insights into students' problem-solving behavior patterns. Empirical results revealed that specific subsets of features played a crucial role in identifying biclusters. Additionally, the study explored the utilization of biclustering on both the action sequence data and timing data, and the inclusion of time-based features enhanced the understanding of students' action sequences and scores in the context of the analysis.

Corrigendum: Interaction between habits as action sequences and goal-directed behavior under time pressure.

[This corrects the article DOI: 10.3389/fnins.2022.996957.].

Dopaminergic D2 receptor modulation of striatal cholinergic interneurons contributes to sequence learning.

Learning action sequences is necessary for normal daily activities. Medium spiny neurons (MSNs) in the dorsal striatum (dStr) encode action sequences through changes in firing at the start and/or stop of action sequences or sustained changes in firing throughout the sequence. Acetylcholine (ACh), released from cholinergic interneurons (ChIs), regulates striatal function by modulating MSN and interneuron excitability, dopamine and glutamate release, and synaptic plasticity. Cholinergic neurons in dStr pause their tonic firing during the performance of learned action sequences. Activation of dopamine type-2 receptors (D2Rs) on ChIs is one mechanism of ChI pausing. In this study we show that deleting D2Rs from ChIs by crossing D2-floxed with ChAT-Cre mice (D2Flox-ChATCre), which inhibits dopamine-mediated ChI pausing and leads to deficits in an operant action sequence task and lower breakpoints in a progressive ratio task. These data suggest that D2Flox-ChATCre mice have reduced motivation to work for sucrose reward, but show no generalized motor skill deficits. D2Flox-ChATCre mice perform similarly to controls in a simple reversal learning task, indicating normal behavioral flexibility, a cognitive function associated with ChIs. In vivo electrophysiological recordings show that D2Flox-ChatCre mice have deficits in sequence encoding, with fewer dStr MSNs encoding entire action sequences compared to controls. Thus, ChI D2R deletion appears to impair a neural substrate of action chunking. Virally replacing D2Rs in dStr ChIs in adult mice improves action sequence learning, but not the lower breakpoints, further suggesting that D2Rs on ChIs in the dStr are critical for sequence learning, but not for driving the motivational aspects of the task.

Dorsal striatum coding for the timely execution of action sequences.

The automatic initiation of actions can be highly functional. But occasionally these actions cannot be withheld and are released at inappropriate times, impulsively. Striatal activity has been shown to participate in the timing of action sequence initiation and it has been linked to impulsivity. Using a self-initiated task, we trained adult male rats to withhold a rewarded action sequence until a waiting time interval has elapsed. By analyzing neuronal activity we show that the striatal response preceding the initiation of the learned sequence is strongly modulated by the time subjects wait before eliciting the sequence. Interestingly, the modulation is steeper in adolescent rats, which show a strong prevalence of impulsive responses compared to adults. We hypothesize this anticipatory striatal activity reflects the animals' subjective reward expectation, based on the elapsed waiting time, while the steeper waiting modulation in adolescence reflects age-related differences in temporal discounting, internal urgency states, or explore-exploit balance.

Interaction between habits as action sequences and goal-directed behavior under time pressure.

Human behavior consists in large parts of action sequences that are often repeated in mostly the same way. Through extensive repetition, sequential responses become automatic or habitual, but our environment often confronts us with events to which we have to react flexibly and in a goal-directed manner. To assess how implicitly learned action sequences interfere with goal-directed control, we developed a novel behavioral paradigm in which we combined action sequence learning through repetition with a goal-directed task component. So-called dual-target trials require the goal-directed selection of the response with the highest reward probability in a fast succession of trials with short response deadlines. Importantly, the response primed by the learned action sequence is sometimes different from that required by the goal-directed task. As expected, we found that participants learned the action sequence through repetition, as evidenced by reduced reaction times (RT) and error rates (ER), while still acting in a goal-directed manner in dual-target trials. Specifically, we found that the learned action sequence biased choices in the goal-directed task toward the sequential response, and this effect was more pronounced the better individuals had learned the sequence. Our novel task may help shed light on the acquisition of automatic behavioral patterns and habits through extensive repetition, allows to assess positive features of habitual behavior (e.g., increased response speed and reduced error rates), and importantly also the interaction of habitual and goal-directed behaviors under time pressure.

How the Mind Creates Structure: Hierarchical Learning of Action Sequences.

Humans have the astonishing capacity to quickly adapt to varying environmental demands and reach complex goals in the absence of extrinsic rewards. Part of what underlies this capacity is the ability to flexibly reuse and recombine previous experiences, and to plan future courses of action in a psychological space that is shaped by these experiences. Decades of research have suggested that humans use hierarchical representations for efficient planning and flexibility, but the origin of these representations has remained elusive. This study investigates how 73 participants learned hierarchical representations through experience, in a task in which they had to perform complex action sequences to obtain rewards. Complex action sequences were composed of simpler action sequences, which were not rewarded, but whose completion was signaled to participants. We investigated the process with which participants learned to perform simpler action sequences and combined them into complex action sequences. After learning action sequences, participants completed a transfer phase in which either simple sequences or complex sequences were manipulated without notice. Relearning progressed slower when simple than complex sequences were changed, in accordance with a hierarchical representations in which lower levels are quickly consolidated, potentially stabilizing exploration, while higher levels remain malleable, with benefits for flexible recombination.

Dorsomedial Striatal Activity Tracks Completion of Behavioral Sequences in Rats.

For proper execution of goal-directed behaviors, individuals require both a general representation of the goal and an ability to monitor their own progress toward that goal. Here, we examine how dorsomedial striatum (DMS), a region pivotal for forming associations among stimuli, actions, and outcomes, encodes the execution of goal-directed action sequences that require self-monitoring of behavior. We trained rats to complete a sequence of at least five consecutive lever presses (without visiting the reward port) to obtain a reward and recorded the activity of individual cells in DMS while rats performed the task. We found that the pattern of DMS activity gradually changed during the execution of the sequence, permitting accurate decoding of sequence progress from neural activity at a population level. Moreover, this sequence-related activity was blunted on trials where rats did not complete a sufficient number of presses. Overall, these data suggest a link between DMS activity and the execution of behavioral sequences that require monitoring of ongoing behavior.

Nigrostriatal dopamine signals sequence-specific action-outcome prediction errors.

Dopamine has been suggested to encode cue-reward prediction errors during Pavlovian conditioning, signaling discrepancies between actual versus expected reward predicted by the cues.1-5 While this theory has been widely applied to reinforcement learning concerning instrumental actions, whether dopamine represents action-outcome prediction errors and how it controls sequential behavior remain largely unknown. The vast majority of previous studies examining dopamine responses primarily have used discrete reward-predictive stimuli,1-15 whether Pavlovian conditioned stimuli for which no action is required to earn reward or explicit discriminative stimuli that essentially instruct an animal how and when to respond for reward. Here, by training mice to perform optogenetic intracranial self-stimulation, we examined how self-initiated goal-directed behavior influences nigrostriatal dopamine transmission during single and sequential instrumental actions, in behavioral contexts with minimal overt changes in the animal's external environment. We found that dopamine release evoked by direct optogenetic stimulation was dramatically reduced when delivered as the consequence of the animal's own action, relative to non-contingent passive stimulation. This dopamine suppression generalized to food rewards was specific to the reinforced action, was temporally restricted to counteract the expected outcome, and exhibited sequence-selectivity consistent with hierarchical control of sequential behavior. These findings demonstrate that nigrostriatal dopamine signals sequence-specific prediction errors in action-outcome associations, with fundamental implications for reinforcement learning and instrumental behavior in health and disease.

Contributions of the basal ganglia to action sequence learning and performance.

Animals engage in intricately woven and choreographed action sequences that are constructed from trial-and-error learning. The mechanisms by which the brain links together individual actions which are later recalled as fluid chains of behavior are not fully understood, but there is broad consensus that the basal ganglia play a crucial role in this process. This paper presents a comprehensive review of the role of the basal ganglia in action sequencing, with a focus on whether the computational framework of reinforcement learning can capture key behavioral features of sequencing and the neural mechanisms that underlie them. While a simple neurocomputational model of reinforcement learning can capture key features of action sequence learning, this model is not sufficient to capture goal-directed control of sequences or their hierarchical representation. The hierarchical structure of action sequences, in particular, poses a challenge for building better models of action sequencing, and it is in this regard that further investigations into basal ganglia information processing may be informative.

Reinforcement Learning With Parsimonious Computation and a Forgetting Process.

Decision-making is assumed to be supported by model-free and model-based systems: the model-free system is based purely on experience, while the model-based system uses a cognitive map of the environment and is more accurate. The recently developed multistep decision-making task and its computational model can dissociate the contributions of the two systems and have been used widely. This study used this task and model to understand our value-based learning process and tested alternative algorithms for the model-free and model-based learning systems. The task used in this study had a deterministic transition structure, and the degree of use of this structure in learning is estimated as the relative contribution of the model-based system to choices. We obtained data from 29 participants and fitted them with various computational models that differ in the model-free and model-based assumptions. The results of model comparison and parameter estimation showed that the participants update the value of action sequences and not each action. Additionally, the model fit was improved substantially by assuming that the learning mechanism includes a forgetting process, where the values of unselected options change to a certain default value over time. We also examined the relationships between the estimated parameters and psychopathology and other traits measured by self-reported questionnaires, and the results suggested that the difference in model assumptions can change the conclusion. In particular, inclusion of the forgetting process in the computational models had a strong impact on estimation of the weighting parameter of the model-free and model-based systems.

Co-actors represent the order of each other's actions.

Previous research has shown that people represent each other's tasks and actions when acting together. However, less is known about how co-actors represent each other's action sequences. Here, we asked whether co-actors represent the order of each other's actions within an action sequence, or whether they merely represent the intended end state of a joint action together with their own contribution. In the present study, two co-actors concurrently performed action sequences composed of two actions. We predicted that if co-actors represent the order of each other's actions, they should experience interference when the order of their actions differs. Supporting this prediction, the results of six experiments consistently showed that co-actors moved more slowly when performing the same actions in a different order compared to performing the same actions in the same order. In line with findings from bimanual movement tasks, our results indicate that interference can arise due to differences in movement parameters and due to differences in the perceptual characteristics of movement goals. The present findings extend previous research on co-representation, providing evidence that people represent not only the elements of another's task, but also their temporal structure.

The sweetness of successful goal pursuit: Approach-motivated pregoal states enhance the reward positivity during goal pursuit.

Traditionally, the reward positivity (RewP) is thought to index a binary performance monitoring system sensitive to approach motivation. However, recent theoretical models have argued that feedback processing extends beyond simple "good" vs. "bad" associations, such that performance monitoring incorporates the complex, multi-step sequence of behaviors often necessary to attain rewards. The present study sought to go beyond simple stimulus-response paradigms to examine how approach-motivated states occurring in multi-step goal pursuit impacts the RewP. Additionally, outcome frequency was varied to examine how the P3, a neural marker of expectancy, influences the RewP. Using a modified monetary incentive delay paradigm, participants played a reaction time game where multiple correct responses were required to attain a reward. Additionally, each trial had the potential for a reward (approach-motivated state) or no reward (neutral state). Results revealed that RewP amplitudes were larger after reward trial win feedback than after reward trial no-win feedback across multiple stages of goal pursuit. Additionally, after for controlling outcome frequency via the P3, RewP amplitudes were larger in reward trials than in neutral trials across incremental stages of goal pursuit. The RewP appears to be sensitive to feedback indicating successfully completing sub-goals during pursuit of a goal, even when no immediate reward is given. Approach motivation enhances performance monitoring when multiple steps are needed to attain a desired outcome, which may increase the likelihood of goal acquisition and attainment.

Striatal Neuropeptides Enhance Selection and Rejection of Sequential Actions.

The striatum is the primary input nucleus for the basal ganglia, and receives glutamatergic afferents from the cortex. Under the hypothesis that basal ganglia perform action selection, these cortical afferents encode potential "action requests." Previous studies have suggested the striatum may utilize a mutually inhibitory network of medium spiny neurons (MSNs) to filter these requests so that only those of high salience are selected. However, the mechanisms enabling the striatum to perform clean, rapid switching between distinct actions that form part of a learned action sequence are still poorly understood. Substance P (SP) and enkephalin are neuropeptides co-released with GABA in MSNs preferentially expressing D1 or D2 dopamine receptors respectively. SP has a facilitatory effect on subsequent glutamatergic inputs to target MSNs, while enkephalin has an inhibitory effect. Blocking the action of SP in the striatum is also known to affect behavioral transitions. We constructed phenomenological models of the effects of SP and enkephalin, and integrated these into a hybrid model of basal ganglia comprising a spiking striatal microcircuit and rate-coded populations representing other major structures. We demonstrated that diffuse neuropeptide connectivity enhanced the selection of unordered action requests, and that for true action sequences, where action semantics define a fixed structure, a patterning of the SP connectivity reflecting this ordering enhanced selection of actions presented in the correct sequential order and suppressed incorrect ordering. We also showed that selective pruning of SP connections allowed context-sensitive inhibition of specific undesirable requests that otherwise interfered with selection of an action group. Our model suggests that the interaction of SP and enkephalin enhances the contrast between selection and rejection of action requests, and that patterned SP connectivity in the striatum allows the "chunking" of actions and improves selection of sequences. Efficient execution of action sequences may therefore result from a combination of ordered cortical inputs and patterned neuropeptide connectivity within striatum.