Reconstructing fine-grained cognition from brain activity.

We describe the Sketch-and-Stitch method for bringing together a cognitive model and EEG to reconstruct the cognition of a subject. The method was tested in the context of a video game where the actions are highly interdependent and variable: simply changing whether a key was pressed or not for a 30th of a second can lead to a very different outcome. The Sketch level identifies the critical events in the game and the Stitch level fills in the detailed actions between these events. The critical events tend to produce robust EEG signals and the cognitive model provides probabilities of various transitions between critical events and the distribution of intervals between these events. This information can be combined in a hidden semi-Markov model that identifies the most probable sequence of critical events and when they happened. The Stitch level selects detailed actions from an extensive library of model games to produce these critical events. The decision about which sequence of actions to select from the library is made on the basis of how well they would produce weaker aspects of the EEG signal. The resulting approach can produce quite compelling replays of actual games from the EEG of a subject.

Spatiotemporal analysis of event-related fMRI to reveal cognitive states.

Cognitive science has a rich history of developing theories of processing that characterize the mental steps involved in performance of many tasks. Recent work in neuroimaging and machine learning has greatly improved our ability to link cognitive processes with what is happening in the brain. This article analyzes a hidden semi-Markov model-multivoxel pattern-analysis (HSMM-MVPA) methodology that we have developed for inferring the sequence of brain states one traverses in the performance of a cognitive task. The method is applied to a functional magnetic resonance imaging (fMRI) experiment where task boundaries are known that should separate states. The method is able to accurately identify those boundaries. Then, applying the method to synthetic data, we explore more fully those factors that influence performance of the method: signal-to-noise ratio, numbers of states, state sojourn times, and numbers of underlying experimental conditions. The results indicate the types of experimental tasks where applications of the HSMM-MVPA method are likely to yield accurate and insightful results.

The Common Time Course of Memory Processes Revealed.

Magnetoencephalography (MEG) was used to compare memory processes in two experiments, one involving recognition of word pairs and the other involving recall of newly learned arithmetic facts. A combination of hidden semi-Markov models and multivariate pattern analysis was used to locate brief "bumps" in the sensor data that marked the onset of different stages of cognitive processing. These bumps identified a separation between a retrieval stage that identified relevant information in memory and a decision stage that determined what response was implied by that information. The encoding, retrieval, decision, and response stages displayed striking similarities across the two experiments in their duration and brain activation patterns. Retrieval and decision processes involve distinct brain activation patterns. We conclude that memory processes for two different tasks, associative recognition versus arithmetic retrieval, follow a common spatiotemporal neural pattern and that both tasks have distinct retrieval and decision stages.

When math operations have visuospatial meanings versus purely symbolic definitions: Which solving stages and brain regions are affected?

How does processing differ during purely symbolic problem solving versus when mathematical operations can be mentally associated with meaningful (here, visuospatial) referents? Learners were trained on novel math operations (↓, ↑), that were defined strictly symbolically or in terms of a visuospatial interpretation (operands mapped to dimensions of shaded areas, answer = total area). During testing (scanner session), no visuospatial representations were displayed. However, we expected visuospatially-trained learners to form mental visuospatial representations for problems, and exhibit distinct activations. Since some solution intervals were long (~10s) and visuospatial representations might only be instantiated in some stages during solving, group differences were difficult to detect when treating the solving interval as a whole. However, an HSMM-MVPA process (Anderson and Fincham, 2014a) to parse fMRI data identified four distinct problem-solving stages in each group, dubbed: 1) encode; 2) plan; 3) compute; and 4) respond. We assessed stage-specific differences across groups. During encoding, several regions implicated in general semantic processing and/or mental imagery were more active in visuospatially-trained learners, including: bilateral supramarginal, precuneus, cuneus, parahippocampus, and left middle temporal regions. Four of these regions again emerged in the computation stage: precuneus, right supramarginal/angular, left supramarginal/inferior parietal, and left parahippocampal gyrus. Thus, mental visuospatial representations may not just inform initial problem interpretation (followed by symbolic computation), but may scaffold on-going computation. In the second stage, higher activations were found among symbolically-trained solvers in frontal regions (R. medial and inferior and L. superior) and the right angular and middle temporal gyrus. Activations in contrasting regions may shed light on solvers' degree of use of symbolic versus mental visuospatial strategies, even in absence of behavioral differences.

Change-point detection of cognitive states across multiple trials in functional neuroimaging.

Many functional neuroimaging-based studies involve repetitions of a task that may require several phases, or states, of mental activity. An appealing idea is to use relevant brain regions to identify the states. We developed a novel change-point methodology that adapts to the repeated trial structure of such experiments by assuming the number of states stays fixed across similar trials while allowing the timing of change-points to change across trials. Model fitting is based on reversible-jump MCMC. Simulation studies verified its ability to identify change-points successfully. We applied this technique to data collected via functional magnetic resonance imaging (fMRI) while each of 20 subjects solved unfamiliar arithmetic problems. Our methodology supplies both a summary of state dimensionality and uncertainty assessments about number of states and the timing of state transitions. Copyright © 2016 John Wiley & Sons, Ltd.

Hidden Stages of Cognition Revealed in Patterns of Brain Activation.

To advance cognitive theory, researchers must be able to parse the performance of a task into its significant mental stages. In this article, we describe a new method that uses functional MRI brain activation to identify when participants are engaged in different cognitive stages on individual trials. The method combines multivoxel pattern analysis to identify cognitive stages and hidden semi-Markov models to identify their durations. This method, applied to a problem-solving task, identified four distinct stages: encoding, planning, solving, and responding. We examined whether these stages corresponded to their ascribed functions by testing whether they are affected by appropriate factors. Planning-stage duration increased as the method for solving the problem became less obvious, whereas solving-stage duration increased as the number of calculations to produce the answer increased. Responding-stage duration increased with the difficulty of the motor actions required to produce the answer.

Phases of learning: How skill acquisition impacts cognitive processing.

This fMRI study examines the changes in participants' information processing as they repeatedly solve the same mathematical problem. We show that the majority of practice-related speedup is produced by discrete changes in cognitive processing. Because the points at which these changes take place vary from problem to problem, and the underlying information processing steps vary in duration, the existence of such discrete changes can be hard to detect. Using two converging approaches, we establish the existence of three learning phases. When solving a problem in one of these learning phases, participants can go through three cognitive stages: Encoding, Solving, and Responding. Each cognitive stage is associated with a unique brain signature. Using a bottom-up approach combining multi-voxel pattern analysis and hidden semi-Markov modeling, we identify the duration of that stage on any particular trial from participants brain activation patterns. For our top-down approach we developed an ACT-R model of these cognitive stages and simulated how they change over the course of learning. The Solving stage of the first learning phase is long and involves a sequence of arithmetic computations. Participants transition to the second learning phase when they can retrieve the answer, thereby drastically reducing the duration of the Solving stage. With continued practice, participants then transition to the third learning phase when they recognize the problem as a single unit and produce the answer as an automatic response. The duration of this third learning phase is dominated by the Responding stage.

Visuospatial referents facilitate the learning and transfer of mathematical operations: extending the role of the angular gyrus.

Different external representations for learning and solving mathematical operations may affect learning and transfer. To explore the effects of learning representations, learners were each introduced to two new operations (b↑n and b↓n) via either formulas or graphical representations. Both groups became adept at solving regular (trained) problems. During transfer, no external formulas or graphs were present; however, graph learners' knowledge could allow them to mentally associate problem expressions with visuospatial referents. The angular gyrus (AG) has recently been hypothesized to map problems to mental referents (e.g., symbolic answers; Grabner, Ansari, Koschutnig, Reishofer, & Ebner Human Brain Mapping, 34, 1013-1024, 2013), and we sought to test this hypothesis for visuospatial referents. To determine whether the AG and other math (horizontal intraparietal sulcus) and visuospatial (fusiform and posterior superior parietal lobule [PSPL]) regions were implicated in processing visuospatial mental referents, we included two types of transfer problems, computational and relational, which differed in referential load (one graph vs. two). During solving, the activations in AG, PSPL, and fusiform reflected the referential load manipulation among graph but not formula learners. Furthermore, the AG was more active among graph learners overall, which is consistent with its hypothesized referential role. Behavioral performance was comparable across the groups on computational transfer problems, which could be solved in a way that incorporated learners' respective procedures for regular problems. However, graph learners were more successful on relational transfer problems, which assessed their understanding of the relations between pairs of similar problems within and across operations. On such problems, their behavioral performance correlated with activation in the AG, fusiform, and a relational processing region (BA 10).

End effects and cross-dimensional interference in identification of time and length: Evidence for a common memory mechanism.

Memory plays a critical role in time estimation, yet detailed mechanisms underlying temporal memory have not been fully understood. The current functional magnetic resonance imaging (fMRI) study investigated memory phenomena in absolute identification of time durations and line lengths. In both time and length identification, participants responded faster to end-of-range stimuli (e.g., the shortest or longest items of the stimulus set) than to middle stimuli. Participants performed worse in the incongruent condition (mismatch between time and length in the stimulus position) than in the congruent condition, indicating cross-dimensional interference between time and length. Both phenomena reflect increased difficulty of retrieving information relevant to the current context in the presence of context-irrelevant information. A region in the lateral inferior prefrontal cortex showed a greater response to the middle stimuli and in the incongruent condition suggesting greater demands for controlled memory retrieval. A cognitive model based on the ACT-R (Adaptive Control of Thought - Rational) declarative memory mechanisms accounted for the major behavioral and imaging results. The results suggest that contextual effects in temporal memory can be understood in terms of domain-general memory principles established outside the time estimation domain.

Discovering the structure of mathematical problem solving.

The goal of this research is to discover the stages of mathematical problem solving, the factors that influence the duration of these stages, and how these stages are related to the learning of a new mathematical competence. Using a combination of multivariate pattern analysis (MVPA) and hidden Markov models (HMM), we found that participants went through 5 major phases in solving a class of problems: A Define Phase where they identified the problem to be solved, an Encode Phase where they encoded the needed information, a Compute Phase where they performed the necessary arithmetic calculations, a Transform Phase where they performed any mathematical transformations, and a Respond Phase where they entered an answer. The Define Phase is characterized by activity in visual attention and default network regions, the Encode Phase by activity in visual regions, the Compute Phase by activity in regions active in mathematical tasks, the Transform Phase by activity in mathematical and response regions, and the Respond phase by activity in motor regions. The duration of the Compute and Transform Phases were the only ones that varied with condition. Two features distinguished the mastery trials on which participants came to understand a new problem type. First, the duration of late phases of the problem solution increased. Second, there was increased activation in the rostrolateral prefrontal cortex (RLPFC) and angular gyrus (AG), regions associated with metacognition. This indicates the importance of reflection to successful learning.

Extending problem-solving procedures through reflection.

A large-sample (n=75) fMRI study guided the development of a theory of how people extend their problem-solving procedures by reflecting on them. Both children and adults were trained on a new mathematical procedure and then were challenged with novel problems that required them to change and extend their procedure to solve these problems. The fMRI data were analyzed using a combination of hidden Markov models (HMMs) and multi-voxel pattern analysis (MVPA). This HMM-MVPA analysis revealed the existence of 4 stages: Encoding, Planning, Solving, and Responding. Using this analysis as a guide, an ACT-R model was developed that improved the performance of the HMM-MVPA and explained the variation in the durations of the stages across 128 different problems. The model assumes that participants can reflect on declarative representations of the steps of their problem-solving procedures. A Metacognitive module can hold these steps, modify them, create new declarative steps, and rehearse them. The Metacognitive module is associated with activity in the rostrolateral prefrontal cortex (RLPFC). The ACT-R model predicts the activity in the RLPFC and other regions associated with its other cognitive modules (e.g., vision, retrieval). Differences between children and adults seemed related to differences in background knowledge and computational fluency, but not to the differences in their capability to modify procedures.

Tracking the Cognitive Band in an Open-Ended Task.

Open-ended tasks can be decomposed into the three levels of Newell's Cognitive Band: the Unit-Task level, the Operation level, and the Deliberate-Act level. We analyzed the video game Co-op Space Fortress at these levels, reporting both the match of a cognitive model to subject behavior and the use of electroencephalogram (EEG) to track subject cognition. The Unit Task level in this game involves coordinating with a partner to kill a fortress. At this highest level of the Cognitive Band, there is a good match between subject behavior and the model. The EEG signals were also strong enough to track when Unit Tasks succeeded or failed. The intermediate Operation level in this task involves legs of flight to achieve a kill. The EEG signals associated with these operations are much weaker than the signals associated with the Unit Tasks. Still, it was possible to reconstruct subject play with much better than chance success. There were significant differences in the leg behavior of subjects and models. Model behavior did not provide a good basis for interpreting a subject's behavior at this level. At the lowest Deliberate-Act level, we observed overlapping key actions, which the model did not display. Such overlapping key actions also frustrated efforts to identify EEG signals of motor actions. We conclude that the Unit-task level is the appropriate level both for understanding open-ended tasks and for using EEG to track the performance of open-ended tasks.

Neural imaging to track mental states while using an intelligent tutoring system.

Hemodynamic measures of brain activity can be used to interpret a student's mental state when they are interacting with an intelligent tutoring system. Functional magnetic resonance imaging (fMRI) data were collected while students worked with a tutoring system that taught an algebra isomorph. A cognitive model predicted the distribution of solution times from measures of problem complexity. Separately, a linear discriminant analysis used fMRI data to predict whether or not students were engaged in problem solving. A hidden Markov algorithm merged these two sources of information to predict the mental states of students during problem-solving episodes. The algorithm was trained on data from 1 day of interaction and tested with data from a later day. In terms of predicting what state a student was in during a 2-s period, the algorithm achieved 87% accuracy on the training data and 83% accuracy on the test data. The results illustrate the importance of integrating the bottom-up information from imaging data with the top-down information from a cognitive model.

Using brain imaging to track problem solving in a complex state space.

This paper describes how behavioral and imaging data can be combined with a Hidden Markov Model (HMM) to track participants' trajectories through a complex state space. Participants completed a problem-solving variant of a memory game that involved 625 distinct states, 24 operators, and an astronomical number of paths through the state space. Three sources of information were used for classification purposes. First, an Imperfect Memory Model was used to estimate transition probabilities for the HMM. Second, behavioral data provided information about the timing of different events. Third, multivoxel pattern analysis of the imaging data was used to identify features of the operators. By combining the three sources of information, an HMM algorithm was able to efficiently identify the most probable path that participants took through the state space, achieving over 80% accuracy. These results support the approach as a general methodology for tracking mental states that occur during individual problem-solving episodes.

Tracking children's mental states while solving algebra equations.

Behavioral and function magnetic resonance imagery (fMRI) data were combined to infer the mental states of students as they interacted with an intelligent tutoring system. Sixteen children interacted with a computer tutor for solving linear equations over a six-day period (days 0-5), with days 1 and 5 occurring in an fMRI scanner. Hidden Markov model algorithms combined a model of student behavior with multi-voxel imaging pattern data to predict the mental states of students. We separately assessed the algorithms' ability to predict which step in a problem-solving sequence was performed and whether the step was performed correctly. For day 1, the data patterns of other students were used to predict the mental states of a target student. These predictions were improved on day 5 by adding information about the target student's behavioral and imaging data from day 1. Successful tracking of mental states depended on using the combination of a behavioral model and multi-voxel pattern analysis, illustrating the effectiveness of an integrated approach to tracking the cognition of individuals in real time as they perform complex tasks.

Lateral inferior prefrontal cortex and anterior cingulate cortex are engaged at different stages in the solution of insight problems.

Two studies used puzzles that required participants to find a word that satisfied a set of constraints. The first study used a remote-association task, where participants had to find a word that would form compound words with 3 other words. The second study required participants to complete a word fragment with an associate of another word. Both studies produced distinct patterns of activity in the lateral inferior prefrontal cortex (LIPFC) and the anterior cingulate cortex (ACC). Activation in the LIPFC rose only as long as the participants were trying to retrieve the solution and dropped off as soon as the solution was obtained. However, activation in the ACC increased upon the retrieval of a solution, reflecting the need to process that solution. The data of the second experiment are fit by an information-processing model that interprets the activity in the LIPFC as reflecting retrieval operations and the activity in the ACC as reflecting subgoal setting.

Brain regions engaged by part- and whole-task performance in a video game: a model-based test of the decomposition hypothesis.

Part- and whole-task conditions were created by manipulating the presence of certain components of the Space Fortress video game. A cognitive model was created for two-part games that could be combined into a model that performed the whole game. The model generated predictions both for behavioral patterns and activation patterns in various brain regions. The activation predictions concerned both tonic activation that was constant in these regions during performance of the game and phasic activation that occurred when there was resource competition. The model's predictions were confirmed about how tonic and phasic activation in different regions would vary with condition. These results support the Decomposition Hypothesis that the execution of a complex task can be decomposed into a set of information-processing components and that these components combine unchanged in different task conditions. In addition, individual differences in learning gains were predicted by individual differences in phasic activation in those regions that displayed highest tonic activity. This individual difference pattern suggests that the rate of learning of a complex skill is determined by capacity limits.

Cognitive and metacognitive activity in mathematical problem solving: prefrontal and parietal patterns.

Students were taught an algorithm for solving a new class of mathematical problems. Occasionally in the sequence of problems, they encountered exception problems that required that they extend the algorithm. Regular and exception problems were associated with different patterns of brain activation. Some regions showed a Cognitive pattern of being active only until the problem was solved and no difference between regular or exception problems. Other regions showed a Metacognitive pattern of greater activity for exception problems and activity that extended into the post-solution period, particularly when an error was made. The Cognitive regions included some of parietal and prefrontal regions associated with the triple-code theory of (Dehaene, S., Piazza, M., Pinel, P., & Cohen, L. (2003). Three parietal circuits for number processing. Cognitive Neuropsychology, 20, 487-506) and associated with algebra equation solving in the ACT-R theory (Anderson, J. R. (2005). Human symbol manipulation within an 911 integrated cognitive architecture. Cognitive science, 29, 313-342. Metacognitive regions included the superior prefrontal gyrus, the angular gyrus of the triple-code theory, and frontopolar regions.

A central circuit of the mind.

The methodologies of cognitive architectures and functional magnetic resonance imaging can mutually inform each other. For example, four modules of the ACT-R (adaptive control of thought - rational) cognitive architecture have been associated with four brain regions that are active in complex tasks. Activity in a lateral inferior prefrontal region reflects retrieval of information in a declarative module; activity in a posterior parietal region reflects changes to problem representations in an imaginal module; activity in the anterior cingulate cortex reflects the updates of control information in a goal module; and activity in the caudate nucleus reflects execution of productions in a procedural module. Differential patterns of activation in such central regions can reveal the time course of different components of complex cognition.

Role of prefrontal and parietal cortices in associative learning.

Two studies were performed that compared a "Paired" condition in which participants studied paired associates with a "Generated" condition in which participants completed word fragments to produce paired associates. In both tasks, participants were responsible for memory of the material either studied or generated. The experiments revealed significant differences between the responses of a predefined prefrontal region and a predefined parietal region. The parietal region responded more in the Generated condition than the Paired condition, whereas there was no difference in the prefrontal region. On the other hand, the prefrontal region responded to the delay between study and test in both the Paired and Generated conditions, whereas the parietal region only responded to delay in the Generated condition. This pattern of results is consistent with the hypothesis that the parietal region is responsive to changes in problem representation and the prefrontal region to retrieval operations. An information-processing model embodying these assumptions was fit to the blood oxygen level-dependent responses in these regions.