Tracking the Cognitive Band in an Open-Ended Task.

Open-ended tasks can be decomposed into the three levels of Newell's Cognitive Band: the Unit-Task level, the Operation level, and the Deliberate-Act level. We analyzed the video game Co-op Space Fortress at these levels, reporting both the match of a cognitive model to subject behavior and the use of electroencephalogram (EEG) to track subject cognition. The Unit Task level in this game involves coordinating with a partner to kill a fortress. At this highest level of the Cognitive Band, there is a good match between subject behavior and the model. The EEG signals were also strong enough to track when Unit Tasks succeeded or failed. The intermediate Operation level in this task involves legs of flight to achieve a kill. The EEG signals associated with these operations are much weaker than the signals associated with the Unit Tasks. Still, it was possible to reconstruct subject play with much better than chance success. There were significant differences in the leg behavior of subjects and models. Model behavior did not provide a good basis for interpreting a subject's behavior at this level. At the lowest Deliberate-Act level, we observed overlapping key actions, which the model did not display. Such overlapping key actions also frustrated efforts to identify EEG signals of motor actions. We conclude that the Unit-task level is the appropriate level both for understanding open-ended tasks and for using EEG to track the performance of open-ended tasks.

Reconstructing fine-grained cognition from brain activity.

We describe the Sketch-and-Stitch method for bringing together a cognitive model and EEG to reconstruct the cognition of a subject. The method was tested in the context of a video game where the actions are highly interdependent and variable: simply changing whether a key was pressed or not for a 30th of a second can lead to a very different outcome. The Sketch level identifies the critical events in the game and the Stitch level fills in the detailed actions between these events. The critical events tend to produce robust EEG signals and the cognitive model provides probabilities of various transitions between critical events and the distribution of intervals between these events. This information can be combined in a hidden semi-Markov model that identifies the most probable sequence of critical events and when they happened. The Stitch level selects detailed actions from an extensive library of model games to produce these critical events. The decision about which sequence of actions to select from the library is made on the basis of how well they would produce weaker aspects of the EEG signal. The resulting approach can produce quite compelling replays of actual games from the EEG of a subject.

Spatiotemporal analysis of event-related fMRI to reveal cognitive states.

Cognitive science has a rich history of developing theories of processing that characterize the mental steps involved in performance of many tasks. Recent work in neuroimaging and machine learning has greatly improved our ability to link cognitive processes with what is happening in the brain. This article analyzes a hidden semi-Markov model-multivoxel pattern-analysis (HSMM-MVPA) methodology that we have developed for inferring the sequence of brain states one traverses in the performance of a cognitive task. The method is applied to a functional magnetic resonance imaging (fMRI) experiment where task boundaries are known that should separate states. The method is able to accurately identify those boundaries. Then, applying the method to synthetic data, we explore more fully those factors that influence performance of the method: signal-to-noise ratio, numbers of states, state sojourn times, and numbers of underlying experimental conditions. The results indicate the types of experimental tasks where applications of the HSMM-MVPA method are likely to yield accurate and insightful results.

The Common Time Course of Memory Processes Revealed.

Magnetoencephalography (MEG) was used to compare memory processes in two experiments, one involving recognition of word pairs and the other involving recall of newly learned arithmetic facts. A combination of hidden semi-Markov models and multivariate pattern analysis was used to locate brief "bumps" in the sensor data that marked the onset of different stages of cognitive processing. These bumps identified a separation between a retrieval stage that identified relevant information in memory and a decision stage that determined what response was implied by that information. The encoding, retrieval, decision, and response stages displayed striking similarities across the two experiments in their duration and brain activation patterns. Retrieval and decision processes involve distinct brain activation patterns. We conclude that memory processes for two different tasks, associative recognition versus arithmetic retrieval, follow a common spatiotemporal neural pattern and that both tasks have distinct retrieval and decision stages.

When math operations have visuospatial meanings versus purely symbolic definitions: Which solving stages and brain regions are affected?

How does processing differ during purely symbolic problem solving versus when mathematical operations can be mentally associated with meaningful (here, visuospatial) referents? Learners were trained on novel math operations (↓, ↑), that were defined strictly symbolically or in terms of a visuospatial interpretation (operands mapped to dimensions of shaded areas, answer = total area). During testing (scanner session), no visuospatial representations were displayed. However, we expected visuospatially-trained learners to form mental visuospatial representations for problems, and exhibit distinct activations. Since some solution intervals were long (~10s) and visuospatial representations might only be instantiated in some stages during solving, group differences were difficult to detect when treating the solving interval as a whole. However, an HSMM-MVPA process (Anderson and Fincham, 2014a) to parse fMRI data identified four distinct problem-solving stages in each group, dubbed: 1) encode; 2) plan; 3) compute; and 4) respond. We assessed stage-specific differences across groups. During encoding, several regions implicated in general semantic processing and/or mental imagery were more active in visuospatially-trained learners, including: bilateral supramarginal, precuneus, cuneus, parahippocampus, and left middle temporal regions. Four of these regions again emerged in the computation stage: precuneus, right supramarginal/angular, left supramarginal/inferior parietal, and left parahippocampal gyrus. Thus, mental visuospatial representations may not just inform initial problem interpretation (followed by symbolic computation), but may scaffold on-going computation. In the second stage, higher activations were found among symbolically-trained solvers in frontal regions (R. medial and inferior and L. superior) and the right angular and middle temporal gyrus. Activations in contrasting regions may shed light on solvers' degree of use of symbolic versus mental visuospatial strategies, even in absence of behavioral differences.

Change-point detection of cognitive states across multiple trials in functional neuroimaging.

Many functional neuroimaging-based studies involve repetitions of a task that may require several phases, or states, of mental activity. An appealing idea is to use relevant brain regions to identify the states. We developed a novel change-point methodology that adapts to the repeated trial structure of such experiments by assuming the number of states stays fixed across similar trials while allowing the timing of change-points to change across trials. Model fitting is based on reversible-jump MCMC. Simulation studies verified its ability to identify change-points successfully. We applied this technique to data collected via functional magnetic resonance imaging (fMRI) while each of 20 subjects solved unfamiliar arithmetic problems. Our methodology supplies both a summary of state dimensionality and uncertainty assessments about number of states and the timing of state transitions. Copyright © 2016 John Wiley & Sons, Ltd.

Hidden Stages of Cognition Revealed in Patterns of Brain Activation.

To advance cognitive theory, researchers must be able to parse the performance of a task into its significant mental stages. In this article, we describe a new method that uses functional MRI brain activation to identify when participants are engaged in different cognitive stages on individual trials. The method combines multivoxel pattern analysis to identify cognitive stages and hidden semi-Markov models to identify their durations. This method, applied to a problem-solving task, identified four distinct stages: encoding, planning, solving, and responding. We examined whether these stages corresponded to their ascribed functions by testing whether they are affected by appropriate factors. Planning-stage duration increased as the method for solving the problem became less obvious, whereas solving-stage duration increased as the number of calculations to produce the answer increased. Responding-stage duration increased with the difficulty of the motor actions required to produce the answer.

Phases of learning: How skill acquisition impacts cognitive processing.

This fMRI study examines the changes in participants' information processing as they repeatedly solve the same mathematical problem. We show that the majority of practice-related speedup is produced by discrete changes in cognitive processing. Because the points at which these changes take place vary from problem to problem, and the underlying information processing steps vary in duration, the existence of such discrete changes can be hard to detect. Using two converging approaches, we establish the existence of three learning phases. When solving a problem in one of these learning phases, participants can go through three cognitive stages: Encoding, Solving, and Responding. Each cognitive stage is associated with a unique brain signature. Using a bottom-up approach combining multi-voxel pattern analysis and hidden semi-Markov modeling, we identify the duration of that stage on any particular trial from participants brain activation patterns. For our top-down approach we developed an ACT-R model of these cognitive stages and simulated how they change over the course of learning. The Solving stage of the first learning phase is long and involves a sequence of arithmetic computations. Participants transition to the second learning phase when they can retrieve the answer, thereby drastically reducing the duration of the Solving stage. With continued practice, participants then transition to the third learning phase when they recognize the problem as a single unit and produce the answer as an automatic response. The duration of this third learning phase is dominated by the Responding stage.

The sequential structure of brain activation predicts skill.

In an fMRI study, participants were trained to play a complex video game. They were scanned early and then again after substantial practice. While better players showed greater activation in one region (right dorsal striatum) their relative skill was better diagnosed by considering the sequential structure of whole brain activation. Using a cognitive model that played this game, we extracted a characterization of the mental states that are involved in playing a game and the statistical structure of the transitions among these states. There was a strong correspondence between this measure of sequential structure and the skill of different players. Using multi-voxel pattern analysis, it was possible to recognize, with relatively high accuracy, the cognitive states participants were in during particular scans. We used the sequential structure of these activation-recognized states to predict the skill of individual players. These findings indicate that important features about information-processing strategies can be identified from a model-based analysis of the sequential structure of brain activation.

End effects and cross-dimensional interference in identification of time and length: Evidence for a common memory mechanism.

Memory plays a critical role in time estimation, yet detailed mechanisms underlying temporal memory have not been fully understood. The current functional magnetic resonance imaging (fMRI) study investigated memory phenomena in absolute identification of time durations and line lengths. In both time and length identification, participants responded faster to end-of-range stimuli (e.g., the shortest or longest items of the stimulus set) than to middle stimuli. Participants performed worse in the incongruent condition (mismatch between time and length in the stimulus position) than in the congruent condition, indicating cross-dimensional interference between time and length. Both phenomena reflect increased difficulty of retrieving information relevant to the current context in the presence of context-irrelevant information. A region in the lateral inferior prefrontal cortex showed a greater response to the middle stimuli and in the incongruent condition suggesting greater demands for controlled memory retrieval. A cognitive model based on the ACT-R (Adaptive Control of Thought - Rational) declarative memory mechanisms accounted for the major behavioral and imaging results. The results suggest that contextual effects in temporal memory can be understood in terms of domain-general memory principles established outside the time estimation domain.

Visuospatial referents facilitate the learning and transfer of mathematical operations: extending the role of the angular gyrus.

Different external representations for learning and solving mathematical operations may affect learning and transfer. To explore the effects of learning representations, learners were each introduced to two new operations (b↑n and b↓n) via either formulas or graphical representations. Both groups became adept at solving regular (trained) problems. During transfer, no external formulas or graphs were present; however, graph learners' knowledge could allow them to mentally associate problem expressions with visuospatial referents. The angular gyrus (AG) has recently been hypothesized to map problems to mental referents (e.g., symbolic answers; Grabner, Ansari, Koschutnig, Reishofer, & Ebner Human Brain Mapping, 34, 1013-1024, 2013), and we sought to test this hypothesis for visuospatial referents. To determine whether the AG and other math (horizontal intraparietal sulcus) and visuospatial (fusiform and posterior superior parietal lobule [PSPL]) regions were implicated in processing visuospatial mental referents, we included two types of transfer problems, computational and relational, which differed in referential load (one graph vs. two). During solving, the activations in AG, PSPL, and fusiform reflected the referential load manipulation among graph but not formula learners. Furthermore, the AG was more active among graph learners overall, which is consistent with its hypothesized referential role. Behavioral performance was comparable across the groups on computational transfer problems, which could be solved in a way that incorporated learners' respective procedures for regular problems. However, graph learners were more successful on relational transfer problems, which assessed their understanding of the relations between pairs of similar problems within and across operations. On such problems, their behavioral performance correlated with activation in the AG, fusiform, and a relational processing region (BA 10).

Extending problem-solving procedures through reflection.

A large-sample (n=75) fMRI study guided the development of a theory of how people extend their problem-solving procedures by reflecting on them. Both children and adults were trained on a new mathematical procedure and then were challenged with novel problems that required them to change and extend their procedure to solve these problems. The fMRI data were analyzed using a combination of hidden Markov models (HMMs) and multi-voxel pattern analysis (MVPA). This HMM-MVPA analysis revealed the existence of 4 stages: Encoding, Planning, Solving, and Responding. Using this analysis as a guide, an ACT-R model was developed that improved the performance of the HMM-MVPA and explained the variation in the durations of the stages across 128 different problems. The model assumes that participants can reflect on declarative representations of the steps of their problem-solving procedures. A Metacognitive module can hold these steps, modify them, create new declarative steps, and rehearse them. The Metacognitive module is associated with activity in the rostrolateral prefrontal cortex (RLPFC). The ACT-R model predicts the activity in the RLPFC and other regions associated with its other cognitive modules (e.g., vision, retrieval). Differences between children and adults seemed related to differences in background knowledge and computational fluency, but not to the differences in their capability to modify procedures.

Discovering the structure of mathematical problem solving.

The goal of this research is to discover the stages of mathematical problem solving, the factors that influence the duration of these stages, and how these stages are related to the learning of a new mathematical competence. Using a combination of multivariate pattern analysis (MVPA) and hidden Markov models (HMM), we found that participants went through 5 major phases in solving a class of problems: A Define Phase where they identified the problem to be solved, an Encode Phase where they encoded the needed information, a Compute Phase where they performed the necessary arithmetic calculations, a Transform Phase where they performed any mathematical transformations, and a Respond Phase where they entered an answer. The Define Phase is characterized by activity in visual attention and default network regions, the Encode Phase by activity in visual regions, the Compute Phase by activity in regions active in mathematical tasks, the Transform Phase by activity in mathematical and response regions, and the Respond phase by activity in motor regions. The duration of the Compute and Transform Phases were the only ones that varied with condition. Two features distinguished the mastery trials on which participants came to understand a new problem type. First, the duration of late phases of the problem solution increased. Second, there was increased activation in the rostrolateral prefrontal cortex (RLPFC) and angular gyrus (AG), regions associated with metacognition. This indicates the importance of reflection to successful learning.

Detecting math problem solving strategies: an investigation into the use of retrospective self-reports, latency and fMRI data.

This research explores how to determine when mathematical problems are solved by retrieval versus computation strategies. Past research has indicated that verbal reports, solution latencies, and neural imaging all provide imperfect indicators of this distinction. Participants in the current study solved mathematical problems involving two distinct problem types, called 'Pyramid' and 'Formula' problems. Participants were given extensive training solving 3 select Pyramid and 3 select Formula problems. Trained problems were highly practiced, whereas untrained problems were not. The distinction between untrained and trained problems was observed in the data. Untrained problems took longer to solve, more often used procedural strategies and showed a greater activation in the horizontal intraparietal sulcus (HIPS) when compared to trained problems. A classifier fit to the neural distinction between trained-untrained problems successfully predicted training within and between the two problem types. We employed this classifier to generate a prediction of strategy use. By combining evidence from the classifier, problem solving latencies, and retrospective reports, we predicted the strategy used to solve each problem in the scanner and gained unexpected insight into the distinction between different strategies.

Discovering the sequential structure of thought.

Multi-voxel pattern recognition techniques combined with Hidden Markov models can be used to discover the mental states that people go through in performing a task. The combined method identifies both the mental states and how their durations vary with experimental conditions. We apply this method to a task where participants solve novel mathematical problems. We identify four states in the solution of these problems: Encoding, Planning, Solving, and Respond. The method allows us to interpret what participants are doing on individual problem-solving trials. The duration of the planning state varies on a trial-to-trial basis with novelty of the problem. The duration of solution stage similarly varies with the amount of computation needed to produce a solution once a plan is devised. The response stage similarly varies with the complexity of the answer produced. In addition, we identified a number of effects that ran counter to a prior model of the task. Thus, we were able to decompose the overall problem-solving time into estimates of its components and in way that serves to guide theory.

Brain networks supporting execution of mathematical skills versus acquisition of new mathematical competence.

This fMRI study examines how students extend their mathematical competence. Students solved a set of algebra-like problems. These problems included Regular Problems that have a known solution technique and Exception Problems that but did not have a known technique. Two distinct networks of activity were uncovered. There was a Cognitive Network that was mainly active during the solution of problems and showed little difference between Regular Problems and Exception Problems. There was also a Metacognitive Network that was more engaged during a reflection period after the solution and was much more engaged for Exception Problems than Regular Problems. The Cognitive Network overlaps with prefrontal and parietal regions identified in the ACT-R theory of algebra problem solving and regions identified in the triple-code theory as involved in basic mathematical cognition. The Metacognitive Network included angular gyrus, middle temporal gyrus, and anterior prefrontal regions. This network is mainly engaged by the need to modify the solution procedure and not by the difficulty of the problem. Only the Metacognitive Network decreased with practice on the Exception Problems. Activity in the Cognitive Network during the solution of an Exception Problem predicted both success on that problem and future mastery. Activity in the angular gyrus and middle temporal gyrus during feedback on errors predicted future mastery.

Using brain imaging to track problem solving in a complex state space.

This paper describes how behavioral and imaging data can be combined with a Hidden Markov Model (HMM) to track participants' trajectories through a complex state space. Participants completed a problem-solving variant of a memory game that involved 625 distinct states, 24 operators, and an astronomical number of paths through the state space. Three sources of information were used for classification purposes. First, an Imperfect Memory Model was used to estimate transition probabilities for the HMM. Second, behavioral data provided information about the timing of different events. Third, multivoxel pattern analysis of the imaging data was used to identify features of the operators. By combining the three sources of information, an HMM algorithm was able to efficiently identify the most probable path that participants took through the state space, achieving over 80% accuracy. These results support the approach as a general methodology for tracking mental states that occur during individual problem-solving episodes.

Tracking children's mental states while solving algebra equations.

Behavioral and function magnetic resonance imagery (fMRI) data were combined to infer the mental states of students as they interacted with an intelligent tutoring system. Sixteen children interacted with a computer tutor for solving linear equations over a six-day period (days 0-5), with days 1 and 5 occurring in an fMRI scanner. Hidden Markov model algorithms combined a model of student behavior with multi-voxel imaging pattern data to predict the mental states of students. We separately assessed the algorithms' ability to predict which step in a problem-solving sequence was performed and whether the step was performed correctly. For day 1, the data patterns of other students were used to predict the mental states of a target student. These predictions were improved on day 5 by adding information about the target student's behavioral and imaging data from day 1. Successful tracking of mental states depended on using the combination of a behavioral model and multi-voxel pattern analysis, illustrating the effectiveness of an integrated approach to tracking the cognition of individuals in real time as they perform complex tasks.

The neural correlates of competition during memory retrieval are modulated by attention to the cues.

As people learn more facts about a concept, those facts become more difficult to remember. This is called the fan effect, where fan refers to the number of facts known about a concept. Increasing fan has been shown to decrease accuracy and increase response time and left ventrolateral prefrontal cortex (VLPFC) activity during retrieval. In this study, participants learned 36 arbitrary person-location pairings and made recognition decisions while we recorded brain activity using fMRI. We separately manipulated the fan of each person and location, as well as the training procedure with which each pair was studied. In the person focus condition, participants studied pairs with a picture of the person's face and used the person as a retrieval cue during training. In the location focus condition, participants studied pairs with a picture of the location and used the location as a retrieval cue during training. We found that the fan of the focused cue had a greater effect on response time, accuracy, and left VLPFC activity during retrieval than the fan of the unfocused cue. We also found that the parahippocampal place area (PPA) was more active during the recognition of pairs studied in the location focus condition, but not when the fan of the location was high. Overall, we found opposite effects of fan on VLPFC and PPA that were modulated by cue focus.

Brain regions engaged by part- and whole-task performance in a video game: a model-based test of the decomposition hypothesis.

Part- and whole-task conditions were created by manipulating the presence of certain components of the Space Fortress video game. A cognitive model was created for two-part games that could be combined into a model that performed the whole game. The model generated predictions both for behavioral patterns and activation patterns in various brain regions. The activation predictions concerned both tonic activation that was constant in these regions during performance of the game and phasic activation that occurred when there was resource competition. The model's predictions were confirmed about how tonic and phasic activation in different regions would vary with condition. These results support the Decomposition Hypothesis that the execution of a complex task can be decomposed into a set of information-processing components and that these components combine unchanged in different task conditions. In addition, individual differences in learning gains were predicted by individual differences in phasic activation in those regions that displayed highest tonic activity. This individual difference pattern suggests that the rate of learning of a complex skill is determined by capacity limits.