Word Type and Frequency Effects on Lexical Decisions Are Process-dependent and Start Early.

When encountering letter strings, we rapidly determine whether they are words. The speed of such lexical decisions (LDs) is affected by word frequency. Apart from influencing late, decision-related, processing stages, frequency has also been shown to affect very early stages, and even the processing of nonwords. We developed a detailed account of the different frequency effects involved in LDs by (1) dividing LDs into processing stages using a combination of Hidden semi-Markov models and multivariate pattern analysis applied to EEG data and (2) using generalized additive mixed models to investigate how the effect of continuous word and nonword frequency differs between these stages. We discovered six stages shared between word types, with the fifth stage consisting of two substages for pseudowords only. In the earliest stages, visual processing was completed faster for frequent words, but took longer for word-like nonwords. Later stages involved an orthographic familiarity assessment followed by an elaborate decision process, both affected differently by frequency. We therefore conclude that frequency indeed affects all processes involved in LDs and that the magnitude and direction of these effects differ both by process and word type.

A whole-task brain model of associative recognition that accounts for human behavior and neuroimaging data.

Brain models typically focus either on low-level biological detail or on qualitative behavioral effects. In contrast, we present a biologically-plausible spiking-neuron model of associative learning and recognition that accounts for both human behavior and low-level brain activity across the whole task. Based on cognitive theories and insights from machine-learning analyses of M/EEG data, the model proceeds through five processing stages: stimulus encoding, familiarity judgement, associative retrieval, decision making, and motor response. The results matched human response times and source-localized MEG data in occipital, temporal, prefrontal, and precentral brain regions; as well as a classic fMRI effect in prefrontal cortex. This required two main conceptual advances: a basal-ganglia-thalamus action-selection system that relies on brief thalamic pulses to change the functional connectivity of the cortex, and a new unsupervised learning rule that causes very strong pattern separation in the hippocampus. The resulting model shows how low-level brain activity can result in goal-directed cognitive behavior in humans.

Inter-individual single-trial classification of MEG data using M-CCA.

Neuroscientific studies often involve some form of group analysis over multiple participants. This requires alignment of recordings across participants. A naive solution is to assume that participants' recordings can be aligned anatomically in sensor space. However, this assumption is likely violated due to anatomical and functional differences between individual brains. In magnetoencephalography (MEG) recordings the problem of inter-subject alignment is exacerbated by the susceptibility of MEG to individual cortical folding patterns as well as the inter-subject variability of sensor locations over the brain due to the use of a fixed helmet. Hence, an approach to combine MEG data over individual brains should relax the assumptions that a) brain anatomy and function are tightly linked and b) that the same sensors capture functionally comparable brain activation across individuals. Here we use multiset canonical correlation analysis (M-CCA) to find a common representation of MEG activations recorded from 15 participants performing a grasping task. The M-CCA algorithm was applied to transform the data of a set of multiple participants into a common space with maximum correlation between participants. Importantly, we derive a method to transform data from a new, previously unseen participant into this common representation. This makes it useful for applications that require transfer of models derived from a group of individuals to new individuals. We demonstrate the usefulness and superiority of the approach with respect to previously used approaches. Finally, we show that our approach requires only a small number of labeled data from the new participant. The proposed method demonstrates that functionally motivated common spaces have potential applications in reducing training time of online brain-computer interfaces, where models can be pre-trained on previous participants/sessions. Moreover, inter-subject alignment via M-CCA has the potential for combining data of different participants and could become helpful in future endeavors on large open datasets.

Decoding study-independent mind-wandering from EEG using convolutional neural networks.

Objective. Mind-wandering is a mental phenomenon where the internal thought process disengages from the external environment periodically. In the current study, we trained EEG classifiers using convolutional neural networks (CNNs) to track mind-wandering across studies.Approach. We transformed the input from raw EEG to band-frequency information (power), single-trial ERP (stERP) patterns, and connectivity matrices between channels (based on inter-site phase clustering). We trained CNN models for each input type from each EEG channel as the input model for the meta-learner. To verify the generalizability, we used leave-N-participant-out cross-validations (N= 6) and tested the meta-learner on the data from an independent study for across-study predictions.Main results. The current results show limited generalizability across participants and tasks. Nevertheless, our meta-learner trained with the stERPs performed the best among the state-of-the-art neural networks. The mapping of each input model to the output of the meta-learner indicates the importance of each EEG channel.Significance. Our study makes the first attempt to train study-independent mind-wandering classifiers. The results indicate that this remains challenging. The stacking neural network design we used allows an easy inspection of channel importance and feature maps.

Multitasking While Driving: Central Bottleneck or Problem State Interference?

OBJECTIVE: The objective of this work was to investigate if visuospatial attention and working memory load interact at a central control resource or at a task-specific, information processing resource during driving. BACKGROUND: In previous multitasking driving experiments, interactions between different cognitive concepts (e.g., attention and working memory) have been found. These interactions have been attributed to a central bottleneck or to the so-called problem-state bottleneck, related to working memory usage. METHOD: We developed two different cognitive models in the cognitive architecture ACT-R, which implement the central vs. problem-state bottleneck. The models performed a driving task, during which we varied visuospatial attention and working memory load. We evaluated the model by conducting an experiment with human participants and compared the behavioral data to the model's behavior. RESULTS: The problem-state-bottleneck model could account for decreased driving performance due to working memory load as well as increased visuospatial attentional demands as compared to the central-bottleneck model, which could not account for effects of increased working memory load. CONCLUSION: The interaction between working memory and visuospatial attention in our dual tasking experiment can be best characterized by a bottleneck in the working memory. The model results suggest that as working memory load becomes higher, drivers manage to perform fewer control actions, which leads to decreasing driving performance. APPLICATION: Predictions about the effect of different mental loads can be used to quantify the contribution of each subtask allowing for precise assessments of the current overall mental load, which automated driving systems may adapt to.

Capturing Dynamic Performance in a Cognitive Model: Estimating ACT-R Memory Parameters With the Linear Ballistic Accumulator.

The parameters governing our behavior are in constant flux. Accurately capturing these dynamics in cognitive models poses a challenge to modelers. Here, we demonstrate a mapping of ACT-R's declarative memory onto the linear ballistic accumulator (LBA), a mathematical model describing a competition between evidence accumulation processes. We show that this mapping provides a method for inferring individual ACT-R parameters without requiring the modeler to build and fit an entire ACT-R model. Existing parameter estimation methods for the LBA can be used, instead of the computationally expensive parameter sweeps that are traditionally done. We conduct a parameter recovery study to confirm that the LBA can recover ACT-R parameters from simulated data. Then, as a proof of concept, we use the LBA to estimate ACT-R parameters from an empirical dataset. The resulting parameter estimates provide a cognitively meaningful explanation for observed differences in behavior over time and between individuals. In addition, we find that the mapping between ACT-R and LBA lends a more concrete interpretation to ACT-R's latency factor parameter, namely as a measure of response caution. This work contributes to a growing movement towards integrating formal modeling approaches in cognitive science.

Thalamic bursts modulate cortical synchrony locally to switch between states of global functional connectivity in a cognitive task.

Performing a cognitive task requires going through a sequence of functionally diverse stages. Although it is typically assumed that these stages are characterized by distinct states of cortical synchrony that are triggered by sub-cortical events, little reported evidence supports this hypothesis. To test this hypothesis, we first identified cognitive stages in single-trial MEG data of an associative recognition task, showing with a novel method that each stage begins with local modulations of synchrony followed by a state of directed functional connectivity. Second, we developed the first whole-brain model that can simulate cortical synchrony throughout a task. The model suggests that the observed synchrony is caused by thalamocortical bursts at the onset of each stage, targeted at cortical synapses and interacting with the structural anatomical connectivity. These findings confirm that cognitive stages are defined by distinct states of cortical synchrony and explains the network-level mechanisms necessary for reaching stage-dependent synchrony states.

Discovering the brain stages of lexical decision: Behavioral effects originate from a single neural decision process.

Lexical decision (LD) - judging whether a sequence of letters constitutes a word - has been widely investigated. In a typical lexical decision task (LDT), participants are asked to respond whether a sequence of letters is an actual word or a nonword. Although behavioral differences between types of words/nonwords have been robustly detected in LDT, there is an ongoing discussion about the exact cognitive processes that underlie the word identification process in this task. To obtain data-driven evidence on the underlying processes, we recorded electroencephalographic (EEG) data and applied a novel machine-learning method, hidden semi-Markov model multivariate pattern analysis (HsMM-MVPA). In the current study, participants performed an LDT in which we varied the frequency of words (high, low frequency) and "wordlikeness" of non-words (pseudowords, random non-words). The results revealed that models with six processing stages accounted best for the data in all conditions. While most stages were shared, Stage 5 differed between conditions. Together, these results indicate that the differences in word frequency and lexicality effects are driven by a single cognitive processing stage. Based on its latency and topology, we interpret this stage as a Decision process during which participants discriminate between words and nonwords using activated lexical information.

Memory-related cognitive load effects in an interrupted learning task: A model-based explanation.

BACKGROUND: The Cognitive Load Theory provides a well-established framework for investigating aspects of learning situations that demand learners' working memory resources. However, the interplay of these aspects at the cognitive and neural level is still not fully understood. METHOD: We developed four computational models in the cognitive architecture ACT-R to clarify underlying memory-related strategies and mechanisms. Our models account for human data of an experiment that required participants to perform a symbol sequence learning task with embedded interruptions. We explored the inclusion of subsymbolic mechanisms to explain these data and used our final model to generate fMRI predictions. RESULTS: The final model indicates a reasonable fit for reaction times and accuracy and links the fMRI predictions to the Cognitive Load Theory. CONCLUSIONS: Our work emphasizes the influence of task characteristics and supports a process-related view on cognitive load in instructional scenarios. It further contributes to the discussion of underlying mechanisms at a neural level.

Distinguishing vigilance decrement and low task demands from mind-wandering: A machine learning analysis of EEG.

Mind-wandering is a ubiquitous mental phenomenon that is defined as self-generated thought irrelevant to the ongoing task. Mind-wandering tends to occur when people are in a low-vigilance state or when they are performing a very easy task. In the current study, we investigated whether mind-wandering is completely dependent on vigilance and current task demands, or whether it is an independent phenomenon. To this end, we trained support vector machine (SVM) classifiers on EEG data in conditions of low and high vigilance, as well as under conditions of low and high task demands, and subsequently tested those classifiers on participants' self-reported mind-wandering. Participants' momentary mental state was measured by means of intermittent thought probes in which they reported on their current mental state. The results showed that neither the vigilance classifier nor the task demands classifier could predict mind-wandering above-chance level, while a classifier trained on self-reports of mind-wandering was able to do so. This suggests that mind-wandering is a mental state different from low vigilance or performing tasks with low demands-both which could be discriminated from the EEG above chance. Furthermore, we used dipole fitting to source-localize the neural correlates of the most import features in each of the three classifiers, indeed finding a few distinct neural structures between the three phenomena. Our study demonstrates the value of machine-learning classifiers in unveiling patterns in neural data and uncovering the associated neural structures by combining it with an EEG source analysis technique.

A functional spiking-neuron model of activity-silent working memory in humans based on calcium-mediated short-term synaptic plasticity.

In this paper, we present a functional spiking-neuron model of human working memory (WM). This model combines neural firing for encoding of information with activity-silent maintenance. While it used to be widely assumed that information in WM is maintained through persistent recurrent activity, recent studies have shown that information can be maintained without persistent firing; instead, information can be stored in activity-silent states. A candidate mechanism underlying this type of storage is short-term synaptic plasticity (STSP), by which the strength of connections between neurons rapidly changes to encode new information. To demonstrate that STSP can lead to functional behavior, we integrated STSP by means of calcium-mediated synaptic facilitation in a large-scale spiking-neuron model and added a decision mechanism. The model was used to simulate a recent study that measured behavior and EEG activity of participants in three delayed-response tasks. In these tasks, one or two visual gratings had to be maintained in WM, and compared to subsequent probes. The original study demonstrated that WM contents and its priority status could be decoded from neural activity elicited by a task-irrelevant stimulus displayed during the activity-silent maintenance period. In support of our model, we show that it can perform these tasks, and that both its behavior as well as its neural representations are in agreement with the human data. We conclude that information in WM can be effectively maintained in activity-silent states by means of calcium-mediated STSP.

Learning to Approximate Functions Using Nb-Doped SrTiO3 Memristors.

Memristors have attracted interest as neuromorphic computation elements because they show promise in enabling efficient hardware implementations of artificial neurons and synapses. We performed measurements on interface-type memristors to validate their use in neuromorphic hardware. Specifically, we utilized Nb-doped SrTiO3 memristors as synapses in a simulated neural network by arranging them into differential synaptic pairs, with the weight of the connection given by the difference in normalized conductance values between the two paired memristors. This network learned to represent functions through a training process based on a novel supervised learning algorithm, during which discrete voltage pulses were applied to one of the two memristors in each pair. To simulate the fact that both the initial state of the physical memristive devices and the impact of each voltage pulse are unknown we injected noise into the simulation. Nevertheless, discrete updates based on local knowledge were shown to result in robust learning performance. Using this class of memristive devices as the synaptic weight element in a spiking neural network yields, to our knowledge, one of the first models of this kind, capable of learning to be a universal function approximator, and strongly suggests the suitability of these memristors for usage in future computing platforms.

Predicting task-general mind-wandering with EEG.

Mind-wandering refers to the process of thinking task-unrelated thoughts while performing a task. The dynamics of mind-wandering remain elusive because it is difficult to track when someone's mind is wandering based only on behavior. The goal of this study is to develop a machine-learning classifier that can determine someone's mind-wandering state online using electroencephalography (EEG) in a way that generalizes across tasks. In particular, we trained machine-learning models on EEG markers to classify the participants' current state as either mind-wandering or on-task. To be able to examine the task generality of the classifier, two different paradigms were adopted in this study: a sustained attention to response task (SART) and a visual search task. In both tasks, probe questions asking for a self-report of the thoughts at that moment were inserted at random moments, and participants' responses to the probes were used to create labels for the classifier. The 6 trials preceding an off-task response were labeled as mind-wandering, whereas the 6 trials predicting an on-task response were labeled as on-task. The EEG markers used as features for the classifier included single-trial P1, N1, and P3, the power and coherence in the theta (4-8 Hz) and alpha (8.5-12 Hz) bands at PO7, Pz, PO8, and Fz. We used a support vector machine as the training algorithm to learn the connection between EEG markers and the current mind-wandering state. We were able to distinguish between on-task and off-task thinking with an accuracy ranging from 0.50 to 0.85. Moreover, the classifiers were task-general: The average accuracy in across-task prediction was 60%, which was above chance level. Among all the extracted EEG markers, alpha power was most predictive of mind-wandering.

The Common Time Course of Memory Processes Revealed.

Magnetoencephalography (MEG) was used to compare memory processes in two experiments, one involving recognition of word pairs and the other involving recall of newly learned arithmetic facts. A combination of hidden semi-Markov models and multivariate pattern analysis was used to locate brief "bumps" in the sensor data that marked the onset of different stages of cognitive processing. These bumps identified a separation between a retrieval stage that identified relevant information in memory and a decision stage that determined what response was implied by that information. The encoding, retrieval, decision, and response stages displayed striking similarities across the two experiments in their duration and brain activation patterns. Retrieval and decision processes involve distinct brain activation patterns. We conclude that memory processes for two different tasks, associative recognition versus arithmetic retrieval, follow a common spatiotemporal neural pattern and that both tasks have distinct retrieval and decision stages.

Mapping working memory retrieval in space and in time: A combined electroencephalography and electrocorticography approach.

In this study, we investigated the time course and neural correlates of the retrieval process underlying visual working memory. We made use of a rare dataset in which the same task was recorded using both scalp electroencephalography (EEG) and Electrocorticography (ECoG), respectively. This allowed us to examine with great spatial and temporal detail how the retrieval process works, and in particular how the medial temporal lobe (MTL) is involved. In each trial, participants judged whether a probe face had been among a set of recently studied faces. With a method that combines hidden semi-Markov models and multivariate pattern analysis, the neural signal was decomposed into a sequence of latent cognitive stages with information about their durations on a trial-by-trial basis. Analyzed separately, EEG and ECoG data yielded converging results on discovered stages and their interpretation, which reflected 1) a brief pre-attention stage, 2) encoding the stimulus, 3) retrieving the studied set, and 4) making a decision. Combining these stages with the high spatial resolution of ECoG suggested that activity in the temporal cortex reflected item familiarity in the retrieval stage; and that once retrieval is complete, there is active maintenance of the studied face set in the decision stage in the MTL. During this same period, the frontal cortex guides the decision by means of theta coupling with the MTL. These observations generalize previous findings on the role of MTL theta from long-term memory tasks to short-term memory tasks.

Characterizing synchrony patterns across cognitive task stages of associative recognition memory.

Numerous studies seek to understand the role of oscillatory synchronization in cognition. This problem is particularly challenging in the context of complex cognitive behavior, which consists of a sequence of processing steps with uncertain duration. In this study, we analyzed oscillatory connectivity measures in time windows that previous computational models had associated with a specific sequence of processing steps in an associative memory recognition task (visual encoding, familiarity, memory retrieval, decision making, and motor response). The timing of these processing steps was estimated on a single-trial basis with a novel hidden semi-Markov model multivariate pattern analysis (HSMM-MVPA) method. We show that different processing stages are associated with specific patterns of oscillatory connectivity. Visual encoding is characterized by a dense network connecting frontal, posterior, and temporal areas as well as frontal and occipital phase locking in the 4-9 Hz theta band. Familiarity is associated with frontal phase locking in the 9-14 Hz alpha band. Decision making is associated with frontal and temporo-central interhemispheric connections in the alpha band. During decision making, a second network in the theta band that connects left-temporal, central, and occipital areas bears similarity to the neural signature for preparing a motor response. A similar theta band network is also present during the motor response, with additionally alpha band connectivity between right-temporal and posterior areas. This demonstrates that the processing stages discovered with the HSMM-MVPA method are indeed linked to distinct synchronization patterns, leading to a closer understanding of the functional role of oscillations in cognition.

Inter-subject alignment of MEG datasets in a common representational space.

Pooling neural imaging data across subjects requires aligning recordings from different subjects. In magnetoencephalography (MEG) recordings, sensors across subjects are poorly correlated both because of differences in the exact location of the sensors, and structural and functional differences in the brains. It is possible to achieve alignment by assuming that the same regions of different brains correspond across subjects. However, this relies on both the assumption that brain anatomy and function are well correlated, and the strong assumptions that go into solving the under-determined inverse problem given the high-dimensional source space. In this article, we investigated an alternative method that bypasses source-localization. Instead, it analyzes the sensor recordings themselves and aligns their temporal signatures across subjects. We used a multivariate approach, multiset canonical correlation analysis (M-CCA), to transform individual subject data to a low-dimensional common representational space. We evaluated the robustness of this approach over a synthetic dataset, by examining the effect of different factors that add to the noise and individual differences in the data. On an MEG dataset, we demonstrated that M-CCA performs better than a method that assumes perfect sensor correspondence and a method that applies source localization. Last, we described how the standard M-CCA algorithm could be further improved with a regularization term that incorporates spatial sensor information. Hum Brain Mapp 38:4287-4301, 2017. © 2017 Wiley Periodicals, Inc.

Driving and Multitasking: The Good, the Bad, and the Dangerous.

Previous research has shown that multitasking can have a positive or a negative influence on driving performance. The aim of this study was to determine how the interaction between driving circumstances and cognitive requirements of secondary tasks affect a driver's ability to control a car. We created a driving simulator paradigm where participants had to perform one of two scenarios: one with no traffic in the driver's lane, and one with substantial traffic in both lanes, some of which had to be overtaken. Four different secondary task conditions were combined with these driving scenarios. In both driving scenarios, using a tablet resulted in the worst, most dangerous, performance, while passively listening to the radio or answering questions for a radio quiz led to the best driving performance. Interestingly, driving as a single task did not produce better performance than driving in combination with one of the radio tasks, and even tended to be slightly worse. These results suggest that drivers switch to internally focused secondary tasks when nothing else is available during monotonous or repetitive driving environments. This mind wandering potentially has a stronger interference effect with driving than non-visual secondary tasks.

Strategies for memory-based decision making: Modeling behavioral and neural signatures within a cognitive architecture.

How do people use memories to make inferences about real-world objects? We tested three strategies based on predicted patterns of response times and blood-oxygen-level-dependent (BOLD) responses: one strategy that relies solely on recognition memory, a second that retrieves additional knowledge, and a third, lexicographic (i.e., sequential) strategy, that considers knowledge conditionally on the evidence obtained from recognition memory. We implemented the strategies as computational models within the Adaptive Control of Thought-Rational (ACT-R) cognitive architecture, which allowed us to derive behavioral and neural predictions that we then compared to the results of a functional magnetic resonance imaging (fMRI) study in which participants inferred which of two cities is larger. Overall, versions of the lexicographic strategy, according to which knowledge about many but not all alternatives is searched, provided the best account of the joint patterns of response times and BOLD responses. These results provide insights into the interplay between recognition and additional knowledge in memory, hinting at an adaptive use of these two sources of information in decision making. The results highlight the usefulness of implementing models of decision making within a cognitive architecture to derive predictions on the behavioral and neural level.

Tracking cognitive processing stages with MEG: A spatio-temporal model of associative recognition in the brain.

In this study, we investigated the cognitive processing stages underlying associative recognition using MEG. Over the last four decades, a model of associative recognition has been developed in the ACT-R cognitive architecture. This model was first exclusively based on behavior, but was later evaluated and improved based on fMRI and EEG data. Unfortunately, the limited spatial resolution of EEG and the limited temporal resolution of fMRI have made it difficult to fully understand the spatiotemporal dynamics of associative recognition. We therefore conducted an associative recognition experiment with MEG, which combines excellent temporal resolution with reasonable spatial resolution. To analyze the data, we applied non-parametric cluster analyses and a multivariate classifier. This resulted in a detailed spatio-temporal model of associative recognition. After the visual encoding of the stimuli in occipital regions, three separable memory processes took place: a familiarity process (temporal cortex), a recollection process (temporal cortex and supramarginal gyrus), and a representational process (dorsolateral prefrontal cortex). A late decision process (superior parietal cortex) then acted upon the recollected information represented in the prefrontal cortex, culminating in a late response process (motor cortex). We conclude that existing theories of associative recognition, including the ACT-R model, should be adapted to include these processes.