A Scalable Framework for Closed-Loop Neuromodulation with Deep Learning.

Closed-loop neuromodulation measures dynamic neural or physiological activity to optimize interventions for clinical and nonclinical behavioral, cognitive, wellness, attentional, or general task performance enhancement. Conventional closed-loop stimulation approaches can contain biased biomarker detection (decoders and error-based triggering) and stimulation-type application. We present and verify a novel deep learning framework for designing and deploying flexible, data-driven, automated closed-loop neuromodulation that is scalable using diverse datasets, agnostic to stimulation technology (supporting multi-modal stimulation: tACS, tDCS, tFUS, TMS), and without the need for personalized ground-truth performance data. Our approach is based on identified periods of responsiveness - detected states that result in a change in performance when stimulation is applied compared to no stimulation. To demonstrate our framework, we acquire, analyze, and apply a data-driven approach to our open sourced GX dataset, which includes concurrent physiological (ECG, EOG) and neuronal (EEG) measures, paired with continuous vigilance/attention-fatigue tracking, and High-Definition transcranial electrical stimulation (HD-tES). Our framework's decision process for intervention application identified 88.26% of trials as correct applications, showed potential improvement with varying stimulation types, or missed opportunities to stimulate, whereas 11.25% of trials were predicted to stimulate at inopportune times. With emerging datasets and stimulation technologies, our unifying and integrative framework; leveraging deep learning (Convolutional Neural Networks - CNNs); demonstrates the adaptability and feasibility of automated multimodal neuromodulation for both clinical and nonclinical applications.

Similarity judgments predict N400 amplitude differences between taxonomic category members and thematic associates.

Human similarity judgments do not reliably conform to the predictions of leading theories of psychological similarity. Evidence from the triad similarity judgment task shows that people often identify thematic associates like dog and bone as more similar than taxonomic category members like dog and cat, even though thematic associates lack the type of featural or relational similarity that is foundational to theories of psychological similarity. This specific failure to predict human behavior has been addressed as a consequence of education and other individual differences, an artifact of the triad similarity judgment paradigm, or a shortcoming in psychological accounts of similarity. We investigated the judged similarity of semantically-related concepts (taxonomic category members and thematic associates) as it relates to other task-independent measures of semantic knowledge and access. Participants were assessed on reading and language ability, then event-related potentials (ERPs) were collected during a passive, sequential word reading task that presented pseudowords and taxonomically-related, thematically-related, and unrelated word sequences, and, finally, similarity judgments were collected with the classic two-alternative forced-choice triad task. The results uncovered a correspondence between ERP amplitude and triad-based similarity judgments-similarity judgment behavior reliably predicts ERP amplitude during passive word reading, absent of any instruction to consider similarity. It was also found that individual differences in reading and language ability independently predicted ERP amplitude. This evidence suggests that similarity judgments are driven by reliable patterns of thought that are not solely rooted in the interpretation of task goals or reading and language ability.

An Event-Related Potential study of letter spacing during visual word recognition.

Increasing spacing between letters in words (e.g., s p a c e vs. space) helps children and adults read more fluidly and with fewer errors. This effect has been demonstrated behaviorally, chiefly through lexical decision reaction time and total paragraph reading time. To date, however, no electrophysiological work has examined the letter spacing effect, resulting in little insight regarding how letter spacing impacts "hidden" levels of processing between apprehension of the word form and the final behavioral outcome. Here, we examined how varying levels of interletter spacing (crowded, standard, and increased) impact ERPs elicited by words and other item types (pseudowords, illegal strings, and a false font). Results indicate that letter spacing does not impact the ERP within the first second after viewing a wordform, but that it does have downstream effects as indicated by data collected using a priming design. Further, the facilitation of downstream processing afforded by increased spacing seems to be greater for more word-like stimuli (e.g., greater for words than for consonant strings, etc). Results are discussed as being somewhat inconsistent with prevalent views of the mechanism of the letter spacing effect (i.e., crowding).

Time will tell: A longitudinal investigation of brain-behavior relationships during reading development.

ERPs are a powerful tool for the study of reading, as they are both temporally precise and functionally specific. These are essential characteristics for studying a process that unfolds rapidly and consists of multiple, interactive subprocesses. In work with adults, clear, specific models exist linking components of the ERP with individual subprocesses of reading including orthographic decoding, phonological processing, and semantic access (e.g., Grainger & Holcomb, 2009). The relationships between ERP components and reading subprocesses are less clear in development; here, we address two questions regarding these relationships. First, we ask whether there are ERP markers that predict future reading behaviors across a longitudinal year. Second, we ask whether any relationships observed between ERP components and reading behavior across time map onto the better-established relationships between ERPs and reading subprocesses in adults. To address these questions, we acquired ERPs from children engaging in a silent reading task and then, a year later, collected behavioral assessments of their reading ability. We find that ERPs collected in Year 1 do predict reading behaviors a year later. Further, we find that these relationships do conform, at least to some extent, to relationships between ERP components and reading subprocesses observed in adults, with, for example, N250 amplitude in Year 1 predicting phonological awareness in Year 2, and N400 amplitude in Year 1 predicting vocabulary in Year 2.

Evaluating the efficacy of fully automated approaches for the selection of eyeblink ICA components.

Independent component analysis (ICA) offers a powerful approach for the isolation and removal of eyeblink artifacts from EEG signals. Manual identification of the eyeblink ICA component by inspection of scalp map projections, however, is prone to error, particularly when nonartifactual components exhibit topographic distributions similar to the blink. The aim of the present investigation was to determine the extent to which automated approaches for selecting eyeblink-related ICA components could be utilized to replace manual selection. We evaluated popular blink selection methods relying on spatial features (EyeCatch), combined stereotypical spatial and temporal features (ADJUST), and a novel method relying on time series features alone (icablinkmetrics) using both simulated and real EEG data. The results of this investigation suggest that all three methods of automatic component selection are able to accurately identify eyeblink-related ICA components at or above the level of trained human observers. However, icablinkmetrics, in particular, appears to provide an effective means of automating ICA artifact rejection while at the same time eliminating human errors inevitable during manual component selection and false positive component identifications common in other automated approaches. Based upon these findings, best practices for (a) identifying artifactual components via automated means, and (b) reducing the accidental removal of signal-related ICA components are discussed.

An event-related potential study of the relationship between N170 lateralization and phonological awareness in developing readers.

As reading development progresses, the visual processing of word forms becomes increasingly left-lateralized. This is visible, among other ways, as increased left-lateralization of the N170 ERP component. A primary explanation of this effect, the phonological mapping hypothesis, proposes that the left-lateralization of visual word form processing that accompanies reading development is the result of calling upon left hemisphere auditory language regions to perform the linking of orthography with phonology (phonological mapping). A key, but untested, prediction of the phonological mapping hypothesis is thus that individuals with greater phonological awareness should exhibit more left lateralized visual processing of word forms than individuals with poorer phonological awareness. We set out to test this hypothesis here. We accomplished this by collecting ERPs while children grades 5-6 viewed words, objects, and word/object ambiguous items (e.g., "SMILE" shaped like a smile - hereafter referred to as wobjects). Results revealed that, consistent with the phonological mapping hypothesis, individual phonological awareness (but not other measures of reading development) predicted left-lateralization of the N170 component elicited in response to words (but not item types that were not word-like).

Relationships between event-related potentials and behavioral and scholastic measures of reading ability: A large-scale, cross-sectional study.

In the cognitive, computational, neuropsychological, and educational literatures, it is established that children approach text in unique ways, and that even adult readers can differ in the strategies they bring to reading. In the developmental event-related potential (ERP) literature, however, children with differing degrees of reading ability are, the majority of the time, placed in monolithic groups such as 'normal' and 'dyslexic' (e.g. Araújo et al., 2012) and analyzed only at the group level. This is likely done due to methodological concerns - such as low sample size or a lack of statistical power - that can make it difficult to perform analysis at the individual level. Here, we collected ERPs and behavior from > 100 children in grades pre-K-7, as they read unconnected text silently to themselves. This large sample, combined with the statistical power of the Linear Mixed Effects Regression (LMER) technique, enables us to address individual differences in ERP component effects due to reading ability at an unprecedented level of detail. Results indicate that it is possible to predict reading-related report card scores from ERP component amplitudes - especially that of the N250, a component pertaining to sublexical processing (including phonological decoding). Results also reveal relationships between behavioral measures of reading ability and ERP component effects that have previously been elusive, such as the relationship between vocabulary and N400 mean amplitude (cf. Henderson et al., 2011). We conclude that it is possible to meaningfully examine reading-related ERP effects at the single subject level in developing readers, and that this type of analysis can provide novel insights into both behavior and scholastic achievement.

Revisiting the Single-Case Approach to Studying Reading Disorders.

Oscar Marin was a neurologist with a remarkably broad interest in the brain and its function. He was passionate about understanding how the brain processes language and about helping people with acquired language disorders through his science-based practice. Here we honor his memory by presenting a review and commentary charting the cycle of neuroscientific approaches to studying reading disorders over the past century. During this time, "best practices" have changed from individual case studies to group studies and mega-studies and back again to individual studies. We show how, across decades and almost unimaginable advances in neuroimaging technology, the individual approach taken by Oscar Marin has retained its importance.

Individual differences in involvement of the visual object recognition system during visual word recognition.

Individuals with dyslexia often evince reduced activation during reading in left hemisphere (LH) language regions. This can be observed along with increased activation in the right hemisphere (RH), especially in areas associated with object recognition - a pattern referred to as RH compensation. The mechanisms of RH compensation are relatively unclear. We hypothesize that RH compensation occurs when the RH object recognition system is called upon to supplement an underperforming LH visual word form recognition system. We tested this by collecting ERPs while participants with a range of reading abilities viewed words, objects, and word/object ambiguous items (e.g., "SMILE" shaped like a smile). Less experienced readers differentiate words, objects, and ambiguous items less strongly, especially over the RH. We suggest that this lack of differentiation may have negative consequences for dyslexic individuals demonstrating RH compensation.

A direct comparison of active and passive amplification electrodes in the same amplifier system.

BACKGROUND: Active amplification electrodes are becoming more popular for ERP data collection, as they amplify the EEG at the scalp and thereby potentially decrease the influence of ambient electrical noise. However, the performance of active electrodes has not been directly compared with that of passive electrodes in the context of collecting ERPs from a cognitive task. Here, the performance of active and passive amplification electrodes in the same digitizing amplifier system was examined. METHOD: In Experiment 1, interelectrode impedance in an electrically quiet setting was manipulated to determine whether, in such recording conditions, active electrodes can outperform passive ones. In Experiment 2, the performance of active electrodes at the limits of natural skin impedance was explored, as was the relationship between active amplification circuitry and voltage stability in averaged EOG. RESULTS: Results reveal a complex pattern of interrelations between electrode type, impedance, and voltage stability, indicating that which type of electrode is "best" depends non-trivially on the circumstances in which data are being collected. COMPARISON WITH EXISTING METHODS: Traditional, passive electrodes acquired the cleanest data observed in any of the acquisition conditions at very low impedance, but not at any impedance >2 kΩ. CONCLUSION: Active electrodes perform better than passive ones at all impedances other than very low ones; however, this is qualified by the additional finding that during fast voltage fluctuations, such as those most desirable in most ERP studies, active electrodes are less able to accurately follow the EEG than passive ones.

Never Seem to Find the Time: Evaluating the Physiological Time Course of Visual Word Recognition with Regression Analysis of Single Item ERPs.

Visual word recognition is a process that, both hierarchically and in parallel, draws on different types of information ranging from perceptual to orthographic to semantic. A central question concerns when and how these different types of information come online and interact after a word form is initially perceived. Numerous studies addressing aspects of this question have been conducted with a variety of techniques (e.g., behavior, eye-tracking, ERPs), and divergent theoretical models, suggesting different overall speeds of word processing, have coalesced around clusters of mostly method-specific results. Here, we examine the time course of influence of variables ranging from relatively perceptual (e.g., bigram frequency) to relatively semantic (e.g., number of lexical associates) on ERP responses, analyzed at the single item level. Our results, in combination with a critical review of the literature, suggest methodological, analytic, and theoretical factors that may have led to inconsistency in results of past studies; we will argue that consideration of these factors may lead to a reconciliation between divergent views of the speed of word recognition.

PSPs and ERPs: applying the dynamics of post-synaptic potentials to individual units in simulation of temporally extended Event-Related Potential reading data.

The Parallel Distributed Processing (PDP) framework is built on neural-style computation, and is thus well-suited for simulating the neural implementation of cognition. However, relatively little cognitive modeling work has concerned neural measures, instead focusing on behavior. Here, we extend a PDP model of reading-related components in the Event-Related Potential (ERP) to simulation of the N400 repetition effect. We accomplish this by incorporating the dynamics of cortical post-synaptic potentials--the source of the ERP signal--into the model. Simulations demonstrate that application of these dynamics is critical for model elicitation of repetition effects in the time and frequency domains. We conclude that by advancing a neurocomputational understanding of repetition effects, we are able to posit an interpretation of their source that is both explicitly specified and mechanistically different from the well-accepted cognitive one.

Won't get fooled again: An event-related potential study of task and repetition effects on the semantic processing of items without semantics.

A growing body of evidence suggests that semantic access is obligatory. Several studies have demonstrated that brain activity associated with semantic processing, measured in the N400 component of the event-related brain potential (ERP), is elicited even by meaningless, orthographically illegal strings, suggesting that semantic access is not gated by lexicality. However, the downstream consequences of that activity vary by item type, exemplified by the typical finding that N400 activity is reduced by repetition for words and pronounceable nonwords but not for illegal strings. We propose that this lack of repetition effect for illegal strings is caused not by lack of contact with semantics, but by the unrefined nature of that contact under conditions in which illegal strings can be readily categorised as task-irrelevant. To test this, we collected ERPs from participants performing a modified Lexical Decision Task, in which the presence of orthographically illegal acronyms rendered meaningless illegal strings more difficult lures than normal. Confirming our hypothesis, under these conditions illegal strings elicited robust N400 repetition effects, quantitatively and qualitatively similar to those elicited by words, pseudowords, and acronyms.

A neurally plausible parallel distributed processing model of event-related potential word reading data.

The Parallel Distributed Processing (PDP) framework has significant potential for producing models of cognitive tasks that approximate how the brain performs the same tasks. To date, however, there has been relatively little contact between PDP modeling and data from cognitive neuroscience. In an attempt to advance the relationship between explicit, computational models and physiological data collected during the performance of cognitive tasks, we developed a PDP model of visual word recognition which simulates key results from the ERP reading literature, while simultaneously being able to successfully perform lexical decision-a benchmark task for reading models. Simulations reveal that the model's success depends on the implementation of several neurally plausible features in its architecture which are sufficiently domain-general to be relevant to cognitive modeling more generally.

The N400 as a snapshot of interactive processing: Evidence from regression analyses of orthographic neighbor and lexical associate effects.

Linking print with meaning tends to be divided into subprocesses, such as recognition of an input's lexical entry and subsequent access of semantics. However, recent results suggest that the set of semantic features activated by an input is broader than implied by a view wherein access serially follows recognition. EEG was collected from participants who viewed items varying in number and frequency of both orthographic neighbors and lexical associates. Regression analysis of single item ERPs replicated past findings, showing that N400 amplitudes are greater for items with more neighbors, and further revealed that N400 amplitudes increase for items with more lexical associates and with higher frequency neighbors or associates. Together, the data suggest that in the N400 time window semantic features of items broadly related to inputs are active, consistent with models in which semantic access takes place in parallel with stimulus recognition.

A Beautiful Day in the Neighborhood: An Event-Related Potential Study of Lexical Relationships and Prediction in Context.

Two related questions critical to understanding the predictive processes that come online during sentence comprehension are 1) what information is included in the representation created through prediction and 2) at what functional stage does top-down, predicted information begin to affect bottom-up word processing? We investigated these questions by recording event-related potentials (ERPs) as participants read sentences that ended with expected words or with unexpected items (words, pseudowords, or illegal strings) that were either orthographically unrelated to the expected word or were one of its orthographic neighbors. The data show that, regardless of lexical status, attempts at semantic access (N400) for orthographic neighbors of expected words is facilitated relative to the processing of orthographically unrelated items. Our findings support a view of sentence processing wherein orthographically organized information is brought online by prediction and interacts with input prior to any filter on lexical status.

Minding the PS, queues, and PXQs: uniformity of semantic processing across multiple stimulus types.

An assumption in the reading literature is that access to semantics is gated by stimulus properties such as orthographic regularity or familiarity. In the electrophysiological domain, this assumption has led to a debate about the features necessary to initiate semantic processing as indexed by the N400 event-related potential (ERP) component. To examine this, we recorded ERPs to sentences with endings that were familiar and legal (words), familiar and illegal (acronyms), or unfamiliar and illegal (consonant or vowel strings). N400 congruency effects (reduced negativity to expected relative to unexpected endings) were observed for words and acronyms; these were identical in size, timing, and scalp distribution. Notably, clear N400 potentials were also elicited by unfamiliar, illegal strings, suggesting that, at least in a verbal context, semantic access may be attempted for any letter string, regardless of familiarity or regularity.

Better the DVL you know: acronyms reveal the contribution of familiarity to single-word reading.

Current theories of reading are divided between dual-route accounts, which propose that separable processes subserve word recognition for orthographically regular and irregular strings, and connectionist models, which propose a single mechanism mapping form to meaning. These theories make distinct predictions about the processing of acronyms, which can be orthographically illegal and yet familiar, as compared with the processing of pseudowords, which are regular but unfamiliar. This study examined whether acronyms are processed like pseudowords and words. Event-related potentials (ERPs) were recorded as subjects viewed familiar and unfamiliar acronyms, words, pseudowords, illegal strings, and--as the targets of the substantive behavioral task--proper names. Familiar acronyms elicited repetition effects on the N400 component, a functionally specific index of semantic activation processes; repetition effects for familiar acronyms were similar in magnitude, timing, and scalp distribution to those for words and pseudowords. The similarity of the brain response to familiar--but--illegal and unfamiliar--but--legal classes of stimuli is inconsistent with predictions of dual-route models of reading.

The acronym superiority effect.

The visual world is replete with noisy, continuous, perceptually variant linguistic information, which fluent readers rapidly translate from percept to meaning. What are the properties the language comprehension system uses as cues to initiate lexical/semantic access in response to some, but not all, orthographic strings? In the behavioral, electromagnetic, and neuropsychological literatures, orthographic regularity and familiarity have been identified as critical factors. Here, we present a study in the Reicher-Wheeler tradition that manipulates these two properties independently through the use of four stimulus categories: familiar and orthographically regular words, unfamiliar but regular pseudowords, unfamiliar illegal strings, and familiar but orthographically illegal acronyms. We find that, like letters in words and pseudowords, letters in acronyms enjoy an identification benefit relative to similarly illegal, but unfamiliar strings. This supports theories of visual word recognition in which familiarity, rather than orthographic regularity, plays a critical role in gating processing.