Feature Selection Methods for Zero-Shot Learning of Neural Activity.

Dimensionality poses a serious challenge when making predictions from human neuroimaging data. Across imaging modalities, large pools of potential neural features (e.g., responses from particular voxels, electrodes, and temporal windows) have to be related to typically limited sets of stimuli and samples. In recent years, zero-shot prediction models have been introduced for mapping between neural signals and semantic attributes, which allows for classification of stimulus classes not explicitly included in the training set. While choices about feature selection can have a substantial impact when closed-set accuracy, open-set robustness, and runtime are competing design objectives, no systematic study of feature selection for these models has been reported. Instead, a relatively straightforward feature stability approach has been adopted and successfully applied across models and imaging modalities. To characterize the tradeoffs in feature selection for zero-shot learning, we compared correlation-based stability to several other feature selection techniques on comparable data sets from two distinct imaging modalities: functional Magnetic Resonance Imaging and Electrocorticography. While most of the feature selection methods resulted in similar zero-shot prediction accuracies and spatial/spectral patterns of selected features, there was one exception; A novel feature/attribute correlation approach was able to achieve those accuracies with far fewer features, suggesting the potential for simpler prediction models that yield high zero-shot classification accuracy.

Semantic attributes are encoded in human electrocorticographic signals during visual object recognition.

Non-invasive neuroimaging studies have shown that semantic category and attribute information are encoded in neural population activity. Electrocorticography (ECoG) offers several advantages over non-invasive approaches, but the degree to which semantic attribute information is encoded in ECoG responses is not known. We recorded ECoG while patients named objects from 12 semantic categories and then trained high-dimensional encoding models to map semantic attributes to spectral-temporal features of the task-related neural responses. Using these semantic attribute encoding models, untrained objects were decoded with accuracies comparable to whole-brain functional Magnetic Resonance Imaging (fMRI), and we observed that high-gamma activity (70-110Hz) at basal occipitotemporal electrodes was associated with specific semantic dimensions (manmade-animate, canonically large-small, and places-tools). Individual patient results were in close agreement with reports from other imaging modalities on the time course and functional organization of semantic processing along the ventral visual pathway during object recognition. The semantic attribute encoding model approach is critical for decoding objects absent from a training set, as well as for studying complex semantic encodings without artificially restricting stimuli to a small number of semantic categories.

Conceptual knowledge representation: A cross-section of current research.

How is conceptual knowledge encoded in the brain? This special issue of Cognitive Neuropsychology takes stock of current efforts to answer this question through a variety of methods and perspectives. Across this work, three questions recur, each fundamental to knowledge representation in the mind and brain. First, what are the elements of conceptual representation? Second, to what extent are conceptual representations embodied in sensory and motor systems? Third, how are conceptual representations shaped by context, especially linguistic context? In this introductory article we provide relevant background on these themes and introduce how they are addressed by our contributing authors.

Attentional and Contextual Priors in Sound Perception.

Behavioral and neural studies of selective attention have consistently demonstrated that explicit attentional cues to particular perceptual features profoundly alter perception and performance. The statistics of the sensory environment can also provide cues about what perceptual features to expect, but the extent to which these more implicit contextual cues impact perception and performance, as well as their relationship to explicit attentional cues, is not well understood. In this study, the explicit cues, or attentional prior probabilities, and the implicit cues, or contextual prior probabilities, associated with different acoustic frequencies in a detection task were simultaneously manipulated. Both attentional and contextual priors had similarly large but independent impacts on sound detectability, with evidence that listeners tracked and used contextual priors for a variety of sound classes (pure tones, harmonic complexes, and vowels). Further analyses showed that listeners updated their contextual priors rapidly and optimally, given the changing acoustic frequency statistics inherent in the paradigm. A Bayesian Observer model accounted for both attentional and contextual adaptations found with listeners. These results bolster the interpretation of perception as Bayesian inference, and suggest that some effects attributed to selective attention may be a special case of contextual prior integration along a feature axis.

What does the right hemisphere know about phoneme categories?

Innate auditory sensitivities and familiarity with the sounds of language give rise to clear influences of phonemic categories on adult perception of speech. With few exceptions, current models endorse highly left-hemisphere-lateralized mechanisms responsible for the influence of phonemic category on speech perception, based primarily on results from functional imaging and brain-lesion studies. Here we directly test the hypothesis that the right hemisphere does not engage in phonemic analysis. By using fMRI to identify cortical sites sensitive to phonemes in both word and pronounceable nonword contexts, we find evidence that right-hemisphere phonemic sensitivity is limited to a lexical context. We extend the interpretation of these fMRI results through the study of an individual with a left-hemisphere lesion who is right-hemisphere reliant for initial acoustic and phonetic analysis of speech. This individual's performance revealed that the right hemisphere alone was insufficient to allow for typical phonemic category effects but did support the processing of gradient phonetic information in lexical contexts. Taken together, these findings confirm previous claims that the right temporal cortex does not play a primary role in phoneme processing, but they also indicate that lexical context may modulate the involvement of a right hemisphere largely tuned for less abstract dimensions of the speech signal.

Mental encoding and neural decoding of abstract cognitive categories: a commentary and simulation.

The premise of Multi-Voxel Pattern Analysis (MVPA) of functional Magnetic Resonance Image (fMRI) data is that mental encodings or states give rise to patterns of neural activation, which in turn, give rise to patterns of blood-oxygen level dependent (BOLD) responses distributed across sets of voxels. Statistical learning algorithms can then be used to detect relationships between mental encodings and BOLD responses, typically through pattern classification. Amongst many other applications, this technique has been used to evidence abstract category representation in an assortment of brain areas and across a range of cognitive domains. In this commentary, we address a critical domain-general caveat to inferring abstract category representation from MVPA that has been partly overlooked in the recent literature: specifically, the distinction between representing specific exemplars within categories, and representing the abstract categories themselves. Using a simulation, we demonstrate that certain forms of MVPA training and testing do not constitute sufficient evidence of category representation, and illustrate prospective and novel retrospective resolutions for this issue.

Brain-behavior relationships in reading acquisition are modulated by socioeconomic factors.

Functional neuroimaging may provide insights into the achievement gap in reading skill commonly observed across socioeconomic status (SES). Brain activation during reading tasks is known to be associated with individual differences in children's phonological language skills. By selecting children of equivalent phonological skill, yet diverse socioeconomic backgrounds, we use functional magnetic resonance imaging (fMRI) to demonstrate that a child's experience, as operationalized by SES, can systematically modulate the relationship between phonological language skills and reading-related brain activity in left fusiform and perisylvian regions. Specifically, at lower socioeconomic levels, individual differences in skill result in large differences in brain activation. In contrast, as SES increases, this relationship between phonological language skill and activation is attenuated. Socioeconomic background factors are thus found to modulate brain-behavior relationships in reading, indicating that cognitive, social, and neurobiological influences on reading development are fundamentally intertwined.

Two brain pathways for attended and ignored words.

The dependency of word processing on spare attentional resources has been debated for several decades. Recent research in the study of selective attention has emphasized the role of task load in determining the fate of ignored information. In parallel to behavioral evidence, neuroimaging data show that the activation generated by unattended stimuli is eliminated in task-relevant brain regions during high attentional load tasks. We conducted an fMRI experiment to explore how word encoding proceeds in a high load situation. Participants saw a rapid series of stimuli consisting of overlapping drawings and letter strings (words or nonwords). In different blocks, task instructions directed attention to either the drawings or the letters, and subjects responded to immediate repetition of items in the attended dimension. To look at the effect of attention on word processing, we compared brain activations for words and nonwords under the two attentional conditions. As compared to nonwords, word stimuli drove responses in left frontal, left temporal and parietal areas when letters were attended. However, although the behavioral measures suggested that ignored words were not analyzed when drawings were attended, a comparison of ignored words to ignored nonwords indicated the involvement of several regions including left insula, right cerebellum and bilateral pulvinar. Interestingly, word-specific activations found when attended and ignored words were compared showed no anatomical overlap, suggesting a change in processing pathways for attended and ignored words presented in a high attentional load task.