Improved prediction of behavioral and neural similarity spaces using pruned DNNs.

Deep Neural Networks (DNNs) have become an important tool for modeling brain and behavior. One key area of interest has been to apply these networks to model human similarity judgements. Several previous works have used the embeddings from the penultimate layer of vision DNNs and showed that a reweighting of these features improves the fit between human similarity judgments and DNNs. These studies underline the idea that these embeddings form a good basis set but lack the correct level of salience. Here we re-examined the grounds for this idea and on the contrary, we hypothesized that these embeddings, beyond forming a good basis set, also have the correct level of salience to account for similarity judgments. It is just that the huge dimensional embedding needs to be pruned to select those features relevant for the considered domain for which a similarity space is modeled. In Study 1 we supervised DNN pruning based on a subset of human similarity judgments. We found that pruning: i) improved out-of-sample prediction of human similarity judgments from DNN embeddings, ii) produced better alignment with WordNet hierarchy, and iii) retained much higher classification accuracy than reweighting. Study 2 showed that pruning by neurobiological data is highly effective in improving out-of-sample prediction of brain-derived representational dissimilarity matrices from DNN embeddings, at times fleshing out isomorphisms not otherwise observable. Using pruned DNNs, image-level heatmaps can be produced to identify image sections whose features load on dimensions coded by a brain area. Pruning supervised by human brain/behavior therefore effectively identifies alignable dimensions of knowledge between DNNs and humans and constitutes an effective method for understanding the organization of knowledge in neural networks.

Representations of conceptual information during automatic and active semantic access.

When we read a word or see an object, conceptual meaning is automatically accessed. However, previous research investigating non-perceptual sensitivity to semantic class has employed active tasks. In this fMRI study, we tested whether conceptual representations in regions constituting the semantic network are invoked during passive semantic access and whether these representations are modulated by the need to access deeper knowledge. Seventeen healthy subjects performed a semantically active typicality judgment task and a semantically passive phonetic decision task, in both the written and the spoken input-modalities. Stimuli consisted of one hundred forty-four concepts drawn from six semantic categories. Multivariate Pattern Analysis (MVPA) revealed that the left posterior middle temporal gyrus (pMTG), posterior ventral temporal cortex (pVTC) and pars triangularis of the left inferior frontal gyrus (IFG) showed a stronger sensitivity to semantic category when active rather than passive semantic access is required. Using a cross-task training/testing classifier, we determined that conceptual representations were not only active in these regions during passive semantic access but that the neural representation of these categories was common to both active and passive access. Collectively, these results show that while representations in the pMTG, pVTC and IFG are strongly modulated by active conceptual access, consistent representational patterns are present during active and passive conceptual access in these same regions.

General and feature-based semantic representations in the semantic network.

How semantic representations are manifest over the brain remains a topic of active debate. A semantic representation may be determined by specific semantic features (e.g. sensorimotor information), or may abstract away from specific features and represent generalized semantic characteristics (general semantic representation). Here we tested whether nodes of the semantic system code for a general semantic representation and/or possess representational spaces linked to particular semantic features. In an fMRI study, eighteen participants performed a typicality judgment task with written words drawn from sixteen different categories. Multivariate pattern analysis (MVPA) and representational similarity analysis (RSA) were adopted to investigate the sensitivity of the brain regions to semantic content and the type of semantic representation coded (general or feature-based). We replicated previous findings of sensitivity to general semantic similarity in posterior middle/inferior temporal gyrus (pMTG/ITG) and precuneus (PC) and additionally observed general semantic representations in ventromedial prefrontal cortex (PFC). Finally, two brain regions of the semantic network were sensitive to semantic features: the left pMTG/ITG was sensitive to haptic perception and the left ventral temporal cortex (VTC) to size. This finding supports the involvement of both general semantic representation and feature-based representations in the brain's semantic system.