Computational reconstruction of mental representations using human behavior.

Revealing how the mind represents information is a longstanding goal of cognitive science. However, there is currently no framework for reconstructing the broad range of mental representations that humans possess. Here, we ask participants to indicate what they perceive in images made of random visual features in a deep neural network. We then infer associations between the semantic features of their responses and the visual features of the images. This allows us to reconstruct the mental representations of multiple visual concepts, both those supplied by participants and other concepts extrapolated from the same semantic space. We validate these reconstructions in separate participants and further generalize our approach to predict behavior for new stimuli and in a new task. Finally, we reconstruct the mental representations of individual observers and of a neural network. This framework enables a large-scale investigation of conceptual representations.

Reclassifying guesses to increase signal-to-noise ratio in psychological experiments.

This paper introduces a novel procedure that can increase the signal-to-noise ratio in psychological experiments that use accuracy as a selection variable for another dependent variable. This procedure relies on the fact that some correct responses result from guesses and reclassifies them as incorrect responses using a trial-by-trial reclassification evidence such as response time. It selects the optimal reclassification evidence criterion beyond which correct responses should be reclassified as incorrect responses. We show that the more difficult the task and the fewer the response alternatives, the more to be gained from this reclassification procedure. We illustrate the procedure on behavioral and ERP data from two different datasets (Caplette et al. NeuroImage 218, 116994, 2020; Faghel-Soubeyrand et al. Journal of Experimental Psychology: General 148, 1834-1841, 2019) using response time as reclassification evidence. In both cases, the reclassification procedure increased signal-to-noise ratio by more than 13%. Matlab and Python implementations of the reclassification procedure are openly available ( https://github.com/GroupeLaboGosselin/Reclassification ).

Rhythmic Information Sampling in the Brain during Visual Recognition.

When we fixate an object, visual information is continuously received on the retina. Several studies observed behavioral oscillations in perceptual sensitivity across such stimulus time, and these fluctuations have been linked to brain oscillations. However, whether specific brain areas show oscillations across stimulus time (i.e., different time points of the stimulus being more or less processed, in a rhythmic fashion) has not been investigated. Here, we revealed random areas of face images at random moments across time and recorded the brain activity of male and female human participants using MEG while they performed two recognition tasks. This allowed us to quantify how each snapshot of visual information coming from the stimulus is processed across time and across the brain. Oscillations across stimulus time (rhythmic sampling) were mostly visible in early visual areas, at theta, alpha, and low beta frequencies. We also found that they contributed to brain activity more than previously investigated rhythmic processing (oscillations in the processing of a single snapshot of visual information). Nonrhythmic sampling was also visible at later latencies across the visual cortex, either in the form of a transient processing of early stimulus time points or of a sustained processing of the whole stimulus. Our results suggest that successive cycles of ongoing brain oscillations process stimulus information incoming at successive moments. Together, these results advance our understanding of the oscillatory neural dynamics associated with visual processing and show the importance of considering the temporal dimension of stimuli when studying visual recognition.SIGNIFICANCE STATEMENT Several behavioral studies have observed oscillations in perceptual sensitivity over the duration of stimulus presentation, and these fluctuations have been linked to brain oscillations. However, oscillations across stimulus time in the brain have not been studied. Here, we developed an MEG paradigm to quantify how visual information received at each moment during fixation is processed through time and across the brain. We showed that different snapshots of a stimulus are distinctly processed in many brain areas and that these fluctuations are oscillatory in early visual areas. Oscillations across stimulus time were more prevalent than previously studied oscillations across processing time. These results increase our understanding of how neural oscillations interact with the visual processing of temporal stimuli.

Flexible time course of spatial frequency use during scene categorization.

Human observers can quickly and accurately categorize scenes. This remarkable ability is related to the usage of information at different spatial frequencies (SFs) following a coarse-to-fine pattern: Low SFs, conveying coarse layout information, are thought to be used earlier than high SFs, representing more fine-grained information. Alternatives to this pattern have rarely been considered. Here, we probed all possible SF usage strategies randomly with high resolution in both the SF and time dimensions at two categorization levels. We show that correct basic-level categorizations of indoor scenes are linked to the sampling of relatively high SFs, whereas correct outdoor scene categorizations are predicted by an early use of high SFs and a later use of low SFs (fine-to-coarse pattern of SF usage). Superordinate-level categorizations (indoor vs. outdoor scenes) rely on lower SFs early on, followed by a shift to higher SFs and a subsequent shift back to lower SFs in late stages. In summary, our results show no consistent pattern of SF usage across tasks and only partially replicate the diagnostic SFs found in previous studies. We therefore propose that SF sampling strategies of observers differ with varying stimulus and task characteristics, thus favouring the notion of flexible SF usage.

Object expectations alter information use during visual recognition.

Prior expectations influence how we perceive and recognize objects. However, how they do so remains unclear, especially in the case of real-world complex objects. Expectations of objects may affect which features are used to recognize them subsequently. In this study, we used reverse correlation to reveal with high precision how the use of information across time is modulated by real-world object expectations in a visual recognition task. We show that coarse information leads to accurate responses earlier when an object is expected, indicating that observers use diagnostic features earlier in this situation. We also demonstrate an increased variability in the use of coarse information depending on the expected object, indicating that observers adopt a more specialized recognition strategy when they expect a specific object. In summary, our results reveal potential mechanisms underlying the effect of expectations on the recognition of complex objects.

Disentangling presentation and processing times in the brain.

Visual object recognition seems to occur almost instantaneously. However, not only does it require hundreds of milliseconds of processing, but our eyes also typically fixate the object for hundreds of milliseconds. Consequently, information reaching our eyes at different moments is processed in the brain together. Moreover, information received at different moments during fixation is likely to be processed differently, notably because different features might be selectively attended at different moments. Here, we introduce a novel reverse correlation paradigm that allows us to uncover with millisecond precision the processing time course of specific information received on the retina at specific moments. Using faces as stimuli, we observed that processing at several electrodes and latencies was different depending on the moment at which information was received. Some of these variations were caused by a disruption occurring 160-200 â€‹ms after the face onset, suggesting a role of the N170 ERP component in gating information processing; others hinted at temporal compression and integration mechanisms. Importantly, the observed differences were not explained by simple adaptation or repetition priming, they were modulated by the task, and they were correlated with differences in behavior. These results suggest that top-down routines of information sampling are applied to the continuous visual input, even within a single eye fixation.

Real-world expectations and their affective value modulate object processing.

It is well known that expectations influence how we perceive the world. Yet the neural mechanisms underlying this process remain unclear. Studies about the effects of prior expectations have focused so far on artificial contingencies between simple neutral cues and events. Real-world expectations are however often generated from complex associations between contexts and objects learned over a lifetime. Additionally, these expectations may contain some affective value and recent proposals present conflicting hypotheses about the mechanisms underlying affect in predictions. In this study, we used fMRI to investigate how object processing is influenced by realistic context-based expectations, and how affect impacts these expectations. First, we show that the precuneus, the inferotemporal cortex and the frontal cortex are more active during object recognition when expectations have been elicited a priori, irrespectively of their validity or their affective intensity. This result supports previous hypotheses according to which these brain areas integrate contextual expectations with object sensory information. Notably, these brain areas are different from those responsible for simultaneous context-object interactions, dissociating the two processes. Then, we show that early visual areas, on the contrary, are more active during object recognition when no prior expectation has been elicited by a context. Lastly, BOLD activity was shown to be enhanced in early visual areas when objects are less expected, but only when contexts are neutral; the reverse effect is observed when contexts are affective. This result supports the proposal that affect modulates the weighting of sensory information during predictions. Together, our results help elucidate the neural mechanisms of real-world expectations.

Hand position alters vision by modulating the time course of spatial frequency use.

The nervous system gives preferential treatment to objects near the hands that are candidates for action. It is not yet understood how this process is achieved. Here we show evidence for the mechanism that underlies this process having used an experimental technique that maps the use of spatial frequencies (SFs) during object recognition across time. We used this technique to replicate and characterize with greater precision the coarse-to-fine SF sampling observed in previous studies. Then we show that the visual processing of real-world objects near an observer's hands is biased toward the use of low-SF information, around 288 ms. Conversely, high-SF information presented around 113 ms impaired object recognition when objects were presented near the hands. Notably, both of these effects happened relatively late during object recognition and suggest that the modulation of SF use by hand position is at least partly attentional in nature. (PsycINFO Database Record

Atypical Time Course of Object Recognition in Autism Spectrum Disorder.

In neurotypical observers, it is widely believed that the visual system samples the world in a coarse-to-fine fashion. Past studies on Autism Spectrum Disorder (ASD) have identified atypical responses to fine visual information but did not investigate the time course of the sampling of information at different levels of granularity (i.e. Spatial Frequencies, SF). Here, we examined this question during an object recognition task in ASD and neurotypical observers using a novel experimental paradigm. Our results confirm and characterize with unprecedented precision a coarse-to-fine sampling of SF information in neurotypical observers. In ASD observers, we discovered a different pattern of SF sampling across time: in the first 80 ms, high SFs lead ASD observers to a higher accuracy than neurotypical observers, and these SFs are sampled differently across time in the two subject groups. Our results might be related to the absence of a mandatory precedence of global information, and to top-down processing abnormalities in ASD.

Affective and contextual values modulate spatial frequency use in object recognition.

Visual object recognition is of fundamental importance in our everyday interaction with the environment. Recent models of visual perception emphasize the role of top-down predictions facilitating object recognition via initial guesses that limit the number of object representations that need to be considered. Several results suggest that this rapid and efficient object processing relies on the early extraction and processing of low spatial frequencies (LSF). The present study aimed to investigate the SF content of visual object representations and its modulation by contextual and affective values of the perceived object during a picture-name verification task. Stimuli consisted of pictures of objects equalized in SF content and categorized as having low or high affective and contextual values. To access the SF content of stored visual representations of objects, SFs of each image were then randomly sampled on a trial-by-trial basis. Results reveal that intermediate SFs between 14 and 24 cycles per object (2.3-4 cycles per degree) are correlated with fast and accurate identification for all categories of objects. Moreover, there was a significant interaction between affective and contextual values over the SFs correlating with fast recognition. These results suggest that affective and contextual values of a visual object modulate the SF content of its internal representation, thus highlighting the flexibility of the visual recognition system.

Real-world interattribute distances lead to inefficient face gender categorization.

The processing of interattribute distances is believed to be critical for upright face categorization. A recent study by Taschereau-Dumouchel, Rossion, Schyns, and Gosselin (2010) challenged this idea by showing that participants were nearly at chance when asked to identify faces on the sole basis of real-world interattribute distances, while they were nearly perfect when all other facial cues were shown. However, it remains possible that humans are highly tuned to interattribute distances but that the information conveyed by these cues is scarce. We tested this hypothesis by contrasting the efficiencies-a measure of performance that factors out task difficulty-of 60 observers in 6 face gender categorization tasks. Our main finding is that efficiencies for faces that varied only in terms of their interattribute distances were an order of magnitude lower than efficiencies for faces that varied in all respects, except their interattribute distances, or in all respects. These results provide a definitive blow to the idea that real-world interattribute distances are critical for upright face processing. (PsycINFO Database Record (c) 2014 APA, all rights reserved).