A simple rule to describe interactions between visual categories.

Humans can rapidly categorise visual objects when presented in isolation. However, in everyday life we encounter multiple objects at the same time. Far less is known about how simultaneously active object representations interact. We examined such interactions by asking participants to categorise a target object at the basic (Experiment 1) or the superordinate (Experiment 2) level while the representation of another object was still active. We found that the "prime" object strongly modulated the response to the target implying that the prime's category was rapidly and automatically accessed, influencing subsequent categorical processing. Using drift diffusion modelling, we show that a prime, whose category is different from that of the target, interferes with target processing primarily during the evidence accumulation stage. This suggests that the state of category-processing neurons is altered by an active representation and this modifies the processing of other categories. Interestingly, the strength of interference increases with the similarity between the distractor and the target category. Considering these results and previous studies, we propose a general principle that category interactions are determined by the distance from a distractor's representation to the target's task-relevant categorical boundary. We argue that this principle arises from the specific architectural organisation of categories in the brain.

At 120 msec you can spot the animal but you don't yet know it's a dog.

Earlier studies suggested that the visual system processes information at the basic level (e.g., dog) faster than at the subordinate (e.g., Dalmatian) or superordinate (e.g., animals) levels. However, the advantage of the basic category over the superordinate category in object recognition has been challenged recently, and the hierarchical nature of visual categorization is now a matter of debate. To address this issue, we used a forced-choice saccadic task in which a target and a distractor image were displayed simultaneously on each trial and participants had to saccade as fast as possible toward the image containing animal targets based on different categorization levels. This protocol enables us to investigate the first 100-120 msec, a previously unexplored temporal window, of visual object categorization. The first result is a surprising stability of the saccade latency (median RT ∼ 155 msec) regardless of the animal target category and the dissimilarity of target and distractor image sets. Accuracy was high (around 80% correct) for categorization tasks that can be solved at the superordinate level but dropped to almost chance levels for basic level categorization. At the basic level, the highest accuracy (62%) was obtained when distractors were restricted to another dissimilar basic category. Computational simulations based on the saliency map model showed that the results could not be predicted by pure bottom-up saliency differences between images. Our results support a model of visual recognition in which the visual system can rapidly access relatively coarse visual representations that provide information at the superordinate level of an object, but where additional visual analysis is required to allow more detailed categorization at the basic level.

Stimulus duration and diversity do not reverse the advantage for superordinate-level representations: the animal is seen before the bird.

Basic-level categorization has long been thought to be the entry level for object representations. However, this view is now challenged. In particular, Macé et al. [M.J.-M. Macé et al. (2009) PLoS One, 4, e5927] showed that basic-level categorization (such as 'bird') requires a longer processing time than superordinate-level categorization (such as 'animal'). It has been argued that this result depends on the brief stimulus presentation times used in their study, which would degrade the visual information available. Here, we used a go/no-go paradigm to test whether the superordinate-level advantage could be observed with longer stimulus durations, and also investigated the impact of manipulating the target and distractor set heterogeneity. Our results clearly show that presentation time had no effect on categorization performance. Both target and distractor diversity influenced performance, but basic-level categories were never accessed faster or with higher accuracy than superordinate-level categories. These results argue in favor of coarse to fine visual processing to access perceptual representations.

Incongruent object/context relationships in visual scenes: where are they processed in the brain?

Rapid object visual categorization in briefly flashed natural scenes is influenced by the surrounding context. The neural correlates underlying reduced categorization performance in response to incongruent object/context associations remain unclear and were investigated in the present study using fMRI. Participants were instructed to categorize objects in briefly presented scenes (exposure duration=100ms). Half of the scenes consisted of objects pasted in an expected (congruent) context, whereas for the other half, objects were embedded in incongruent contexts. Object categorization was more accurate and faster in congruent relative to incongruent scenes. Moreover, we found that the two types of scenes elicited different patterns of cerebral activation. In particular, the processing of incongruent scenes induced increased activations in the parahippocampal cortex, as well as in the right frontal cortex. This higher activity may indicate additional neural processing of the novel (non experienced) contextual associations that were inherent to the incongruent scenes. Moreover, our results suggest that the locus of object categorization impairment due to contextual incongruence is in the right anterior parahippocampal cortex. Indeed in this region activity was correlated with the reaction time increase observed with incongruent scenes. Representations for associations between objects and their usual context of appearance might be encoded in the right anterior parahippocampal cortex.

Humans and monkeys share visual representations.

Conceptual abilities in animals have been shown at several levels of abstraction, but it is unclear whether the analogy with humans results from convergent evolution or from shared brain mechanisms inherited from a common origin. Macaque monkeys can access "non-similarity-based concepts," such as when sorting pictures containing a superordinate target category (animal, tree, etc.) among other scenes. However, such performances could result from low-level visual processing based on learned regularities of the photographs, such as for scene categorization by artificial systems. By using pictures of man-made objects or animals embedded in man-made or natural contexts, the present study clearly establishes that macaque monkeys based their categorical decision on the presence of the animal targets regardless of the scene backgrounds. However, as is found with humans, monkeys performed better with categorically congruent object/context associations, especially when small object sizes favored background information. The accuracy improvements and the response-speed gains attributable to superordinate category congruency in monkeys were strikingly similar to those of human subjects tested with the same task and stimuli. These results suggest analogous processing of visual information during the activation of abstract representations in both humans and monkeys; they imply a large overlap between superordinate visual representations in humans and macaques as well as the implicit use of experienced associations between object and context.

A need for more information uptake but not focused attention to access basic-level representations.

Complex visual scenes can be categorized at the superordinate level (e.g., animal/non-animal or vehicle/non-vehicle) without focused attention. However, rapid visual categorization at the basic level (e.g., dog/non-dog or car/non-car) requires additional processing time. Such finer categorization might, thus, require attentional resources. This hypothesis was tested in the current study with a dual-task paradigm in which subjects performed a basic-level categorization task in peripheral vision either alone (single-task condition) or concurrently with an attentionally demanding letter discrimination task (dual-task condition). Our results indicate that basic-level categorization of either biological (dog/non-dog animal) or man-made (car/non-car vehicle) stimuli requires more information uptake but can, nevertheless, be performed when attention is not fully available, presumably because it is supported by hardwired, specialized neuronal networks.

Spotting animals in natural scenes: efficiency of humans and monkeys at very low contrasts.

The ability of monkeys to categorize objects in visual stimuli such as natural scenes might rely on sets of low-level visual cues without any underlying conceptual abilities. Using a go/no-go rapid animal/non-animal categorization task with briefly flashed achromatic natural scenes, we show that both human and monkey performance is very robust to large variations of stimulus luminance and contrast. When mean luminance was increased or decreased by 25-50%, accuracy and speed impairments were small. The largest impairment was found at the highest luminance value with monkeys being mainly impaired in accuracy (drop of 6% correct vs. <1.5% in humans), whereas humans were mainly impaired in reaction time (20 ms increase in median reaction time vs. 4 ms in monkeys). Contrast reductions induced a large deterioration of image definition, but performance was again remarkably robust. Subjects scored well above chance level, even when the contrast was only 12% of the original photographs ( approximately 81% correct in monkeys; approximately 79% correct in humans). Accuracy decreased with contrast reduction but only reached chance level -in both species- for the most extreme condition, when only 3% of the original contrast remained. A progressive reaction time increase was observed that reached 72 ms in monkeys and 66 ms in humans. These results demonstrate the remarkable robustness of the primate visual system in processing objects in natural scenes with large random variations in luminance and contrast. They illustrate the similarity with which performance is impaired in monkeys and humans with such stimulus manipulations. They finally show that in an animal categorization task, the performance of both monkeys and humans is largely independent of cues relying on global luminance or the fine definition of stimuli.

Rapid categorization of faces and objects in a patient with impaired object recognition.

We tested rapid-categorization in a patient who was impaired in face and object recognition. Photographs of natural scenes were displayed for 100 ms. Participants had to press a key when they saw an animal among various objects as distractors or human faces among animal faces as distractors. Though the patient was impaired at figure/ground segregation, recognized very few objects and faces, she categorized animals and faces with a performance ranging between 70 and 86% correct. Displaying pictures in isolation did not improve performance. The results suggest that rapid categorization can be accomplished on the basis of coarse information without overt recognition.

Rapid visual categorization of natural scene contexts with equalized amplitude spectrum and increasing phase noise.

This study aimed to determine the extent to which rapid visual context categorization relies on global scene statistics, such as diagnostic amplitude spectrum information. We measured performance in a Natural vs. Man-made context categorization task using a set of achromatic photographs of natural scenes equalized in average luminance, global contrast, and spectral energy. Results suggest that the visual system might use amplitude spectrum characteristics of the scenes to speed up context categorization processes. In a second experiment, we measured performance impairments with a parametric degradation of phase information applied to power spectrum averaged scenes. Results showed that performance accuracy was virtually unaffected up to 50% of phase blurring, but then rapidly fell to chance level following a sharp sigmoid curve. Response time analysis showed that subjects tended to make their fastest responses based on the presence of diagnostic man-made information; if no man-made characteristics enable to reach rapidly a decision threshold, because of a natural scene display or a high level of noise, the alternative decision for a natural response became increasingly favored. This two-phase strategy could maximize categorization performance if the diagnostic features of man-made environments tolerate higher levels of noise than natural features, as proposed recently.

Early interference of context congruence on object processing in rapid visual categorization of natural scenes.

Whereas most scientists agree that scene context can influence object recognition, the time course of such object/context interactions is still unknown. To determine the earliest interactions between object and context processing, we used a rapid go/no-go categorization task in which natural scenes were briefly flashed and subjects required to respond as fast as possible to animal targets. Targets were pasted on congruent (natural) or incongruent (urban) contexts. Experiment 1 showed that pasting a target on another congruent background induced performance impairments, whereas segregation of targets on a blank background had very little effect on behavior. Experiment 2 used animals pasted on congruent or incongruent contexts. Context incongruence induced a 10% drop of correct hits and a 16-ms increase in median reaction times, affecting even the earliest behavioral responses. Experiment 3 replicated the congruency effect with other subjects and other stimuli, thus demonstrating its robustness. Object and context must be processed in parallel with continuous interactions possibly through feed-forward co-activation of populations of visual neurons selective to diagnostic features. Facilitation would be induced by the customary co-activation of "congruent" populations of neurons whereas interference would take place when conflictual populations of neurons fire simultaneously.

Parallel processing in high-level categorization of natural images.

Models of visual processing often include an initial parallel stage that is restricted to relatively low-level features, whereas activation of higher-level object descriptions is generally assumed to require attention. Here we report that even high-level object representations can be accessed in parallel: in a rapid animal versus non-animal categorization task, both behavioral and electrophysiological data show that human subjects were as fast at responding to two simultaneously presented natural images as they were to a single one. The implication is that even complex natural images can be processed in parallel without the need for sequential focal attention.

Briefly Flashed Scenes Can Be Stored in Long-Term Memory.

The capacity of human memory is impressive. Previous reports have shown that when asked to memorize images, participants can recognize several thousands of visual objects in great details even with a single viewing of a few seconds per image. In this experiment, we tested recognition performance for natural scenes that participants saw for 20 ms only once (untrained group) or 22 times over many days (trained group) in an unrelated task. 400 images (200 previously viewed and 200 novel images) were flashed one at a time and participants were asked to lift their finger from a pad whenever they thought they had already seen the image (go/no-go paradigm). Compared to previous reports of excellent recognition performance with only single presentations of a few seconds, untrained participants were able to recognize only 64% of the 200 images they had seen few minutes before. On the other hand, trained participants, who had processed the flashed images (20 ms) several times, could correctly recognize 89% of them. EEG recordings confirmed these behavioral results. As early as 230 ms after stimulus onset, a significant event-related-potential (ERP) difference between familiar and new images was observed for the trained but not for the untrained group. These results show that briefly flashed unmasked scenes can be incidentally stored in long-term memory when repeated.

Effects of task requirements on rapid natural scene processing: from common sensory encoding to distinct decisional mechanisms.

Using manual responses, human participants are remarkably fast and accurate at deciding if a natural scene contains an animal, but recent data show that they are even faster to indicate with saccadic eye movements which of 2 scenes contains an animal. How could it be that 2 images can apparently be processed faster than a single image? To better understand the origin of this speed advantage in forced-choice categorization, the present study used a masking procedure to compare 4 tasks in which sensory, decisional, and motor aspects were systematically varied. With stimulus onset asynchronies (SOAs) above 40 ms, there were substantial differences in sensitivity between tasks, as determined by d' measurements, with an advantage for tasks using a single image. However, with SOAs below 30-40 ms, sensitivity was similar for all experiments, despite very large differences in reaction time. This suggests that the initial part of the sensory encoding relies on common and parallel processing across a large range of tasks, whether participants have to categorize the image or locate a target in 1 of 2 scenes.

Processing scene context: fast categorization and object interference.

The extent to which object identification is influenced by the background of the scene is still controversial. On the one hand, the global context of a scene might be considered as an ultimate representation, suggesting that object processing is performed almost systematically before scene context analysis. Alternatively, the gist of a scene could be extracted sufficiently early to be able to influence object categorization. It is thus essential to assess the processing time of scene context. In the present study, we used a go/no-go rapid visual categorization task in which subjects had to respond as fast as possible when they saw a "man-made environment", or a "natural environment", that was flashed for only 26 ms. "Man-made" and "natural" scenes were categorized with very high accuracy (both around 96%) and very short reaction times (median RT both around 390 ms). Compared with previous results from our group, these data demonstrate that global context categorization is remarkably fast: (1) it is as fast as object categorization [Fabre-Thorpe, M., Delorme, A., Marlot, C., & Thorpe, S. (2001). A limit to the speed of processing in ultra-rapid visual categorization of novel natural scenes. Journal of Cognitive Neuroscience, 13(2), 171-180]; (2) it is faster than contextual categorization at more detailed levels such as sea, mountain, indoor or urban contexts [Rousselet, G. A., Joubert, O. R., & Fabre-Thorpe, M. (2005). How long to get to the "gist" of real-world natural scenes? Visual Cognition, 12(6), 852-877]. Further analysis showed that the efficiency of contextual categorization was impaired by the presence of a salient object in the scene especially when the object was incongruent with the context. Processing of natural scenes might thus involve in parallel the extraction of the global gist of the scene and the concurrent object processing leading to categorization. These data also suggest early interactions between scene and object representations compatible with contextual influences on object categorization in a parallel network.

How parallel is visual processing in the ventral pathway?

Visual object perception is usually studied by presenting one object at a time at the fovea. However, the world around us is composed of multiple objects. The way our visual system deals with this complexity has remained controversial in the literature. Some models claim that the ventral pathway, a set of visual cortical areas responsible for object recognition, can process only one or very few objects at a time without ambiguity. Other models argue in favor of a massively parallel processing of objects in a scene. Recent experiments in monkeys have provided important data about this issue. The ventral pathway seems to be able to perform complex analyses on several objects simultaneously, but only during a short time period. Subsequently only one or very few objects are explicitly selected and consciously perceived. Here, we survey the implications of these new findings for our understanding of object processing.

Rapid categorization of natural scenes in monkeys: target predictability and processing speed.

Three monkeys performed a categorization task and a recognition task with briefly flashed natural images, using in alternation either a large variety of familiar target images (animal or food) or a single (totally predictable) target. The processing time was 20 ms shorter in the recognition task in which false alarms showed that monkeys relied on low-level cues (color, form, orientation, etc.). The 20-ms additional delay necessary in monkeys to perform the categorization task is compared with the 40-ms delay previously found for humans performing similar tasks. With such short additional processing time, it is argued that neither monkeys nor humans have time to develop a fully integrated object representation in the categorization task and must rely on coarse intermediate representations.

The time course of visual processing: backward masking and natural scene categorisation.

Human observers are very good at deciding whether briefly flashed novel images contain an animal and previous work has shown that the underlying visual processing can be performed in under 150 ms. Here we used a masking paradigm to determine how information accumulates over time during such high-level categorisation tasks. As the delay between test image and mask is increased, both behavioural accuracy and differential ERP amplitude rapidly increase to reach asymptotic levels around 40-60 ms. Such results imply that processing at each stage in the visual system is remarkably rapid, with information accumulating almost continuously following the onset of activation.

Interaction of top-down and bottom-up processing in the fast visual analysis of natural scenes.

The influence of task requirements on the fast visual processing of natural scenes was studied in 14 human subjects performing in alternation an "animal" categorization task and a single-photograph recognition task. Target photographs were randomly mixed with non-target images and flashed for only 20 ms. Subjects had to respond to targets within 1 s. Processing time for image-recognition was 30-40 ms shorter than for the categorization task, both for the fastest behavioral responses and for the latency at which event related potentials evoked by target and non-target stimuli started to diverge. The faster processing in image-recognition is shown to be due to the use of low-level cues, but source analysis produced evidence that, regardless of the task, the dipoles accounting for the differential activity had the same localization and orientation in the occipito-temporal cortex. We suggest that both tasks involve the same visual pathway and the same decisional brain area but because of the total predictability of the target in the image recognition task, the first wave of bottom-up feed-forward information is speeded up by top-down influences that might originate in the prefrontal cortex and preset lower levels of the visual pathway to the known target features.

Spatiotemporal analyses of the N170 for human faces, animal faces and objects in natural scenes.

We assessed the specificity to human faces of the N170 ERP component in the context of natural scenes. Subjects categorized photographs containing human faces, animal faces and various objects. Spatiotemporal topography analyses were performed on the individual ERP data. ERPs elicited by animal faces were similar to human faces ERPs but with a delayed face activity. In the N170 time window, ERPs to human and animal faces had a different topography compared with object ERPs. Such data suggest that N170 generators might process various stimuli with a coarse facial organization and show the care that must be taken in comparing scalp signal to faces and other objects as they are probably generated, at least partially, by different cortical sources.

Processing of one, two or four natural scenes in humans: the limits of parallelism.

The visual processing of objects in natural scenes is fast and efficient, as indexed by behavioral and ERP data [Nature 381 (1996) 520]. The results from a recent experiment suggested that such fast routines work in parallel across the visual field when subjects were presented with two natural scenes simultaneously [Nature Neurosci. 5 (2002) 629]. In the present experiment, the visual system was driven to its limits by presenting one, two or four scenes simultaneously. Behavior and ERP reveal a clear cost in processing an increasing number of scenes. However, a parallel-late selection model can still account for the results. This model is developed and discussed with reference to behavioral, single-unit and ERP data.