A simple rule to describe interactions between visual categories.

Humans can rapidly categorise visual objects when presented in isolation. However, in everyday life we encounter multiple objects at the same time. Far less is known about how simultaneously active object representations interact. We examined such interactions by asking participants to categorise a target object at the basic (Experiment 1) or the superordinate (Experiment 2) level while the representation of another object was still active. We found that the "prime" object strongly modulated the response to the target implying that the prime's category was rapidly and automatically accessed, influencing subsequent categorical processing. Using drift diffusion modelling, we show that a prime, whose category is different from that of the target, interferes with target processing primarily during the evidence accumulation stage. This suggests that the state of category-processing neurons is altered by an active representation and this modifies the processing of other categories. Interestingly, the strength of interference increases with the similarity between the distractor and the target category. Considering these results and previous studies, we propose a general principle that category interactions are determined by the distance from a distractor's representation to the target's task-relevant categorical boundary. We argue that this principle arises from the specific architectural organisation of categories in the brain.

Briefly Flashed Scenes Can Be Stored in Long-Term Memory.

The capacity of human memory is impressive. Previous reports have shown that when asked to memorize images, participants can recognize several thousands of visual objects in great details even with a single viewing of a few seconds per image. In this experiment, we tested recognition performance for natural scenes that participants saw for 20 ms only once (untrained group) or 22 times over many days (trained group) in an unrelated task. 400 images (200 previously viewed and 200 novel images) were flashed one at a time and participants were asked to lift their finger from a pad whenever they thought they had already seen the image (go/no-go paradigm). Compared to previous reports of excellent recognition performance with only single presentations of a few seconds, untrained participants were able to recognize only 64% of the 200 images they had seen few minutes before. On the other hand, trained participants, who had processed the flashed images (20 ms) several times, could correctly recognize 89% of them. EEG recordings confirmed these behavioral results. As early as 230 ms after stimulus onset, a significant event-related-potential (ERP) difference between familiar and new images was observed for the trained but not for the untrained group. These results show that briefly flashed unmasked scenes can be incidentally stored in long-term memory when repeated.

At 120 msec you can spot the animal but you don't yet know it's a dog.

Earlier studies suggested that the visual system processes information at the basic level (e.g., dog) faster than at the subordinate (e.g., Dalmatian) or superordinate (e.g., animals) levels. However, the advantage of the basic category over the superordinate category in object recognition has been challenged recently, and the hierarchical nature of visual categorization is now a matter of debate. To address this issue, we used a forced-choice saccadic task in which a target and a distractor image were displayed simultaneously on each trial and participants had to saccade as fast as possible toward the image containing animal targets based on different categorization levels. This protocol enables us to investigate the first 100-120 msec, a previously unexplored temporal window, of visual object categorization. The first result is a surprising stability of the saccade latency (median RT ∼ 155 msec) regardless of the animal target category and the dissimilarity of target and distractor image sets. Accuracy was high (around 80% correct) for categorization tasks that can be solved at the superordinate level but dropped to almost chance levels for basic level categorization. At the basic level, the highest accuracy (62%) was obtained when distractors were restricted to another dissimilar basic category. Computational simulations based on the saliency map model showed that the results could not be predicted by pure bottom-up saliency differences between images. Our results support a model of visual recognition in which the visual system can rapidly access relatively coarse visual representations that provide information at the superordinate level of an object, but where additional visual analysis is required to allow more detailed categorization at the basic level.

Stimulus duration and diversity do not reverse the advantage for superordinate-level representations: the animal is seen before the bird.

Basic-level categorization has long been thought to be the entry level for object representations. However, this view is now challenged. In particular, Macé et al. [M.J.-M. Macé et al. (2009) PLoS One, 4, e5927] showed that basic-level categorization (such as 'bird') requires a longer processing time than superordinate-level categorization (such as 'animal'). It has been argued that this result depends on the brief stimulus presentation times used in their study, which would degrade the visual information available. Here, we used a go/no-go paradigm to test whether the superordinate-level advantage could be observed with longer stimulus durations, and also investigated the impact of manipulating the target and distractor set heterogeneity. Our results clearly show that presentation time had no effect on categorization performance. Both target and distractor diversity influenced performance, but basic-level categories were never accessed faster or with higher accuracy than superordinate-level categories. These results argue in favor of coarse to fine visual processing to access perceptual representations.

Incongruent object/context relationships in visual scenes: where are they processed in the brain?

Rapid object visual categorization in briefly flashed natural scenes is influenced by the surrounding context. The neural correlates underlying reduced categorization performance in response to incongruent object/context associations remain unclear and were investigated in the present study using fMRI. Participants were instructed to categorize objects in briefly presented scenes (exposure duration=100ms). Half of the scenes consisted of objects pasted in an expected (congruent) context, whereas for the other half, objects were embedded in incongruent contexts. Object categorization was more accurate and faster in congruent relative to incongruent scenes. Moreover, we found that the two types of scenes elicited different patterns of cerebral activation. In particular, the processing of incongruent scenes induced increased activations in the parahippocampal cortex, as well as in the right frontal cortex. This higher activity may indicate additional neural processing of the novel (non experienced) contextual associations that were inherent to the incongruent scenes. Moreover, our results suggest that the locus of object categorization impairment due to contextual incongruence is in the right anterior parahippocampal cortex. Indeed in this region activity was correlated with the reaction time increase observed with incongruent scenes. Representations for associations between objects and their usual context of appearance might be encoded in the right anterior parahippocampal cortex.

Animal detection precedes access to scene category.

The processes underlying object recognition are fundamental for the understanding of visual perception. Humans can recognize many objects rapidly even in complex scenes, a task that still presents major challenges for computer vision systems. A common experimental demonstration of this ability is the rapid animal detection protocol, where human participants earliest responses to report the presence/absence of animals in natural scenes are observed at 250-270 ms latencies. One of the hypotheses to account for such speed is that people would not actually recognize an animal per se, but rather base their decision on global scene statistics. These global statistics (also referred to as spatial envelope or gist) have been shown to be computationally easy to process and could thus be used as a proxy for coarse object recognition. Here, using a saccadic choice task, which allows us to investigate a previously inaccessible temporal window of visual processing, we showed that animal - but not vehicle - detection clearly precedes scene categorization. This asynchrony is in addition validated by a late contextual modulation of animal detection, starting simultaneously with the availability of scene category. Interestingly, the advantage for animal over scene categorization is in opposition to the results of simulations using standard computational models. Taken together, these results challenge the idea that rapid animal detection might be based on early access of global scene statistics, and rather suggests a process based on the extraction of specific local complex features that might be hardwired in the visual system.

A need for more information uptake but not focused attention to access basic-level representations.

Complex visual scenes can be categorized at the superordinate level (e.g., animal/non-animal or vehicle/non-vehicle) without focused attention. However, rapid visual categorization at the basic level (e.g., dog/non-dog or car/non-car) requires additional processing time. Such finer categorization might, thus, require attentional resources. This hypothesis was tested in the current study with a dual-task paradigm in which subjects performed a basic-level categorization task in peripheral vision either alone (single-task condition) or concurrently with an attentionally demanding letter discrimination task (dual-task condition). Our results indicate that basic-level categorization of either biological (dog/non-dog animal) or man-made (car/non-car vehicle) stimuli requires more information uptake but can, nevertheless, be performed when attention is not fully available, presumably because it is supported by hardwired, specialized neuronal networks.

The characteristics and limits of rapid visual categorization.

Visual categorization appears both effortless and virtually instantaneous. The study by Thorpe et al. (1996) was the first to estimate the processing time necessary to perform fast visual categorization of animals in briefly flashed (20 ms) natural photographs. They observed a large differential EEG activity between target and distracter correct trials that developed from 150 ms after stimulus onset, a value that was later shown to be even shorter in monkeys! With such strong processing time constraints, it was difficult to escape the conclusion that rapid visual categorization was relying on massively parallel, essentially feed-forward processing of visual information. Since 1996, we have conducted a large number of studies to determine the characteristics and limits of fast visual categorization. The present chapter will review some of the main results obtained. I will argue that rapid object categorizations in natural scenes can be done without focused attention and are most likely based on coarse and unconscious visual representations activated with the first available (magnocellular) visual information. Fast visual processing proved efficient for the categorization of large superordinate object or scene categories, but shows its limits when more detailed basic representations are required. The representations for basic objects (dogs, cars) or scenes (mountain or sea landscapes) need additional processing time to be activated. This finding is at odds with the widely accepted idea that such basic representations are at the entry level of the system. Interestingly, focused attention is still not required to perform these time consuming basic categorizations. Finally we will show that object and context processing can interact very early in an ascending wave of visual information processing. We will discuss how such data could result from our experience with a highly structured and predictable surrounding world that shaped neuronal visual selectivity.

Humans and monkeys share visual representations.

Conceptual abilities in animals have been shown at several levels of abstraction, but it is unclear whether the analogy with humans results from convergent evolution or from shared brain mechanisms inherited from a common origin. Macaque monkeys can access "non-similarity-based concepts," such as when sorting pictures containing a superordinate target category (animal, tree, etc.) among other scenes. However, such performances could result from low-level visual processing based on learned regularities of the photographs, such as for scene categorization by artificial systems. By using pictures of man-made objects or animals embedded in man-made or natural contexts, the present study clearly establishes that macaque monkeys based their categorical decision on the presence of the animal targets regardless of the scene backgrounds. However, as is found with humans, monkeys performed better with categorically congruent object/context associations, especially when small object sizes favored background information. The accuracy improvements and the response-speed gains attributable to superordinate category congruency in monkeys were strikingly similar to those of human subjects tested with the same task and stimuli. These results suggest analogous processing of visual information during the activation of abstract representations in both humans and monkeys; they imply a large overlap between superordinate visual representations in humans and macaques as well as the implicit use of experienced associations between object and context.

Rapid categorization of faces and objects in a patient with impaired object recognition.

We tested rapid-categorization in a patient who was impaired in face and object recognition. Photographs of natural scenes were displayed for 100 ms. Participants had to press a key when they saw an animal among various objects as distractors or human faces among animal faces as distractors. Though the patient was impaired at figure/ground segregation, recognized very few objects and faces, she categorized animals and faces with a performance ranging between 70 and 86% correct. Displaying pictures in isolation did not improve performance. The results suggest that rapid categorization can be accomplished on the basis of coarse information without overt recognition.

Spotting animals in natural scenes: efficiency of humans and monkeys at very low contrasts.

The ability of monkeys to categorize objects in visual stimuli such as natural scenes might rely on sets of low-level visual cues without any underlying conceptual abilities. Using a go/no-go rapid animal/non-animal categorization task with briefly flashed achromatic natural scenes, we show that both human and monkey performance is very robust to large variations of stimulus luminance and contrast. When mean luminance was increased or decreased by 25-50%, accuracy and speed impairments were small. The largest impairment was found at the highest luminance value with monkeys being mainly impaired in accuracy (drop of 6% correct vs. <1.5% in humans), whereas humans were mainly impaired in reaction time (20 ms increase in median reaction time vs. 4 ms in monkeys). Contrast reductions induced a large deterioration of image definition, but performance was again remarkably robust. Subjects scored well above chance level, even when the contrast was only 12% of the original photographs ( approximately 81% correct in monkeys; approximately 79% correct in humans). Accuracy decreased with contrast reduction but only reached chance level -in both species- for the most extreme condition, when only 3% of the original contrast remained. A progressive reaction time increase was observed that reached 72 ms in monkeys and 66 ms in humans. These results demonstrate the remarkable robustness of the primate visual system in processing objects in natural scenes with large random variations in luminance and contrast. They illustrate the similarity with which performance is impaired in monkeys and humans with such stimulus manipulations. They finally show that in an animal categorization task, the performance of both monkeys and humans is largely independent of cues relying on global luminance or the fine definition of stimuli.

Key visual features for rapid categorization of animals in natural scenes.

In speeded categorization tasks, decisions could be based on diagnostic target features or they may need the activation of complete representations of the object. Depending on task requirements, the priming of feature detectors through top-down expectation might lower the threshold of selective units or speed up the rate of information accumulation. In the present paper, 40 subjects performed a rapid go/no-go animal/non-animal categorization task with 400 briefly flashed natural scenes to study how performance depends on physical scene characteristics, target configuration, and the presence or absence of diagnostic animal features. Performance was evaluated both in terms of accuracy and speed and d' curves were plotted as a function of reaction time (RT). Such d' curves give an estimation of the processing dynamics for studied features and characteristics over the entire subject population. Global image characteristics such as color and brightness do not critically influence categorization speed, although they slightly influence accuracy. Global critical factors include the presence of a canonical animal posture and animal/background size ratio suggesting the role of coarse global form. Performance was best for both accuracy and speed, when the animal was in a typical posture and when it occupied about 20-30% of the image. The presence of diagnostic animal features was another critical factor. Performance was significantly impaired both in accuracy (drop 3.3-7.5%) and speed (median RT increase 7-16 ms) when diagnostic animal parts (eyes, mouth, and limbs) were missing. Such animal features were shown to influence performance very early when only 15-25% of the response had been produced. In agreement with other experimental and modeling studies, our results support fast diagnostic recognition of animals based on key intermediate features and priming based on the subject's expertise.

The temporal interplay between conscious and unconscious perceptual streams.

An optimal correspondence of temporal information between the physical world and our perceptual world is important for survival. In the current study, we demonstrate a novel temporal illusion in which the cause of a perceptual event is perceived after the event itself. We used a paradigm referred to as motion-induced blindness (MIB), in which a static visual target presented on a constantly rotating background disappears and reappears from awareness periodically, with the dynamic characteristics of bistable perception. A sudden stimulus onset (e.g., a flash) presented during a period of perceptual suppression (i.e., during MIB) is known to trigger the almost instantaneous reappearance of the suppressed target. Surprisingly, however, we report here that although the sudden flash is the cause of the static target's reappearance (the corresponding effect), it is systematically perceived as occurring after this reappearance. Further investigation revealed that this illusory temporal reversal is caused by an approximately 100 ms advantage for the unconscious representation of the perceptually suppressed target to access consciousness, as compared to the newly presented flash. This new temporal illusion therefore reveals the normally hidden delays in bringing new visual events to awareness.

The time-course of visual categorizations: you spot the animal faster than the bird.

BACKGROUND: Since the pioneering study by Rosch and colleagues in the 70s, it is commonly agreed that basic level perceptual categories (dog, chair...) are accessed faster than superordinate ones (animal, furniture...). Nevertheless, the speed at which objects presented in natural images can be processed in a rapid go/no-go visual superordinate categorization task has challenged this "basic level advantage". PRINCIPAL FINDINGS: Using the same task, we compared human processing speed when categorizing natural scenes as containing either an animal (superordinate level), or a specific animal (bird or dog, basic level). Human subjects require an additional 40-65 ms to decide whether an animal is a bird or a dog and most errors are induced by non-target animals. Indeed, processing time is tightly linked with the type of non-targets objects. Without any exemplar of the same superordinate category to ignore, the basic level category is accessed as fast as the superordinate category, whereas the presence of animal non-targets induces both an increase in reaction time and a decrease in accuracy. CONCLUSIONS AND SIGNIFICANCE: These results support the parallel distributed processing theory (PDP) and might reconciliate controversial studies recently published. The visual system can quickly access a coarse/abstract visual representation that allows fast decision for superordinate categorization of objects but additional time-consuming visual analysis would be necessary for a decision at the basic level based on more detailed representations.

Rapid visual categorization of natural scene contexts with equalized amplitude spectrum and increasing phase noise.

This study aimed to determine the extent to which rapid visual context categorization relies on global scene statistics, such as diagnostic amplitude spectrum information. We measured performance in a Natural vs. Man-made context categorization task using a set of achromatic photographs of natural scenes equalized in average luminance, global contrast, and spectral energy. Results suggest that the visual system might use amplitude spectrum characteristics of the scenes to speed up context categorization processes. In a second experiment, we measured performance impairments with a parametric degradation of phase information applied to power spectrum averaged scenes. Results showed that performance accuracy was virtually unaffected up to 50% of phase blurring, but then rapidly fell to chance level following a sharp sigmoid curve. Response time analysis showed that subjects tended to make their fastest responses based on the presence of diagnostic man-made information; if no man-made characteristics enable to reach rapidly a decision threshold, because of a natural scene display or a high level of noise, the alternative decision for a natural response became increasingly favored. This two-phase strategy could maximize categorization performance if the diagnostic features of man-made environments tolerate higher levels of noise than natural features, as proposed recently.

Early interference of context congruence on object processing in rapid visual categorization of natural scenes.

Whereas most scientists agree that scene context can influence object recognition, the time course of such object/context interactions is still unknown. To determine the earliest interactions between object and context processing, we used a rapid go/no-go categorization task in which natural scenes were briefly flashed and subjects required to respond as fast as possible to animal targets. Targets were pasted on congruent (natural) or incongruent (urban) contexts. Experiment 1 showed that pasting a target on another congruent background induced performance impairments, whereas segregation of targets on a blank background had very little effect on behavior. Experiment 2 used animals pasted on congruent or incongruent contexts. Context incongruence induced a 10% drop of correct hits and a 16-ms increase in median reaction times, affecting even the earliest behavioral responses. Experiment 3 replicated the congruency effect with other subjects and other stimuli, thus demonstrating its robustness. Object and context must be processed in parallel with continuous interactions possibly through feed-forward co-activation of populations of visual neurons selective to diagnostic features. Facilitation would be induced by the customary co-activation of "congruent" populations of neurons whereas interference would take place when conflictual populations of neurons fire simultaneously.

Processing scene context: fast categorization and object interference.

The extent to which object identification is influenced by the background of the scene is still controversial. On the one hand, the global context of a scene might be considered as an ultimate representation, suggesting that object processing is performed almost systematically before scene context analysis. Alternatively, the gist of a scene could be extracted sufficiently early to be able to influence object categorization. It is thus essential to assess the processing time of scene context. In the present study, we used a go/no-go rapid visual categorization task in which subjects had to respond as fast as possible when they saw a "man-made environment", or a "natural environment", that was flashed for only 26 ms. "Man-made" and "natural" scenes were categorized with very high accuracy (both around 96%) and very short reaction times (median RT both around 390 ms). Compared with previous results from our group, these data demonstrate that global context categorization is remarkably fast: (1) it is as fast as object categorization [Fabre-Thorpe, M., Delorme, A., Marlot, C., & Thorpe, S. (2001). A limit to the speed of processing in ultra-rapid visual categorization of novel natural scenes. Journal of Cognitive Neuroscience, 13(2), 171-180]; (2) it is faster than contextual categorization at more detailed levels such as sea, mountain, indoor or urban contexts [Rousselet, G. A., Joubert, O. R., & Fabre-Thorpe, M. (2005). How long to get to the "gist" of real-world natural scenes? Visual Cognition, 12(6), 852-877]. Further analysis showed that the efficiency of contextual categorization was impaired by the presence of a salient object in the scene especially when the object was incongruent with the context. Processing of natural scenes might thus involve in parallel the extraction of the global gist of the scene and the concurrent object processing leading to categorization. These data also suggest early interactions between scene and object representations compatible with contextual influences on object categorization in a parallel network.

Effects of task requirements on rapid natural scene processing: from common sensory encoding to distinct decisional mechanisms.

Using manual responses, human participants are remarkably fast and accurate at deciding if a natural scene contains an animal, but recent data show that they are even faster to indicate with saccadic eye movements which of 2 scenes contains an animal. How could it be that 2 images can apparently be processed faster than a single image? To better understand the origin of this speed advantage in forced-choice categorization, the present study used a masking procedure to compare 4 tasks in which sensory, decisional, and motor aspects were systematically varied. With stimulus onset asynchronies (SOAs) above 40 ms, there were substantial differences in sensitivity between tasks, as determined by d' measurements, with an advantage for tasks using a single image. However, with SOAs below 30-40 ms, sensitivity was similar for all experiments, despite very large differences in reaction time. This suggests that the initial part of the sensory encoding relies on common and parallel processing across a large range of tasks, whether participants have to categorize the image or locate a target in 1 of 2 scenes.

Limits of event-related potential differences in tracking object processing speed.

We report results from two experiments in which subjects had to categorize briefly presented upright or inverted natural scenes. In the first experiment, subjects decided whether images contained animals or human faces presented at different scales. Behavioral results showed virtually identical processing speed between the two categories and very limited effects of inversion. One type of event-related potential (ERP) comparison, potentially capturing low-level physical differences, showed large effects with onsets at about 150 msec in the animal task. However, in the human face task, those differences started as early as 100 msec. In the second experiment, subjects responded to close-up views of animal faces or human faces in an attempt to limit physical differences between image sets. This manipulation almost completely eliminated small differences before 100 msec in both tasks. But again, despite very similar behavioral performances and short reaction times in both tasks, human faces were associated with earlier ERP differences compared with animal faces. Finally, in both experiments, as an alternative way to determine processing speed, we compared the ERP with the same images when seen as targets and nontargets in different tasks. Surprisingly, all task-dependent ERP differences had relatively long latencies. We conclude that task-dependent ERP differences fail to capture object processing speed, at least for some categories like faces. We discuss models of object processing that might explain our results, as well as alternative approaches.

Rapid categorization of foveal and extrafoveal natural images: associated ERPs and effects of lateralization.

Humans are fast and accurate at performing an animal categorization task with natural photographs briefly flashed centrally. Here, this central categorization task is compared to a three position task in which photographs could appear randomly either centrally, or at 3.6 degrees eccentricity (right or left) of the fixation point. A mild behavioral impairment was found with peripheral stimuli with no evidence in support of hemispheric superiority; but enlarging the window of spatial attention to three possible stimuli locations had no behavioral cost on the processing of central images. Performance in the central categorization task has been associated with a large difference between the potentials evoked to target and non-target correct trials, starting about 150 ms after stimulus onset on frontal sites. Present results show that this activity originates within extrastriate visual cortices and probably reflects perceptual stimuli differences processed within areas involved in object recognition. Latencies, slopes, and peak amplitudes of this differential activity were invariant to stimulus position and attentional load. Stimulus location uncertainty and lateralization did not affect speed of visual processing.