Hybrid visual and memory search for scenes and objects with variable viewpoints.

In hybrid search, observers search visual arrays for any of several target types held in memory. The key finding in hybrid search is that response times (RTs) increase as a linear function of the number of items in a display (visual set size), but RTs increase linearly with the log of the memory set size. Previous experiments have shown this result for specific targets (find exactly this picture of a boot on a blank background) and for broad categorical targets (find any animal). Arguably, these are rather unnatural situations. In the real world, objects are parts of scenes and are seen from multiple viewpoints. The present experiments generalize the hybrid search findings to scenes (Experiment 1) and multiple viewpoints (Experiment 2). The results replicated the basic pattern of hybrid search results: RTs increased logarithmically with the number of scene photos/categories held in memory. Experiment 3 controls the experiment for which viewpoints were seen in an initial learning phase. The results replicate the findings of Experiment 2. Experiment 4 compares hybrid search for specific viewpoints, variable viewpoints, and categorical targets. Search difficulty increases from specific viewpoints to variable viewpoints and then to categorical targets. The results of the four experiments show the generality of logarithmic search through memory in hybrid search.

Hiding the Rabbit: Using a genetic algorithm to investigate shape guidance in visual search.

During visual search, attention is guided by specific features, including shape. Our understanding of shape guidance is limited to specific attributes (closures and line terminations) that do not fully explain the richness of preattentive shape processing. We used a novel genetic algorithm method to explore shape space and to stimulate hypotheses about shape guidance. Initially, observers searched for targets among 12 random distractors defined, in radial frequency space, by the amplitude and phase of 10 radial frequencies. Reaction time (RT) was the measure of "fitness." To evolve toward an easier search task, distractors with faster RTs survived to the next generation, "mated," and produced offspring (new distractors for the next generation of search). To evolve a harder search, surviving distractors were those yielding longer RTs. Within eight generations of evolution, the method succeeds in producing visual searches either harder or easier than the starting search. In radial frequency space, easy distractors evolve amplitude × frequency spectra that are dissimilar to the target, whereas hard distractors evolve spectra that are more similar to the target. This method also works with naturally shaped targets (e.g., rabbit silhouettes). Interestingly, the most inefficient distractors featured a combination of a body and ear distractors that did not resemble the rabbit (visually or in spectrum). Adding extra ears to these distractors did not impact the search spectrally and instead made it easier to confirm a rabbit, once it was found. In general, these experiments show that shapes that are clearly distinct when attended are similar to each other preattentively.

Order, please! Explicit sequence learning in hybrid search in younger and older age.

Sequence learning effects in simple perceptual and motor tasks are largely unaffected by normal aging. However, less is known about sequence learning in more complex cognitive tasks that involve attention and memory processes and how this changes with age. In this study, we examined whether incidental and intentional sequence learning would facilitate hybrid visual and memory search in younger and older adults. Observers performed a hybrid search task, in which they memorized four or 16 target objects and searched for any of those target objects in displays with four or 16 objects. The memorized targets appeared either in a repeating sequential order or in random order. In the first experiment, observers were not told about the sequence before the experiment. Only a subset of younger adults and none of the older adults incidentally learned the sequence. The "learners" acquired explicit knowledge about the sequence and searched faster in the sequence compared to random condition. In the second experiment, observers were told about the sequence before the search task. Both younger and older adults searched faster in sequence blocks than random blocks. Older adults, however, showed this sequence-learning effect only in blocks with smaller target sets. Our findings indicate that explicit sequence knowledge can facilitate hybrid search, as it allows observers to predict the next target and restrict their visual and memory search. In older age, the sequence-learning effect is constrained by load, presumably due to age-related decline in executive functions.

Relationships between expertise and distinctiveness: Abnormal medical images lead to enhanced memory performance only in experts.

Memories are encoded in a manner that depends on our knowledge and expectations ("schemas"). Consistent with this, expertise tends to improve memory: Experts have elaborated schemas in their domains of expertise, allowing them to efficiently represent information in this domain (e.g., chess experts have enhanced memory for realistic chess layouts). On the other hand, in most situations, people tend to remember abnormal or surprising items best-those that are also rare or out-of-the-ordinary occurrences (e.g., surprising-but not random-chess board configurations). This occurs, in part, because such images are distinctive relative to other images. In the current work, we ask how these factors interact in a particularly interesting case-the domain of radiology, where experts actively search for abnormalities. Abnormality in mammograms is typically focal but can be perceived in the global "gist" of the image. We ask whether, relative to novices, expert radiologists show improved memory for mammograms. We also test for any additional advantage for abnormal mammograms that can be thought of as unexpected or rare stimuli in screening. We find that experts have enhanced memory for focally abnormal images relative to normal images. However, radiologists showed no memory benefit for images of the breast that were not focally abnormal, but were only abnormal in their gist. Our results speak to the role of schemas and abnormality in expertise; the necessity for spatially localized abnormalities versus abnormalities in the gist in enhancing memory; and the nature of memory and decision-making in radiologists.

Implicitly and explicitly encoded features can guide attention in free viewing.

It is well known that priming, probably by the contents of working memory, can influence subsequent visual task performance. How ubiquitous is this effect? Can incidental exposure to visual stimuli influence the deployment of attention when there is no explicit visual task? Results of two experiments show that a preceding stimulus can influence free-viewing eye movements. A simple change detection task was used as the cover task. The initial memory display was the priming display, while subsequent filler display constituted the free-viewing display of our interest. In Experiment 1, subjects were asked to memorize the number of items in the priming display. Subjects were not explicitly instructed to attend to features, but these might still be implicitly encoded. In Experiment 2, a more complex change detection task required subjects to memorize the number, color, and shape of priming items. Here, prime features were attended and, presumably, explicitly encoded. We were interested to know whether incidentally or explicitly encoded features of prime items would influence attention distribution in the filler display. In both experiments, items sharing color and shape with the prime were attended more often than predicted by chance. Items sharing neither color nor shape were attended less often. Items sharing either color or shape (not both) could also attract attention showing that the priming need not be based on a bound representation of the primed item. Effects were stronger in Experiment 2. No intention or top-down control appears to be needed to produce this priming.

Categorical grouping is not required for guided conjunction search.

Knowledge of target features can guide attention in many conjunction searches in a top-down manner. For example, in search of a red vertical line among blue vertical and red horizontal lines, observers can guide attention toward all red items and all vertical items. In typical conjunction searches, distractors often form perceptually vivid, categorical groups of identical objects. This could favor the efficient search via guidance of attention to these "segmentable" groups. Can attention be guided if the distractors are not neatly segmentable (e.g., if colors vary continuously from red through purple to blue)? We tested search for conjunctions of color × orientation (Experiments 1, 3, 4, 5) or length × orientation (Experiment 2). In segmentable conditions, distractors could form two clear groups (e.g., blue steep and red flat). In non-segmentable conditions, distractors varied smoothly from red to blue and/or steep to flat; thus, discouraging grouping and increasing overall heterogeneity. We found that the efficiency of conjunction search was reasonably high and unaffected by segmentability. The same lack of segmentability had a detrimental effect on feature search (Experiment 4) and on conjunction search, if target information was limited to one feature (e.g., find the odd item in the red set, "subset search," Experiment 3). Guidance in conjunction search may not require grouping and segmentation cues that are very important in other tasks like texture discrimination. Our results support an idea of simultaneous, parallel top-down guidance by multiple features and argue against models suggesting sequential guidance by each feature in turn.

Looking ahead: When do you find the next item in foraging visual search?

Many real-world visual tasks involve searching for multiple instances of a target (e.g., picking ripe berries). What strategies do observers use when collecting items in this type of search? Do they wait to finish collecting the current item before starting to look for the next target, or do they search ahead for future targets? We utilized behavioral and eye-tracking measures to distinguish between these two possibilities in foraging search. Experiment 1 used a color wheel technique in which observers searched for T shapes among L shapes while all items independently cycled through a set of colors. Trials were abruptly terminated, and observers reported both the color and location of the next target that they intended to click. Using observers' color reports to infer target-finding times, we demonstrate that observers found the next item before the time of the click on the current target. We validated these results in Experiment 2 by recording fixation locations around the time of each click. Experiment 3 utilized a different procedure, in which all items were intermittently occluded during the trial. We then calculated a distribution of when targets were visible around the time of each click, allowing us to infer when they were most likely found. In a fourth and final experiment, observers indicated the locations of multiple future targets after the search was abruptly terminated. Together, our results provide converging evidence to demonstrate that observers can find the next target before collecting the current target and can typically forage one to two items ahead.

Choosing or rejecting a food item, does framing matter? And what has sugar to do with it!

The color code of "Traffic Light Labels" (TLL) on food items indicates the amount (e.g., green = low) of fat, saturates, sugar and salt it contains. Consider two ways to select among food items (e.g., two cereal bars) based on their TLLs. You might choose between the two items or you might reject one of the two. Furthermore, differences between choose and reject might be driven more strongly by one factor (e.g., sugar) than by others. In Study 1 our participants made choose or reject decisions between food items with an all-orange TLL (all moderate) and a 2 red/2 green TLL (2 negative/2 positive). Both items had equal energy/caloric content. We found that, independent of the condition (Choose/Reject), participants went home more often with the 2 red/2 green item if sugar was green. This effect was stronger in the Choose than in the Reject condition. In Study 2, we additionally manipulated the energy content (low, high) of the items. In the case where both food items had a low energy content, similar results as in Study 1 were observed. If either or both items had high energy content, the choose/reject interaction with sugar disappeared. Only differences in energy content played a role in the reject condition. Overall, our results can be better explained by an "accentuation hypothesis" than by a "compatibility hypothesis". These findings could be used by choice architects to fight the current obesity crisis.

A half-second glimpse often lets radiologists identify breast cancer cases even when viewing the mammogram of the opposite breast.

Humans are very adept at extracting the "gist" of a scene in a fraction of a second. We have found that radiologists can discriminate normal from abnormal mammograms at above-chance levels after a half-second viewing (d' ∼ 1) but are at chance in localizing the abnormality. This pattern of results suggests that they are detecting a global signal of abnormality. What are the stimulus properties that might support this ability? We investigated the nature of the gist signal in four experiments by asking radiologists to make detection and localization responses about briefly presented mammograms in which the spatial frequency, symmetry, and/or size of the images was manipulated. We show that the signal is stronger in the higher spatial frequencies. Performance does not depend on detection of breaks in the normal symmetry of left and right breasts. Moreover, above-chance classification is possible using images from the normal breast of a patient with overt signs of cancer only in the other breast. Some signal is present in the portions of the parenchyma (breast tissue) that do not contain a lesion or that are in the contralateral breast. This signal does not appear to be a simple assessment of breast density but rather the detection of the abnormal gist may be based on a widely distributed image statistic, learned by experts. The finding that a global signal, related to disease, can be detected in parenchyma that does not contain a lesion has implications for improving breast cancer detection.

Lost in the supermarket: Quantifying the cost of partitioning memory sets in hybrid search.

The items on a memorized grocery list are not relevant in every aisle; for example, it is useless to search for the cabbage in the cereal aisle. It might be beneficial if one could mentally partition the list so only the relevant subset was active, so that vegetables would be activated in the produce section. In four experiments, we explored observers' abilities to partition memory searches. For example, if observers held 16 items in memory, but only eight of the items were relevant, would response times resemble a search through eight or 16 items? In Experiments 1a and 1b, observers were not faster for the partition set; however, they suffered relatively small deficits when "lures" (items from the irrelevant subset) were presented, indicating that they were aware of the partition. In Experiment 2 the partitions were based on semantic distinctions, and again, observers were unable to restrict search to the relevant items. In Experiments 3a and 3b, observers attempted to remove items from the list one trial at a time but did not speed up over the course of a block, indicating that they also could not limit their memory searches. Finally, Experiments 4a, 4b, 4c, and 4d showed that observers were able to limit their memory searches when a subset was relevant for a run of trials. Overall, observers appear to be unable or unwilling to partition memory sets from trial to trial, yet they are capable of restricting search to a memory subset that remains relevant for several trials. This pattern is consistent with a cost to switching between currently relevant memory items.

One visual search, many memory searches: An eye-tracking investigation of hybrid search.

Suppose you go to the supermarket with a shopping list of 10 items held in memory. Your shopping expedition can be seen as a combination of visual search and memory search. This is known as "hybrid search." There is a growing interest in understanding how hybrid search tasks are accomplished. We used eye tracking to examine how manipulating the number of possible targets (the memory set size [MSS]) changes how observers (Os) search. We found that dwell time on each distractor increased with MSS, suggesting a memory search was being executed each time a new distractor was fixated. Meanwhile, although the rate of refixation increased with MSS, it was not nearly enough to suggest a strategy that involves repeatedly searching visual space for subgroups of the target set. These data provide a clear demonstration that hybrid search tasks are carried out via a "one visual search, many memory searches" heuristic in which Os examine items in the visual array once with a very low rate of refixations. For each item selected, Os activate a memory search that produces logarithmic response time increases with increased MSS. Furthermore, the percentage of distractors fixated was strongly modulated by the MSS: More items in the MSS led to a higher percentage of fixated distractors. Searching for more potential targets appears to significantly alter how Os approach the task, ultimately resulting in more eye movements and longer response times.

Even if I showed you where you looked, remembering where you just looked is hard.

People know surprisingly little about their own visual behavior, which can be problematic when learning or executing complex visual tasks such as search of medical images. We investigated whether providing observers with online information about their eye position during search would help them recall their own fixations immediately afterwards. Seventeen observers searched for various objects in "Where's Waldo" images for 3 s. On two-thirds of trials, observers made target present/absent responses. On the other third (critical trials), they were asked to click twelve locations in the scene where they thought they had just fixated. On half of the trials, a gaze-contingent window showed observers their current eye position as a 7.5° diameter "spotlight." The spotlight "illuminated" everything fixated, while the rest of the display was still visible but dimmer. Performance was quantified as the overlap of circles centered on the actual fixations and centered on the reported fixations. Replicating prior work, this overlap was quite low (26%), far from ceiling (66%) and quite close to chance performance (21%). Performance was only slightly better in the spotlight condition (28%, p = 0.03). Giving observers information about their fixation locations by dimming the periphery improved memory for those fixations modestly, at best.

"I am not dead yet!" - The Item responds to Hulleman & Olivers.

The item can only be dispensed within artificial tasks that, although useful in the lab, do not reflect the real world. There, the attended item is the goal of search. Hulleman & Olivers' (H&O's) model can ignore the item only by reducing search to the question of whether a patch of 0s (distractors) contains a 1 (target).

CB Database: A change blindness database for objects in natural indoor scenes.

Change blindness has been a topic of interest in cognitive sciences for decades. Change detection experiments are frequently used for studying various research topics such as attention and perception. However, creating change detection stimuli is tedious and there is no open repository of such stimuli using natural scenes. We introduce the Change Blindness (CB) Database with object changes in 130 colored images of natural indoor scenes. The size and eccentricity are provided for all the changes as well as reaction time data from a baseline experiment. In addition, we have two specialized satellite databases that are subsets of the 130 images. In one set, changes are seen in rooms or in mirrors in those rooms (Mirror Change Database). In the other, changes occur in a room or out a window (Window Change Database). Both the sets have controlled background, change size, and eccentricity. The CB Database is intended to provide researchers with a stimulus set of natural scenes with defined stimulus parameters that can be used for a wide range of experiments. The CB Database can be found at http://search.bwh.harvard.edu/new/CBDatabase.html .

Winter is coming: How humans forage in a temporally structured environment.

Much is known about visual search for single targets, but relatively little about how participants "forage" for multiple targets. One important question is how long participants will search before moving to a new display. Evidence suggests that participants should leave when intake drops below the average rate ("optimal foraging," Charnov, 1976). However, the real world has temporal structure (e.g., seasons) that could influence behavior. Does it matter if winter is coming and the next display will be worse than the last? We gave participants a series of search displays and asked them to collect targets as fast as possible. Target density was structured-rising and falling systematically across trials. We measured the duration for which participants foraged in each display (trials were terminated by participants). Foraging behavior was affected by temporal structure-counter to a simple optimal foraging account, observers searched displays longer when quality was falling compared to rising (Experiments 1 and 2). Additionally, we found that temporal structure altered explicit predictions about display quality (Experiment 2). These results demonstrate that foraging theories need to consider richer models of observers' representations of the world.

Memory search for the first target modulates the magnitude of the attentional blink.

The resolution of temporal attention is limited in a manner that makes it difficult to identify two targets in short succession. This limitation produces the phenomenon known as the attentional blink (AB), in which processing of a first target (T1) impairs identification of a second target (T2). In the AB literature, there is broad agreement that increasing the time it takes to process T1 leads to a larger AB. One might, therefore, predict that increasing the number of possible T1 identities, or target set, from 1 to 16 would lead to a larger AB. We were surprised to find that this manipulation of T1 difficulty had no influence on AB magnitude. In subsequent experiments, we found that AB magnitude interacts with T1 processing time only under certain circumstances. Specifically, when the T1 task was either well masked or had to be completed online, we found a reliable interaction between AB magnitude and the target set size. When neither of these conditions was fulfilled, there was no interaction between target set size and the AB. Previous research found that when the target set changes from trial to trial, trials with more possible targets elicited a larger AB. In the present study, the target set is held constant, reducing the demands on working memory. Nevertheless, AB magnitude still interacts with target set size, as long as the T1 task cannot be processed offline. Thus, the act of searching memory delays subsequent processing, even when the role of working memory has been minimized.

Seek and you shall remember: scene semantics interact with visual search to build better memories.

Memorizing critical objects and their locations is an essential part of everyday life. In the present study, incidental encoding of objects in naturalistic scenes during search was compared to explicit memorization of those scenes. To investigate if prior knowledge of scene structure influences these two types of encoding differently, we used meaningless arrays of objects as well as objects in real-world, semantically meaningful images. Surprisingly, when participants were asked to recall scenes, their memory performance was markedly better for searched objects than for objects they had explicitly tried to memorize, even though participants in the search condition were not explicitly asked to memorize objects. This finding held true even when objects were observed for an equal amount of time in both conditions. Critically, the recall benefit for searched over memorized objects in scenes was eliminated when objects were presented on uniform, non-scene backgrounds rather than in a full scene context. Thus, scene semantics not only help us search for objects in naturalistic scenes, but appear to produce a representation that supports our memory for those objects beyond intentional memorization.

Research on re-searching: interrupted foraging is not disrupted foraging.

In classic visual search, observers typically search for the presence of a target in a scene or display. In foraging tasks, there may be multiple targets in the same display (or "patch"). Observers typically search for and collect these target items in one patch until they decide to leave that patch and move to the next one. This is a highly rule-governed behavior. The current study investigated whether these rules are disrupted when the foraging is interrupted in various manners. In Experiment 1, the foraging was briefly interrupted and then resumed in the same patch. In Experiments 2 and 3, the foraging in each patch either ended voluntarily or compulsorily after a fixed amount of time. In these cases, foraging resumed in a patch only after all patches were visited. Overall, the rules of foraging remained largely intact, though Experiment 2 shows that foraging rules can be overridden by the demand characteristics of the task. The results show that participants tended to perform approximately consistently despite interruptions. The results suggest that foraging behavior in a relatively simple foraging environment is resilient and not easily disrupted by interruption.

Errors in visual search: Are they stochastic or deterministic?

In any visual search task in the lab or in the world, observers will make errors. Those errors can be categorized as "deterministic": If you miss this target in this display once, you will definitely miss it again. Alternatively, errors can be "stochastic", occurring randomly with some probability from trial to trial. Researchers and practitioners have sought to reduce errors in visual search, but different types of errors might require different techniques for mitigation. To empirically categorize errors in a simple search task, our observers searched for the letter "T" among "L" distractors, with each display presented twice. When the letters were clearly visible (white letters on a gray background), the errors were almost completely stochastic (Exp 1). An error made on the first appearance of a display did not predict that an error would be made on the second appearance. When the visibility of the letters was manipulated (letters of different gray levels on a noisy background), the errors became a mix of stochastic and deterministic. Unsurprisingly, lower contrast targets produced more deterministic errors. (Exp 2). Using the stimuli of Exp 2, we tested whether errors could be reduced using cues that guided attention around the display but knew nothing about the content of that display (Exp3a, b). This had no effect, but cueing all item locations did succeed in reducing deterministic errors (Exp3c).