A half-second glimpse often lets radiologists identify breast cancer cases even when viewing the mammogram of the opposite breast.

Humans are very adept at extracting the "gist" of a scene in a fraction of a second. We have found that radiologists can discriminate normal from abnormal mammograms at above-chance levels after a half-second viewing (d' ∼ 1) but are at chance in localizing the abnormality. This pattern of results suggests that they are detecting a global signal of abnormality. What are the stimulus properties that might support this ability? We investigated the nature of the gist signal in four experiments by asking radiologists to make detection and localization responses about briefly presented mammograms in which the spatial frequency, symmetry, and/or size of the images was manipulated. We show that the signal is stronger in the higher spatial frequencies. Performance does not depend on detection of breaks in the normal symmetry of left and right breasts. Moreover, above-chance classification is possible using images from the normal breast of a patient with overt signs of cancer only in the other breast. Some signal is present in the portions of the parenchyma (breast tissue) that do not contain a lesion or that are in the contralateral breast. This signal does not appear to be a simple assessment of breast density but rather the detection of the abnormal gist may be based on a widely distributed image statistic, learned by experts. The finding that a global signal, related to disease, can be detected in parenchyma that does not contain a lesion has implications for improving breast cancer detection.

Early signs of cancer present in the fine detail of mammograms.

The gist of abnormality can be rapidly extracted by medical experts from global information in medical images, such as mammograms, to identify abnormal mammograms with above-chance accuracy-even before any abnormalities are localizable. The current study evaluated the effect of different high-pass filters on expert radiologists' performance in detecting the gist of abnormality in mammograms, especially those acquired prior to any visibly actionable lesions. Thirty-four expert radiologists viewed unaltered and high-pass filtered versions of normal and abnormal mammograms. Abnormal mammograms consisted of obvious abnormalities, subtle abnormalities, and currently normal mammograms from women who would go to develop cancer in 2-3 years. Four levels of high-pass filtering were tested (0.5, 1, 1.5, and 2 cycles per degree (cpd) after brightening and contrast normalizing to the unfiltered mammograms. Overall performance for 0.5 and 1.5 did not change compared to unfiltered but was reduced for 1 and 2 cpd. Critically, filtering that eliminated frequencies below 0.5 and 1.5 cpd significantly boosted performance on mammograms acquired years prior appearance of localizable abnormalities. Filtering at 0.5 did not change the radiologist's decision criteria compared to unfiltered mammograms whereas other filters resulted in more conservative ratings. The findings bring us closer to identifying the characteristics of the gist of the abnormal that affords radiologists detection of the earliest signs of cancer. A 0.5 cpd high-pass filter significantly boosts subtle, global signals of future cancerous abnormalities, potentially providing an image enhancement strategy for rapid assessment of impending cancer risk.

Characterising and dissecting human perception of scene complexity.

Humans can effortlessly assess the complexity of the visual stimuli they encounter. However, our understanding of how we do this, and the relevant factors that result in our perception of scene complexity remain unclear; especially for the natural scenes in which we are constantly immersed. We introduce several new datasets to further understanding of human perception of scene complexity. Our first dataset (VISC-C) contains 800 scenes and 800 corresponding two-dimensional complexity annotations gathered from human observers, allowing exploration for how complexity perception varies across a scene. Our second dataset, (VISC-CI) consists of inverted scenes (reflection on the horizontal axis) with corresponding complexity maps, collected from human observers. Inverting images in this fashion is associated with destruction of semantic scene characteristics when viewed by humans, and hence allows analysis of the impact of semantics on perceptual complexity. We analysed perceptual complexity from both a single-score and a two-dimensional perspective, by evaluating a set of calculable and observable perceptual features based upon grounded psychological research (clutter, symmetry, entropy and openness). We considered these factors' relationship to complexity via hierarchical regressions analyses, tested the efficacy of various neural models against our datasets, and validated our perceptual features against a large and varied complexity dataset consisting of nearly 5000 images. Our results indicate that both global image properties and semantic features are important for complexity perception. We further verified this by combining identified perceptual features with the output of a neural network predictor capable of extracting semantics, and found that we could increase the amount of explained human variance in complexity beyond that of low-level measures alone. Finally, we dissect our best performing prediction network, determining that artificial neurons learn to extract both global image properties and semantic details from scenes for complexity prediction. Based on our experimental results, we propose the "dual information" framework of complexity perception, hypothesising that humans rely on both low-level image features and high-level semantic content to evaluate the complexity of images.

Signals of global symmetry are important for abnormality detection in mammograms.

Purpose: Expert radiologists can detect the "gist of abnormal" in bilateral mammograms even three years prior to onset of cancer. However, their performance decreases if both breasts are not from the same woman, suggesting the ability to detect the abnormality is partly dependent on a global signal present across the two breasts. We aim to detect this implicitly perceived "symmetry" signal by examining its effect on a pre-trained mammography model. Approach: A deep neural network (DNN) with four mammogram view inputs was developed to predict whether the mammograms come from one woman, or two different women as the first step in investigating the symmetry signal. Mammograms were balanced by size, age, density, and machine type. We then evaluated a cancer detection DNN's performance on mammograms from the same and different women. Finally, we used textural analysis methods to further explain the symmetry signal. Results: The developed DNN can detect whether a set of mammograms come from the same or different woman with a base accuracy of 61%. Indeed, a DNN shown mammograms swapped either contralateral or abnormal with a normal mammogram from another woman, resulted in performance decreases. Findings indicate that abnormalities induce a disruption in global mammogram structure resulting in the break in the critical symmetry signal. Conclusion: The global symmetry signal is a textural signal embedded in the parenchyma of bilateral mammograms, which can be extracted. The presence of abnormalities alters textural similarities between the left and right breasts and contributes to the "medical gist signal."

Reliability of radiologists' first impression when interpreting a screening mammogram.

Previous studies showed that radiologists can detect the gist of an abnormality in a mammogram based on a half-second image presentation through global processing of screening mammograms. This study investigated the intra- and inter-observer reliability of the radiologists' initial impressions about the abnormality (or "gist signal"). It also examined if a subset of radiologists produced more reliable and accurate gist signals. Thirty-nine radiologists provided their initial impressions on two separate occasions, viewing each mammogram for half a second each time. The intra-class correlation (ICC) values showed poor to moderate intra-reader reliability. Only 13 radiologists had an ICC of 0.6 or above, which is considered the minimum standard for reliability, and only three radiologists had an ICC exceeding 0.7. The median value for the weighted Cohen's Kappa was 0.478 (interquartile range = 0.419-0.555). The Mann-Whitney U-test showed that the "Gist Experts", defined as those who outperformed others, had significantly higher ICC values (p = 0.002) and weighted Cohen's Kappa scores (p = 0.026). However, even for these experts, the intra-radiologist agreements were not strong, as an ICC of at least 0.75 indicates good reliability and the signal from none of the readers reached this level of reliability as determined by ICC values. The inter-reader reliability of the gist signal was poor, with an ICC score of 0.31 (CI = 0.26-0.37). The Fleiss Kappa score of 0.106 (CI = 0.105-0.106), indicating only slight inter-reader agreement, confirms the findings from the ICC analysis. The intra- and inter-reader reliability analysis showed that the radiologists' initial impressions are not reliable signals. In particular, the absence of an abnormal gist does not reliably signal a normal case, so radiologists should keep searching. This highlights the importance of "discovery scanning," or coarse screening to detect potential targets before ending the visual search.

Modulating human memory for complex scenes with artificially generated images.

Visual memory schemas (VMS) are two-dimensional memorability maps that capture the most memorable regions of a given scene, predicting with a high degree of consistency human observer's memory for the same images. These maps are hypothesized to correlate with a mental framework of knowledge employed by humans to encode visual memories. In this study, we develop a generative model we term 'MEMGAN' constrained by extracted visual memory schemas that generates completely new complex scene images that vary based on their degree of predicted memorability. The generated populations of high and low memorability images are then evaluated for their memorability using a human observer experiment. We gather VMS maps for these generated images from participants in the memory experiment and compare these with the intended target VMS maps. Following the evaluation of observers' memory performance through both VMS-defined memorability and hit rate, we find significantly superior memory performance by human observers for the highly memorable generated images compared to poorly memorable. Implementing and testing a construct from cognitive science allows us to generate images whose memorability we can manipulate at will, as well as providing a tool for further study of mental schemas in humans.

When categories collide: accumulation of information about multiple categories in rapid scene perception.

Experiments have shown that people can rapidly determine if categories such as "animal" or "beach" are present in scenes that are presented for only a few milliseconds. Typically, observers in these experiments report on one prespecified category. For the first time, we show that observers can rapidly extract information about multiple categories. Moreover, we demonstrate task-dependent interactions between accumulating information about different categories in a scene. This interaction can be constructive or destructive, depending on whether the presence of one category can be taken as evidence for or against the presence of the other.

Global processing provides malignancy evidence complementary to the information captured by humans or machines following detailed mammogram inspection.

The information captured by the gist signal, which refers to radiologists' first impression arising from an initial global image processing, is poorly understood. We examined whether the gist signal can provide complementary information to data captured by radiologists (experiment 1), or computer algorithms (experiment 2) based on detailed mammogram inspection. In the first experiment, 19 radiologists assessed a case set twice, once based on a half-second image presentation (i.e., gist signal) and once in the usual viewing condition. Their performances in two viewing conditions were compared using repeated measure correlation (rm-corr). The cancer cases (19 cases × 19 readers) exhibited non-significant trend with rm-corr = 0.012 (p = 0.82, CI: -0.09, 0.12). For normal cases (41 cases × 19 readers), a weak correlation of rm-corr = 0.238 (p < 0.001, CI: 0.17, 0.30) was found. In the second experiment, we combined the abnormality score from a state-of-the-art deep learning-based tool (DL) with the radiological gist signal using a support vector machine (SVM). To obtain the gist signal, 53 radiologists assessed images based on half-second image presentation. The SVM performance for each radiologist and an average reader, whose gist responses were the mean abnormality scores given by all 53 readers to each image was assessed using leave-one-out cross-validation. For the average reader, the AUC for gist, DL, and the SVM, were 0.76 (CI: 0.62-0.86), 0.79 (CI: 0.63-0.89), and 0.88 (CI: 0.79-0.94). For all readers with a gist AUC significantly better than chance-level, the SVM outperformed DL. The gist signal provided malignancy evidence with no or weak associations with the information captured by humans in normal radiologic reporting, which involves detailed mammogram inspection. Adding gist signal to a state-of-the-art deep learning-based tool improved its performance for the breast cancer detection.

Natural cross-modal mappings between visual and auditory features.

The brain may combine information from different sense modalities to enhance the speed and accuracy of detection of objects and events, and the choice of appropriate responses. There is mounting evidence that perceptual experiences that appear to be modality-specific are also influenced by activity from other sensory modalities, even in the absence of awareness of this interaction. In a series of speeded classification tasks, we found spontaneous mappings between the auditory feature of pitch and the visual features of vertical location, size, and spatial frequency but not contrast. By dissociating the task variables from the features that were cross-modally related, we find that the interactions happen in an automatic fashion and are possibly located at the perceptual level.

The Role of Selective Attention in Cross-modal Interactions between Auditory and Visual Features.

Evans and Treisman (2010) showed systematic interactions between audition and vision when participants made speeded classifications in one modality while supposedly ignoring another. We found perceptual facilitation between high pitch and high visual position, high spatial frequency and small size, and interference between high pitch and low position, low spatial frequency and large size, while the converse was the case between low pitch and the same visual features. The present study examined the role of selective attention in these cross-modal interactions. Participants performed speeded classification or search tasks of low or high load while attempting to ignore irrelevant stimuli in a different modality. In both paradigms, congruency between the visual and the irrelevant auditory stimulus had an equal effect in the low and in the high perceptual load conditions. A third experiment tested divided attention, requiring participants to compare stimuli across modalities and respond to the visual-auditory compound. The congruency effect was as large with attention focused on one modality as when it was divided across both. These findings offer converging evidence that cross-modal interactions between corresponding basic features are independent of selective attention.

Defining Image Memorability Using the Visual Memory Schema.

Memorability of an image is a characteristic determined by the human observers' ability to remember images they have seen. Yet recent work on image memorability defines it as an intrinsic property that can be obtained independent of the observer. The current study aims to enhance our understanding and prediction of image memorability, improving upon existing approaches by incorporating the properties of cumulative human annotations. We propose a new concept called the Visual Memory Schema (VMS) referring to an organization of image components human observers share when encoding and recognizing images. The concept of VMS is operationalised by asking human observers to define memorable regions of images they were asked to remember during an episodic memory test. We then statistically assess the consistency of VMSs across observers for either correctly or incorrectly recognised images. The associations of the VMSs with eye fixations and saliency are analysed separately as well. Lastly, we adapt various deep learning architectures for the reconstruction and prediction of memorable regions in images and analyse the results when using transfer learning at the outputs of different convolutional network layers.

Detecting the "gist" of breast cancer in mammograms three years before localized signs of cancer are visible.

OBJECTIVES: After a 500 ms presentation, experts can distinguish abnormal mammograms at above chance levels even when only the breast contralateral to the lesion is shown. Here, we show that this signal of abnormality is detectable 3 years before localized signs of cancer become visible. METHODS: In 4 prospective studies, 59 expert observers from 3 groups viewed 116-200 bilateral mammograms for 500 ms each. Half of the images were prior exams acquired 3 years prior to onset of visible, actionable cancer and half were normal. Exp. 1D included cases having visible abnormalities. Observers rated likelihood of abnormality on a 0-100 scale and categorized breast density. Performance was measured using receiver operating characteristic analysis. RESULTS: In all three groups, observers could detect abnormal images at above chance levels 3 years prior to visible signs of breast cancer (p < 0.001). The results were not due to specific salient cases nor to breast density. Performance was correlated with expertise quantified by the number of mammographic cases read within a year. In Exp. 1D, with cases having visible actionable pathology included, the full group of readers failed to reliably detect abnormal priors; with the exception of a subgroup of the six most experienced observers. CONCLUSIONS: Imaging specialists can detect signals of abnormality in mammograms acquired years before lesions become visible. Detection may depend on expertise acquired by reading large numbers of cases. ADVANCES IN KNOWLEDGE: Global gist signal can serve as imaging risk factor with the potential to identify patients with elevated risk for developing cancer, resulting in improved early cancer diagnosis rates and improved prognosis for females with breast cancer.

Intention, attention and long-term memory for visual scenes: It all depends on the scenes.

Humans have an ability to remember up to 10,000 previously viewed scenes with apparently robust memory for visual detail, a phenomenon that has been interpreted as suggesting a visual memory system of massive capacity. Attempts at explanation have largely focused on the nature of the stimuli and been influenced by theoretical accounts of object recognition. Our own study aims to supplement this by considering two observer-based aspects of visual long-term memory, one strategic, whether the observers are aware or not that their memory will subsequently be tested and the other executive, based on the amount of attentional capacity available during encoding. We describe six studies involving visual scenes ranging in difficulty from complex manmade scenes (d' = 2.54), to door scenes with prominent features removed (d' = 0.79). To ensure processing of the stimuli, all participants have to make a judgement of pleasantness (Experiments 1 and 2) or of the presence or absence of a dot (Experiment 3). Intention to learn influence performance only in the most impoverished condition comprising doors with prominent features removed. Experiments 4-6 investigated the attentional demands of visual long-term memory using a concurrent task procedure. While the demanding task of counting back in threes clearly impaired performance across the range of materials, a lighter load, counting back in ones influences only the most difficult door scenes. Detailed analysis of error patterns indicated that clear differences in performance level between manmade and natural scenes and between unmodified and modified door scenes was reflected in false alarm scores not detections, while concurrent task load affected both. We suggest an interpretation in terms of a two-level process of encoding at the visual feature rather than the whole scene level, with natural images containing many features encoded richly, rapidly and without explicit intent. Only when scenes are selected from a single category and with distinctive detail minimised does memory depend on intention to remember and on the availability of substantial executive capacity.

Inversion effects in the expert classification of mammograms and faces.

A hallmark of a perceptual expert is the ability to detect and categorize stimuli in their domain of expertise after brief exposure. For example, expert radiologists can differentiate between "abnormal" and "normal" mammograms after a 250 ms exposure. It has been speculated that rapid detection depends on a global analysis referred to as holistic perception. Holistic processing in radiology seems similar to holistic perception in which a stimulus like a face is perceived as an integrated whole, not in terms of its individual features. Holistic processing is typically subject to inversion effects in which the inverted image is harder to process/recognize. Is radiological perception similarly subject to inversion effects? Eleven experienced radiologists (> 5 years of radiological experience) and ten resident radiologists (< 5 years of radiological experience) judged upright and inverted bilateral mammograms as "normal" or "abnormal". For comparison, the same participants judged whether upright and inverted faces were "happy" or "neutral". We obtained the expected inversion effect for faces. Expression discrimination was superior for upright faces. For mammograms, experienced radiologists exhibited a similar inversion effect, showing higher accuracy for upright than for inverted mammograms. Less experienced radiology residents performed more poorly than experienced radiologists and demonstrated no inversion effect with mammograms. These results suggest that the ability to discriminate normal from abnormal mammograms is a form of learned, holistic processing.

Radiologists can detect the 'gist' of breast cancer before any overt signs of cancer appear.

Radiologists can detect abnormality in mammograms at above-chance levels after a momentary glimpse of an image. The study investigated this instantaneous perception of an abnormality, known as a "gist" response, when 23 radiologists viewed prior mammograms of women that were reported as normal, but later diagnosed with breast cancer at subsequent screening. Five categories of cases were included: current cancer-containing mammograms, current mammograms of the normal breast contralateral to the cancer, prior mammograms of normal cases, prior mammograms with visible cancer signs in a breast from women who were initially reported as normal, but later diagnosed with breast cancer at subsequent screening in the same breast, and prior mammograms without any visible cancer signs from women labelled as initially normal but subsequently diagnosed with cancer. Our findings suggest that readers can distinguish patients who were diagnosed with cancer, from individuals without breast cancer (normal category), at above-chance levels based on a half-second glimpse of the mammogram even before any lesion becomes visible on the mammogram. Although 20 of the 23 radiologists demonstrated this ability, radiologists' abilities for perceiving the gist of the abnormal varied between the readers and appeared to be linked to expertise. These results could have implications for identifying women of higher than average risk of a future malignancy event, thus impacting upon tailored screening strategies.

Spontaneous retrieval of affective person knowledge in face perception.

In a functional magnetic resonance imaging experiment, we explored whether affective person knowledge based on memories formed from minimal information is spontaneously retrieved in face perception. In the first stage of the experiment, participants were presented with 120 unfamiliar faces. Each face was presented with a description of one of four types of behaviors: aggressive, disgusting, neutral, and nice. In the second stage, participants were scanned while engaged in a one-back recognition task in which they saw the faces that were associated with behaviors and 30 novel faces. Although this task is a simple perceptual task that neither demands person evaluation nor retrieval of person knowledge, neural responses to faces differed as a function of the behaviors. Faces associated with behaviors evoked stronger activity than did novel faces in regions implicated in social cognition--anterior paracingulate cortex and superior temporal sulcus. Explicit memory for the behaviors enhanced the neural response in these regions. Faces associated with disgusting behaviors evoked stronger activity in left anterior insula than did faces associated with aggressive behaviors. This effect was equally strong for faces associated with explicitly recalled behaviors and faces associated with non-recalled behaviors. The findings suggest that affective person knowledge acquired from minimal information is spontaneously retrieved in face perception, engaging neural systems for analysis of social cognition and emotions.

HOW DO RADIOLOGISTS USE THE HUMAN SEARCH ENGINE?

Radiologists perform many 'visual search tasks' in which they look for one or more instances of one or more types of target item in a medical image (e.g. cancer screening). To understand and improve how radiologists do such tasks, it must be understood how the human 'search engine' works. This article briefly reviews some of the relevant work into this aspect of medical image perception. Questions include how attention and the eyes are guided in radiologic search? How is global (image-wide) information used in search? How might properties of human vision and human cognition lead to errors in radiologic search?

Radiologists remember mountains better than radiographs, or do they?

Expertise with encoding material has been shown to aid long-term memory for that material. It is not clear how relevant this expertise is for image memorability (e.g., radiologists' memory for radiographs), and how robust over time. In two studies, we tested scene memory using a standard long-term memory paradigm. One compared the performance of radiologists to naïve observers on two image sets, chest radiographs and everyday scenes, and the other radiologists' memory with immediate as opposed to delayed recognition tests using musculoskeletal radiographs and forest scenes. Radiologists' memory was better than novices for images of expertise but no different for everyday scenes. With the heterogeneity of image sets equated, radiologists' expertise with radiographs afforded them better memory for the musculoskeletal radiographs than forest scenes. Enhanced memory for images of expertise disappeared over time, resulting in chance level performance for both image sets after weeks of delay. Expertise with the material is important for visual memorability but not to the same extent as idiosyncratic detail and variability of the image set. Similar memory decline with time for images of expertise as for everyday scenes further suggests that extended familiarity with an image is not a robust factor for visual memorability.

Perception of objects in natural scenes: is it really attention free?

Studies have suggested attention-free semantic processing of natural scenes in which concurrent tasks leave category detection unimpaired (e.g., F. Li, R. VanRullen, C. Koch, & P. Perona, 2002). Could this ability reflect detection of disjunctive feature sets rather than high-level binding? Participants detected an animal target in a rapid serial visual presentation (RSVP) sequence and then reported its identity and location. They frequently failed to identify or to localize targets that they had correctly detected, suggesting that detection was based only on partial processing. Detection of targets was considerably worse in sequences that also contained humans, presumably because of shared features. When 2 targets were presented in RSVP, a prolonged attentional blink appeared that was almost eliminated when both targets were detected without being identified. The results suggest rapid feature analysis mediating detection, followed by attention-demanding binding for identification and localization.