Featural Representation and Internal Noise Underlie the Eccentricity Effect in Contrast Sensitivity.

Human visual performance for basic visual dimensions (e.g., contrast sensitivity and acuity) peaks at the fovea and decreases with eccentricity. The eccentricity effect is related to the larger visual cortical surface area corresponding to the fovea, but it is unknown if differential feature tuning contributes to this eccentricity effect. Here, we investigated two system-level computations underlying the eccentricity effect: featural representation (tuning) and internal noise. Observers (both sexes) detected a Gabor embedded in filtered white noise which appeared at the fovea or one of four perifoveal locations. We used psychophysical reverse correlation to estimate the weights assigned by the visual system to a range of orientations and spatial frequencies (SFs) in noisy stimuli, which are conventionally interpreted as perceptual sensitivity to the corresponding features. We found higher sensitivity to task-relevant orientations and SFs at the fovea than that at the perifovea, and no difference in selectivity for either orientation or SF. Concurrently, we measured response consistency using a double-pass method, which allowed us to infer the level of internal noise by implementing a noisy observer model. We found lower internal noise at the fovea than that at the perifovea. Finally, individual variability in contrast sensitivity correlated with sensitivity to and selectivity for task-relevant features as well as with internal noise. Moreover, the behavioral eccentricity effect mainly reflects the foveal advantage in orientation sensitivity compared with other computations. These findings suggest that the eccentricity effect stems from a better representation of task-relevant features and lower internal noise at the fovea than that at the perifovea.

Racial Prejudice Affects Representations of Facial Trustworthiness.

What makes faces seem trustworthy? We investigated how racial prejudice predicts the extent to which perceivers employ racially prototypical cues to infer trustworthiness from faces. We constructed participant-level computational models of trustworthiness and White-to-Black prototypicality from U.S. college students' judgments of White (Study 1, N = 206) and Black-White morphed (Study 3, N = 386) synthetic faces. Although the average relationships between models differed across stimuli, both studies revealed that as participants' anti-Black prejudice increased and/or intergroup contact decreased, so too did participants' tendency to conflate White prototypical features with trustworthiness and Black prototypical features with untrustworthiness. Study 2 (N = 324) and Study 4 (N = 397) corroborated that untrustworthy faces constructed from participants with pro-White preferences appeared more Black prototypical to naive U.S. adults, relative to untrustworthy faces modeled from other participants. This work highlights the important role of racial biases in shaping impressions of facial trustworthiness.

What is a compassionate face? Avoided negative affect explains differences between U.S. Americans and Chinese.

While a compassionate face for Germans mirrors others' suffering, for U.S. Americans, a compassionate face is one that expresses a slight smile, partly because U.S. Americans want to avoid feeling negative ("avoided negative affect"; ANA) more than do Germans. The present work examines what people in a non-WEIRD (i.e. Chinese) cultural context think a compassionate face looks like. Additionally, it investigates whether an individually-measured cultural variable (i.e. ANA) can explain differences in conceptualisations of compassion between Chinese and U.S. Americans. Participants in China and the U.S. selected the face that most resembles a compassionate face in a reverse correlation task and completed a measure of ANA. As predicted, Chinese mental representations of a compassionate face included more sadness and less happiness compared to U.S. American mental representations of a compassionate face, and Chinese participants wanted to avoid feeling negative less than did U.S. Americans. Finally, ANA mediated the cultural differences in conceptualisations of compassion. We discuss how ANA and conceptualisations of compassion might be related to how people view the experience versus the expression of different emotions. This work has important implications for therapeutic settings and the meaning of compassion in an increasingly globalised and connected world.

Retinotopic connectivity maps of human visual cortex with unconstrained eye movements.

Human visual cortex contains topographic visual field maps whose organization can be revealed with retinotopic mapping. Unfortunately, constraints posed by standard mapping hinder its use in patients, atypical subject groups, and individuals at either end of the lifespan. This severely limits the conclusions we can draw about visual processing in such individuals. Here, we present a novel data-driven method to estimate connective fields, resulting in fine-grained maps of the functional connectivity between brain areas. We find that inhibitory connectivity fields accompany, and often surround facilitatory fields. The visual field extent of these inhibitory subfields falls off with cortical magnification. We further show that our method is robust to large eye movements and myopic defocus. Importantly, freed from the controlled stimulus conditions in standard mapping experiments, using entertaining stimuli and unconstrained eye movements our approach can generate retinotopic maps, including the periphery visual field hitherto only possible to map with special stimulus displays. Generally, our results show that the connective field method can gain knowledge about retinotopic architecture of visual cortex in patients and participants where this is at best difficult and confounded, if not impossible, with current methods.

What is it about your face that tells me what you want from me? Emotional appeals are associated with specific mental images.

Emotional facial expressions have a communicative function. Besides information about the internal states (emotions) and the intentions of the expresser (action tendencies), they also communicate what the expresser wants the observer to do (appeals). Yet, there is very little research on the association of appeals with specific emotions. The present study has the aim to study the mental association of appeals and expressions through reverse correlation. Using reverse correlation, we estimated the observer-specific internal representations of expressions associated with four different appeals. A second group of participants rated the resulting expressions. As predicted, we found that the appeal to celebrate was uniquely associated with a happy expression and the appeal to empathize with a sad expression. A pleading appeal to stop was more strongly associated with sadness than with anger, whereas a command to stop was comparatively more strongly associated with anger. The results show that observers internally represent appeals as specific emotional expressions.

A compressive sensing approach for inferring cognitive representations with reverse correlation.

Uncovering cognitive representations is an elusive goal that is increasingly pursued using the reverse correlation method, wherein human subjects make judgments about ambiguous stimuli. Employing reverse correlation often entails collecting thousands of stimulus-response pairs, which severely limits the breadth of studies that are feasible using the method. Current techniques to improve efficiency bias the outcome. Here we show that this methodological barrier can be diminished using compressive sensing, an advanced signal processing technique designed to improve sampling efficiency. Simulations are performed to demonstrate that compressive sensing can improve the accuracy of reconstructed cognitive representations and dramatically reduce the required number of stimulus-response pairs. Additionally, compressive sensing is used on human subject data from a previous reverse correlation study, demonstrating a dramatic improvement in reconstruction quality. This work concludes by outlining the potential of compressive sensing to improve representation reconstruction throughout the fields of psychology, neuroscience, and beyond.

Stimulus whitening improves the efficiency of reverse correlation.

Human perception depends upon internal representations of the environment that help to organize the raw information available from the senses by acting as reference patterns. Internal representations are widely characterized using reverse correlation, a method capable of producing unconstrained estimates of the representation itself, all on the basis of simple responses to random stimuli. Despite its advantages, reverse correlation is often infeasible to apply because of its inefficiency-a very large number of stimulus-response trials are required in order to obtain an accurate estimate. Here, we show that an important source of this inefficiency is small, yet nontrivial, correlations that occur by chance between randomly generated stimuli. We demonstrate in simulation that whitening stimuli to remove such correlations before eliciting responses provides greater than 85% improvement in efficiency for a given estimation quality, as well as a two- to fivefold increase in quality for a given sample size. Moreover, unlike conventional approaches, whitening improves the efficiency of reverse correlation without introducing bias into the estimate, or requiring prior knowledge of the target internal representation. Improving the efficiency of reverse correlation with whitening may enable a broader scope of investigations into the individual variability and potential universality of perceptual mechanisms.

Linear Integration of Sensory Evidence over Space and Time Underlies Face Categorization.

Visual object recognition relies on elaborate sensory processes that transform retinal inputs to object representations, but it also requires decision-making processes that read out object representations and function over prolonged time scales. The computational properties of these decision-making processes remain underexplored for object recognition. Here, we study these computations by developing a stochastic multifeature face categorization task. Using quantitative models and tight control of spatiotemporal visual information, we demonstrate that human subjects (five males, eight females) categorize faces through an integration process that first linearly adds the evidence conferred by task-relevant features over space to create aggregated momentary evidence and then linearly integrates it over time with minimum information loss. Discrimination of stimuli along different category boundaries (e.g., identity or expression of a face) is implemented by adjusting feature weights of spatial integration. This linear but flexible integration process over space and time bridges past studies on simple perceptual decisions to complex object recognition behavior.SIGNIFICANCE STATEMENT Although simple perceptual decision-making such as discrimination of random dot motion has been successfully explained as accumulation of sensory evidence, we lack rigorous experimental paradigms to study the mechanisms underlying complex perceptual decision-making such as discrimination of naturalistic faces. We develop a stochastic multifeature face categorization task as a systematic approach to quantify the properties and potential limitations of the decision-making processes during object recognition. We show that human face categorization could be modeled as a linear integration of sensory evidence over space and time. Our framework to study object recognition as a spatiotemporal integration process is broadly applicable to other object categories and bridges past studies of object recognition and perceptual decision-making.

Highly accurate retinotopic maps of the physiological blind spot in human visual cortex.

The physiological blind spot is a naturally occurring scotoma corresponding with the optic disc in the retina of each eye. Even during monocular viewing, observers are usually oblivious to the scotoma, in part because the visual system extrapolates information from the surrounding area. Unfortunately, studying this visual field region with neuroimaging has proven difficult, as it occupies only a small part of retinotopic cortex. Here, we used functional magnetic resonance imaging and a novel data-driven method for mapping the retinotopic organization in and around the blind spot representation in V1. Our approach allowed for highly accurate reconstructions of the extent of an observer's blind spot, and out-performed conventional model-based analyses. This method opens exciting opportunities to study the plasticity of receptive fields after visual field loss, and our data add to evidence suggesting that the neural circuitry responsible for impressions of perceptual completion across the physiological blind spot most likely involves regions of extrastriate cortex-beyond V1.

The Self in the Mind's Eye: Revealing How We Truly See Ourselves Through Reverse Correlation.

Is there a way to visually depict the image people "see" of themselves in their minds' eyes? And if so, what can these mental images tell us about ourselves? We used a computational reverse-correlation technique to explore individuals' mental "self-portraits" of their faces and body shapes in an unbiased, data-driven way (total N = 116 adults). Self-portraits were similar to individuals' real faces but, importantly, also contained clues to each person's self-reported personality traits, which were reliably detected by external observers. Furthermore, people with higher social self-esteem produced more true-to-life self-portraits. Unlike face portraits, body portraits had negligible relationships with individuals' actual body shape, but as with faces, they were influenced by people's beliefs and emotions. We show how psychological beliefs and attitudes about oneself bias the perceptual representation of one's appearance and provide a unique window into the internal mental self-representation-findings that have important implications for mental health and visual culture.

Optimal templates for signal extraction by noisy ideal detectors and human observers.

The optimal template for signal detection in white additive noise is the signal itself: the ideal observer matches each stimulus against this template and selects the stimulus associated with largest match. In the noisy ideal observer, internal noise is added to the decision variable returned by the template. While the ideal observer represents an unrealistic approximation to the human visual process, the noisy ideal observer may be applicable under certain experimental conditions. For template values constrained to lie within a specified range, theory predicts that the template associated with a noisy ideal observer should be a clipped image of the signal, a result which we demonstrate analytically using variational calculus. It is currently unknown whether the human process conforms to theory. We report a targeted analysis of the theoretical prediction for an experimental protocol that maximizes template-matching on the part of human participants. We find indicative evidence to support the theoretical expectation when internal noise is compared across participants, but not within each participant. Our results indicate that implicit knowledge about internal variability in different individuals is reflected by their detection templates; no implicit knowledge is retained for internal-noise fluctuations experienced by a given participant during data collection. The results also indicate that template encoding is constrained by the dynamic range of weight specification, rather than the range of output values transduced by the template-matching process.

Cracking the social code of speech prosody using reverse correlation.

Human listeners excel at forming high-level social representations about each other, even from the briefest of utterances. In particular, pitch is widely recognized as the auditory dimension that conveys most of the information about a speaker's traits, emotional states, and attitudes. While past research has primarily looked at the influence of mean pitch, almost nothing is known about how intonation patterns, i.e., finely tuned pitch trajectories around the mean, may determine social judgments in speech. Here, we introduce an experimental paradigm that combines state-of-the-art voice transformation algorithms with psychophysical reverse correlation and show that two of the most important dimensions of social judgments, a speaker's perceived dominance and trustworthiness, are driven by robust and distinguishing pitch trajectories in short utterances like the word "Hello," which remained remarkably stable whether male or female listeners judged male or female speakers. These findings reveal a unique communicative adaptation that enables listeners to infer social traits regardless of speakers' physical characteristics, such as sex and mean pitch. By characterizing how any given individual's mental representations may differ from this generic code, the method introduced here opens avenues to explore dysprosody and social-cognitive deficits in disorders like autism spectrum and schizophrenia. In addition, once derived experimentally, these prototypes can be applied to novel utterances, thus providing a principled way to modulate personality impressions in arbitrary speech signals.

Improving reverse correlation analysis of faces: Diagnostics of order effects, runs, rater agreement, and image pairs.

Examinations of the reliability and validity of classification images of faces using the reverse correlation approach remain rare. In the present paper, we focus on order effects of trials, compliance, and reliability effects, as well as the degree of contextual contrast of image pairs. We present different diagnostic methods to examine these three aspects using data from 12 reverse correlation studies conducted both in-lab and online with diverse samples (i.e., from Burkina Faso, China, the Netherlands, the U.S., and an international sample) using five different base faces (i.e., female black, female Asian, female and gender-neutral white, and black/white/female/male morphed composite). For each of the 12 studies, we compare the individual CIs of subgroups of likely non-complier respondents and trials with non-contrastful image pairs to individual CIs of likely compliers and contrastful image pairs. In an appendix, we also examine the effects of filtering out data from individual participants and trials on the signal-to-noise ratio of group CIs. R scripts are publicly available for easy implementation of our suggestions in related research.

Nonlinear Processing of Shape Information in Rat Lateral Extrastriate Cortex.

In rodents, the progression of extrastriate areas located laterally to primary visual cortex (V1) has been assigned to a putative object-processing pathway (homologous to the primate ventral stream), based on anatomical considerations. Recently, we found functional support for such attribution (Tafazoli et al., 2017), by showing that this cortical progression is specialized for coding object identity despite view changes, the hallmark property of a ventral-like pathway. Here, we sought to clarify what computations are at the base of such specialization. To this aim, we performed multielectrode recordings from V1 and laterolateral area LL (at the apex of the putative ventral-like hierarchy) of male adult rats, during the presentation of drifting gratings and noise movies. We found that the extent to which neuronal responses were entrained to the phase of the gratings sharply dropped from V1 to LL, along with the quality of the receptive fields inferred through reverse correlation. Concomitantly, the tendency of neurons to respond to different oriented gratings increased, whereas the sharpness of orientation tuning declined. Critically, these trends are consistent with the nonlinear summation of visual inputs that is expected to take place along the ventral stream, according to the predictions of hierarchical models of ventral computations and a meta-analysis of the monkey literature. This suggests an intriguing homology between the mechanisms responsible for building up shape selectivity and transformation tolerance in the visual cortex of primates and rodents, reasserting the potential of the latter as models to investigate ventral stream functions at the circuitry level.SIGNIFICANCE STATEMENT Despite the growing popularity of rodents as models of visual functions, it remains unclear whether their visual cortex contains specialized modules for processing shape information. To addresses this question, we compared how neuronal tuning evolves from rat primary visual cortex (V1) to a downstream visual cortical region (area LL) that previous work has implicated in shape processing. In our experiments, LL neurons displayed a stronger tendency to respond to drifting gratings with different orientations while maintaining a sustained response across the whole duration of the drift cycle. These trends match the increased complexity of pattern selectivity and the augmented tolerance to stimulus translation found in monkey visual temporal cortex, thus revealing a homology between shape processing in rodents and primates.

Comment on "Quantifying the informational value of classification images": A miscomputation of the infoVal metric.

Brinkman et al. (2019) recently introduced an innovative metric-infoVal-to assess the informational value of classification images (CIs) relative to a random distribution. Although this measure constitutes a valuable tool to distinguish random from nonrandom CIs, we identified two noteworthy discrepancies between the mathematical formalization of the infoVal metric and the authors' computation. Specifically, the computation was based on the one norm instead of the Euclidean norm, and the k constant was omitted in the denominator of the ratio that produces infoVal. Accordingly, the simulations and experimental results reported by Brinkman et al. do not build on the correct infoVal computation but on a biased index. Importantly, this discrepancy in the computation affects the statistical power and Type I and error rate of the metric. Here we clarify the nature of the discrepancies in the computation and run Brinkman et al.'s Simulation 1 anew with the correct values, to illustrate their consequences. Overall, we found that relying on the miscomputed infoVal metric can lead to misguided conclusions, and we urge researchers to use the correct values.

Erratum to: Comment on "Quantifying the informational value of classification images": Miscomputation of infoVal metric was a minor issue and is now corrected.

One of the two miscomputations identified in the infoVal metric, namely the omission of the k constant, turns out not to be a miscomputation, since the constant was already taken into account by default in the mad() function from R (see https://www.rdocumentation.org/packages/stats/versions/3.6.2/topics/mad ).

Differentiating between Models of Perceptual Decision Making Using Pupil Size Inferred Confidence.

During perceptual decisions, subjects often rely more strongly on early, rather than late, sensory evidence, even in tasks when both are equally informative about the correct decision. This early psychophysical weighting has been explained by an integration-to-bound decision process, in which the stimulus is ignored after the accumulated evidence reaches a certain bound, or confidence level. Here, we derive predictions about how the average temporal weighting of the evidence depends on a subject's decision confidence in this model. To test these predictions empirically, we devised a method to infer decision confidence from pupil size in 2 male monkeys performing a disparity discrimination task. Our animals' data confirmed the integration-to-bound predictions, with different internal decision bounds and different levels of correlation between pupil size and decision confidence accounting for differences between animals. However, the data were less compatible with two alternative accounts for early psychophysical weighting: attractor dynamics either within the decision area or due to feedback to sensory areas, or a feedforward account due to neuronal response adaptation. This approach also opens the door to using confidence more broadly when studying the neural basis of decision making.SIGNIFICANCE STATEMENT An animal's ability to adjust decisions based on its level of confidence, sometimes referred to as "metacognition," has generated substantial interest in neuroscience. Here, we show how measurements of pupil diameter in macaques can be used to infer their confidence. This technique opens the door to more neurophysiological studies of confidence because it eliminates the need for training on behavioral paradigms to evaluate confidence. We then use this technique to test predictions from competing explanations of why subjects in perceptual decision making often rely more strongly on early evidence: the way in which the strength of this effect should depend on a subject's decision confidence. We find that a bounded decision formation process best explains our empirical data.

Stronger responses to darks along the ventral pathway of the cat visual cortex.

Light increments (brights) and decrements (darks) are differently processed throughout the early visual system. It is well known that a bias towards faster and stronger responses to darks is present in the retina, lateral geniculate nucleus and primary visual cortex. In humans, psychophysical and neurophysiological data indicate that darks are better detected than brights, suggesting that the dark bias found in early visual areas is transmitted across the cortical hierarchy. Here, we tested this assumption by investigating the spatiotemporal features of responses to brights and darks in area 21a, a gateway area of the cat ventral stream, using reverse correlation analysis of a sparse noise stimulus. The receptive field of most 21a neurons exhibited larger dark subfields. Additionally, the amplitude of the responses to darks was considerably greater than those evoked by brights. In the temporal domain, no differences were found between the response peak latency. Thus, the present study supports the notion that bright/dark asymmetries are transmitted throughout the cortical hierarchy and further, that the luminance processing varies as a function of the position in the cortical hierarchy, dark preference being strongly enhanced (in the spatial domain and response amplitude) along the ventral pathway.

Feature-based face representations and image reconstruction from behavioral and neural data.

The reconstruction of images from neural data can provide a unique window into the content of human perceptual representations. Although recent efforts have established the viability of this enterprise using functional magnetic resonance imaging (MRI) patterns, these efforts have relied on a variety of prespecified image features. Here, we take on the twofold task of deriving features directly from empirical data and of using these features for facial image reconstruction. First, we use a method akin to reverse correlation to derive visual features from functional MRI patterns elicited by a large set of homogeneous face exemplars. Then, we combine these features to reconstruct novel face images from the corresponding neural patterns. This approach allows us to estimate collections of features associated with different cortical areas as well as to successfully match image reconstructions to corresponding face exemplars. Furthermore, we establish the robustness and the utility of this approach by reconstructing images from patterns of behavioral data. From a theoretical perspective, the current results provide key insights into the nature of high-level visual representations, and from a practical perspective, these findings make possible a broad range of image-reconstruction applications via a straightforward methodological approach.

Quantifying the informational value of classification images.

Reverse correlation is an influential psychophysical paradigm that uses a participant's responses to randomly varying images to build a classification image (CI), which is commonly interpreted as a visualization of the participant's mental representation. It is unclear, however, how to statistically quantify the amount of signal present in CIs, which limits the interpretability of these images. In this article, we propose a novel metric, infoVal, which assesses informational value relative to a resampled random distribution and can be interpreted like a z score. In the first part, we define the infoVal metric and show, through simulations, that it adheres to typical Type I error rates under various task conditions (internal validity). In the second part, we show that the metric correlates with markers of data quality in empirical reverse-correlation data, such as the subjective recognizability, objective discriminability, and test-retest reliability of the CIs (convergent validity). In the final part, we demonstrate how the infoVal metric can be used to compare the informational value of reverse-correlation datasets, by comparing data acquired online with data acquired in a controlled lab environment. We recommend a new standard of good practice in which researchers assess the infoVal scores of reverse-correlation data in order to ensure that they do not read signal in CIs where no signal is present. The infoVal metric is implemented in the open-source rcicr R package, to facilitate its adoption.