A Phone in a Basket Looks Like a Knife in a Cup: Role-Filler Independence in Visual Processing.

When a piece of fruit is in a bowl, and the bowl is on a table, we appreciate not only the individual objects and their features, but also the relations containment and support, which abstract away from the particular objects involved. Independent representation of roles (e.g., containers vs. supporters) and "fillers" of those roles (e.g., bowls vs. cups, tables vs. chairs) is a core principle of language and higher-level reasoning. But does such role-filler independence also arise in automatic visual processing? Here, we show that it does, by exploring a surprising error that such independence can produce. In four experiments, participants saw a stream of images containing different objects arranged in force-dynamic relations-e.g., a phone contained in a basket, a marker resting on a garbage can, or a knife sitting in a cup. Participants had to respond to a single target image (e.g., a phone in a basket) within a stream of distractors presented under time constraints. Surprisingly, even though participants completed this task quickly and accurately, they false-alarmed more often to images matching the target's relational category than to those that did not-even when those images involved completely different objects. In other words, participants searching for a phone in a basket were more likely to mistakenly respond to a knife in a cup than to a marker on a garbage can. Follow-up experiments ruled out strategic responses and also controlled for various confounding image features. We suggest that visual processing represents relations abstractly, in ways that separate roles from fillers.

Caricaturing Shapes in Visual Memory.

When representing high-level stimuli, such as faces and animals, we tend to emphasize salient features-such as a face's prominent cheekbones or a bird's pointed beak. Such mental caricaturing leaves traces in memory, which exaggerates these distinctive qualities. How broadly does this phenomenon extend? Here, in six experiments (N = 700 adults), we explored how memory automatically caricatures basic units of visual processing-simple geometric shapes-even without task-related demands to do so. Participants saw a novel shape and then immediately adjusted a copy of that shape to match what they had seen. Surprisingly, participants reconstructed shapes in exaggerated form, amplifying curvature, enlarging salient parts, and so on. Follow-up experiments generalized this bias to new parameters, ruled out strategic responding, and amplified the effects in serial transmission. Thus, even the most basic stimuli we encounter are remembered as caricatures of themselves.

Empirical evidence for perspectival similarity.

When a circular coin is rotated in depth, is there any sense in which it comes to resemble an ellipse? While this question is at the center of a rich and divided philosophical tradition (with some scholars answering affirmatively and some negatively), Morales et al. (2020, 2021) took an empirical approach, reporting 10 experiments whose results favor such perspectival similarity. Recently, Burge and Burge (2022) offered a vigorous critique of this work, objecting to its approach and conclusions on both philosophical and empirical grounds. Here, we answer these objections on both fronts. We show that Burge and Burge's critique rests on misunderstandings of Morales et al.'s claims; of the relation between the data and conclusions; and of the philosophical context in which the work appears. Specifically, Burge and Burge attribute to us a much stronger (and stranger) view than we hold, involving the introduction of "a new entity" located "in some intermediate position(s) between the distal shape and the retinal image." We do not hold this view. Indeed, once properly understood, most of Burge and Burge's objections favor Morales et al.'s claims rather than oppose them. Finally, we discuss several questions that remain unanswered, and reflect on a productive path forward on these issues of foundational scientific and philosophical interest. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Chasing an equation for awarenessPutting Ourselves Back in the Equation George Musser Farrar, Straus and Giroux, 2023. 336 pp.

A writer seeks connections between consciousness and fundamental physics.

Seeing and understanding epistemic actions.

Many actions have instrumental aims, in which we move our bodies to achieve a physical outcome in the environment. However, we also perform actions with epistemic aims, in which we move our bodies to acquire information and learn about the world. A large literature on action recognition investigates how observers represent and understand the former class of actions; but what about the latter class? Can one person tell, just by observing another person's movements, what they are trying to learn? Here, five experiments explore epistemic action understanding. We filmed volunteers playing a "physics game" consisting of two rounds: Players shook an opaque box and attempted to determine i) the number of objects hidden inside, or ii) the shape of the objects inside. Then, independent subjects watched these videos and were asked to determine which videos came from which round: Who was shaking for number and who was shaking for shape? Across several variations, observers successfully determined what an actor was trying to learn, based only on their actions (i.e., how they shook the box)-even when the box's contents were identical across rounds. These results demonstrate that humans can infer epistemic intent from physical behaviors, adding a new dimension to research on action understanding.

Six ways of failing to see (and why the differences matter).

Sometimes we look but fail to see: our car keys on a cluttered desk, a repeated word in a carefully proofread email, or a motorcycle at an intersection. Wolfe and colleagues present a unifying, mechanistic framework for understanding these "Looked But Failed to See" errors, explaining how such misses arise from natural constraints on human visual processing. Here, we offer a conceptual taxonomy of six distinct ways we might be said to fail to see, and explore: how these relate to processes in Wolfe et al.'s model; how they can be distinguished experimentally; and, why the differences matter.

Compositionality in visual perception.

Quilty-Dunn et al.'s wide-ranging defense of the Language of Thought Hypothesis (LoTH) argues that vision traffics in abstract, structured representational formats. We agree: Vision, like language, is compositional - just as words compose into phrases, many visual representations contain discrete constituents that combine in systematic ways. Here, we amass evidence extending this proposal, and explore its implications for how vision interfaces with the rest of the mind.

The perception of silence.

Auditory perception is traditionally conceived as the perception of sounds-a friend's voice, a clap of thunder, a minor chord. However, daily life also seems to present us with experiences characterized by the absence of sound-a moment of silence, a gap between thunderclaps, the hush after a musical performance. In these cases, do we positively hear silence? Or do we just fail to hear, and merely judge or infer that it is silent? This longstanding question remains controversial in both the philosophy and science of perception, with prominent theories holding that sounds are the only objects of auditory experience and thus that our encounter with silence is cognitive, not perceptual. However, this debate has largely remained theoretical, without a key empirical test. Here, we introduce an empirical approach to this theoretical dispute, presenting experimental evidence that silence can be genuinely perceived (not just cognitively inferred). We ask whether silences can "substitute" for sounds in event-based auditory illusions-empirical signatures of auditory event representation in which auditory events distort perceived duration. Seven experiments introduce three "silence illusions"-the one-silence-is-more illusion, silence-based warping, and the oddball-silence illusion-each adapted from a prominent perceptual illusion previously thought to arise only from sounds. Subjects were immersed in ambient noise interrupted by silences structurally identical to the sounds in the original illusions. In all cases, silences elicited temporal distortions perfectly analogous to the illusions produced by sounds. Our results suggest that silence is truly heard, not merely inferred, introducing a general approach for studying the perception of absence.

When will AI misclassify? Intuiting failures on natural images.

Machine recognition systems now rival humans in their ability to classify natural images. However, their success is accompanied by a striking failure: a tendency to commit bizarre misclassifications on inputs specifically selected to fool them. What do ordinary people know about the nature and prevalence of such classification errors? Here, five experiments exploit the recent discovery of "natural adversarial examples" to ask whether naive observers can predict when and how machines will misclassify natural images. Whereas classical adversarial examples are inputs that have been minimally perturbed to induce misclassifications, natural adversarial examples are simply unmodified natural photographs that consistently fool a wide variety of machine recognition systems. For example, a bird casting a shadow might be misclassified as a sundial, or a beach umbrella made of straw might be misclassified as a broom. In Experiment 1, subjects accurately predicted which natural images machines would misclassify and which they would not. Experiments 2 through 4 extended this ability to how the images would be misclassified, showing that anticipating machine misclassifications goes beyond merely identifying an image as nonprototypical. Finally, Experiment 5 replicated these findings under more ecologically valid conditions, demonstrating that subjects can anticipate misclassifications not only under two-alternative forced-choice conditions (as in Experiments 1-4), but also when the images appear one at a time in a continuous stream-a skill that may be of value to human-machine teams. We suggest that ordinary people can intuit how easy or hard a natural image is to classify, and we discuss the implications of these results for practical and theoretical issues at the interface of biological and artificial vision.

Seeing fast and thinking slowThe Border Between Seeing and Thinking Ned Block Oxford University Press, 2023. 560 pp.

A philosopher explores perception and cognition.

Problems and Mysteries of the Many Languages of Thought.

"What is the structure of thought?" is as central a question as any in cognitive science. A classic answer to this question has appealed to a Language of Thought (LoT). We point to emerging research from disparate branches of the field that supports the LoT hypothesis, but also uncovers diversity in LoTs across cognitive systems, stages of development, and species. Our letter formulates open research questions for cognitive science concerning the varieties of rules and representations that underwrite various LoT-based systems and how these variations can help researchers taxonomize cognitive systems.

Visual cognition: A new perspective on mental rotation.

Manipulating an object in one's mind has long been thought to mirror physically manipulating that object in allocentric three-dimensional space. A new study revises and clarifies this foundational assumption, identifying a previously unknown role for the observer's point-of-view.

Beautiful on the inside: Aesthetic preferences and the skeletal complexity of shapes.

A plain, blank canvas does not look very beautiful; to make it aesthetically appealing requires adding structure and complexity. But how much structure is best? In other words, what is the relationship between beauty and complexity? It has long been hypothesized that complexity and beauty meet at a "sweet spot," such that the most beautiful images are neither too simple nor too complex. Here, we take a novel experimental approach to this question, using an information-theoretic approach to object representation based on an internal "skeletal" structure. We algorithmically generated a library of two-dimensional polygons and manipulated their complexity by gradually smoothing out their features-essentially decreasing the amount of information in the objects. We then stylized these shapes as "paintings" by rendering them with artistic strokes, and "mounted" them on framed canvases hung in a virtual room. Participants were shown pairs of these mounted shapes (which possessed similar structures but varied in skeletal complexity) and chose which shape looked best by previewing each painting on the canvas. Experiment 1 revealed a "Goldilocks" effect: participants preferred paintings that were neither too simple nor too complex, such that moderately complex shapes were chosen as the most attractive paintings. Experiment 2 isolated the role of complexity per se: when the same shapes were scrambled (such that their structural complexity was undermined, while other visual features were preserved), the Goldilocks effect was dramatically diminished. These findings suggest a quadratic relationship between aesthetics and complexity in ways that go beyond previous measures of each and demonstrate the utility of information-theoretic approaches for exploring high-level aspects of visual experience.

Can You Hear Me Now? Sensitive Comparisons of Human and Machine Perception.

The rise of machine-learning systems that process sensory input has brought with it a rise in comparisons between human and machine perception. But such comparisons face a challenge: Whereas machine perception of some stimulus can often be probed through direct and explicit measures, much of human perceptual knowledge is latent, incomplete, or unavailable for explicit report. Here, we explore how this asymmetry can cause such comparisons to misestimate the overlap in human and machine perception. As a case study, we consider human perception of adversarial speech - synthetic audio commands that are recognized as valid messages by automated speech-recognition systems but that human listeners reportedly hear as meaningless noise. In five experiments, we adapt task designs from the human psychophysics literature to show that even when subjects cannot freely transcribe such speech commands (the previous benchmark for human understanding), they can sometimes demonstrate other forms of understanding, including discriminating adversarial speech from closely matched nonspeech (Experiments 1 and 2), finishing common phrases begun in adversarial speech (Experiments 3 and 4), and solving simple math problems posed in adversarial speech (Experiment 5) - even for stimuli previously described as unintelligible to human listeners. We recommend the adoption of such "sensitive tests" when comparing human and machine perception, and we discuss the broader consequences of such approaches for assessing the overlap between systems.

Melting Ice With Your Mind: Representational Momentum for Physical States.

When a log burns, it transforms from a block of wood into a pile of ash. Such state changes are among the most dramatic ways objects change, going beyond mere changes of position or orientation. How does the mind represent changes of state? A foundational result in visual cognition is that memory extrapolates the positions of moving objects-a distortion called representational momentum. Here, five experiments (N = 400 adults) exploited this phenomenon to investigate mental representations in state space. Participants who viewed objects undergoing state changes (e.g., ice melting, logs burning, or grapes shriveling) remembered them as more changed (e.g., more melted, burned, or shriveled) than they actually were. This pattern extended to several types of state changes, went beyond their low-level properties, and even adhered to their natural trajectories in state space. Thus, mental representations of objects actively incorporate how they change-not only in their relation to their environment, but also in their essential qualities.

Deepfake detection by human crowds, machines, and machine-informed crowds.

The recent emergence of machine-manipulated media raises an important societal question: How can we know whether a video that we watch is real or fake? In two online studies with 15,016 participants, we present authentic videos and deepfakes and ask participants to identify which is which. We compare the performance of ordinary human observers with the leading computer vision deepfake detection model and find them similarly accurate, while making different kinds of mistakes. Together, participants with access to the model's prediction are more accurate than either alone, but inaccurate model predictions often decrease participants' accuracy. To probe the relative strengths and weaknesses of humans and machines as detectors of deepfakes, we examine human and machine performance across video-level features, and we evaluate the impact of preregistered randomized interventions on deepfake detection. We find that manipulations designed to disrupt visual processing of faces hinder human participants' performance while mostly not affecting the model's performance, suggesting a role for specialized cognitive capacities in explaining human deepfake detection performance.

Seeing and speaking: How verbal "description length" encodes visual complexity.

What is the relationship between complexity in the world and complexity in the mind? Intuitively, increasingly complex objects and events should give rise to increasingly complex mental representations (or perhaps a plateau in complexity after a certain point). However, a counterintuitive possibility with roots in information theory is an inverted-U-shaped relationship between the "objective" complexity of some stimulus and the complexity of its mental representation, because excessively complex patterns might be characterized by surprisingly short computational descriptions (e.g., if they are represented as having been generated randomly). Here, we demonstrate that this is the case, using a novel approach that takes the notion of "description" literally. Subjects saw static and dynamic visual stimuli whose objective complexity could be carefully manipulated, and they described these stimuli in their own words by giving free-form spoken descriptions of them. Across three experiments totaling over 10,000 speech clips, spoken descriptions of shapes (Experiment 1), dot arrays (Experiment 2), and dynamic motion paths (Experiment 3) revealed a striking quadratic relationship between the raw complexity of these stimuli and the length of their spoken descriptions. In other words, the simplest and most complex stimuli received the shortest descriptions, while those stimuli with a medium degree of complexity received the longest descriptions. Follow-up analyses explored the particular words used by subjects, allowing us to further explore how such stimuli were represented. We suggest that the mind engages in a kind of lossy compression for overly complex stimuli, and we discuss the utility of such free-form responses for exploring foundational questions about mental representation. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

"Impossible" Somatosensation and the (Ir)rationality of Perception.

Impossible figures represent the world in ways it cannot be. From the work of M. C. Escher to any popular perception textbook, such experiences show how some principles of mental processing can be so entrenched and inflexible as to produce absurd and even incoherent outcomes that could not occur in reality. However, impossible experiences of this sort are mostly limited to visual perception; are there "impossible figures" for other sensory modalities? Here, we import a known magic trick into the laboratory to report and investigate an impossible experience for somatosensation-one that can be physically felt. We show that, even under full-cue conditions with objects that can be freely inspected, subjects can be made to experience a single object alone as feeling heavier than a group of objects that includes the single object as a member-an impossible and phenomenologically striking experience of weight. Moreover, we suggest that this phenomenon-a special case of the size-weight illusion-reflects a kind of "anti-Bayesian" perceptual updating that amplifies a challenge to rational models of perception and cognition.

The Perception of Relations.

The world contains not only objects and features (red apples, glass bowls, wooden tables), but also relations holding between them (apples contained in bowls, bowls supported by tables). Representations of these relations are often developmentally precocious and linguistically privileged; but how does the mind extract them in the first place? Although relations themselves cast no light onto our eyes, a growing body of work suggests that even very sophisticated relations display key signatures of automatic visual processing. Across physical, eventive, and social domains, relations such as support, fit, cause, chase, and even socially interact are extracted rapidly, are impossible to ignore, and influence other perceptual processes. Sophisticated and structured relations are not only judged and understood, but also seen - revealing surprisingly rich content in visual perception itself.