Seeing and understanding epistemic actions.

Many actions have instrumental aims, in which we move our bodies to achieve a physical outcome in the environment. However, we also perform actions with epistemic aims, in which we move our bodies to acquire information and learn about the world. A large literature on action recognition investigates how observers represent and understand the former class of actions; but what about the latter class? Can one person tell, just by observing another person's movements, what they are trying to learn? Here, five experiments explore epistemic action understanding. We filmed volunteers playing a "physics game" consisting of two rounds: Players shook an opaque box and attempted to determine i) the number of objects hidden inside, or ii) the shape of the objects inside. Then, independent subjects watched these videos and were asked to determine which videos came from which round: Who was shaking for number and who was shaking for shape? Across several variations, observers successfully determined what an actor was trying to learn, based only on their actions (i.e., how they shook the box)-even when the box's contents were identical across rounds. These results demonstrate that humans can infer epistemic intent from physical behaviors, adding a new dimension to research on action understanding.

The perception of silence.

Auditory perception is traditionally conceived as the perception of sounds-a friend's voice, a clap of thunder, a minor chord. However, daily life also seems to present us with experiences characterized by the absence of sound-a moment of silence, a gap between thunderclaps, the hush after a musical performance. In these cases, do we positively hear silence? Or do we just fail to hear, and merely judge or infer that it is silent? This longstanding question remains controversial in both the philosophy and science of perception, with prominent theories holding that sounds are the only objects of auditory experience and thus that our encounter with silence is cognitive, not perceptual. However, this debate has largely remained theoretical, without a key empirical test. Here, we introduce an empirical approach to this theoretical dispute, presenting experimental evidence that silence can be genuinely perceived (not just cognitively inferred). We ask whether silences can "substitute" for sounds in event-based auditory illusions-empirical signatures of auditory event representation in which auditory events distort perceived duration. Seven experiments introduce three "silence illusions"-the one-silence-is-more illusion, silence-based warping, and the oddball-silence illusion-each adapted from a prominent perceptual illusion previously thought to arise only from sounds. Subjects were immersed in ambient noise interrupted by silences structurally identical to the sounds in the original illusions. In all cases, silences elicited temporal distortions perfectly analogous to the illusions produced by sounds. Our results suggest that silence is truly heard, not merely inferred, introducing a general approach for studying the perception of absence.

Deepfake detection by human crowds, machines, and machine-informed crowds.

The recent emergence of machine-manipulated media raises an important societal question: How can we know whether a video that we watch is real or fake? In two online studies with 15,016 participants, we present authentic videos and deepfakes and ask participants to identify which is which. We compare the performance of ordinary human observers with the leading computer vision deepfake detection model and find them similarly accurate, while making different kinds of mistakes. Together, participants with access to the model's prediction are more accurate than either alone, but inaccurate model predictions often decrease participants' accuracy. To probe the relative strengths and weaknesses of humans and machines as detectors of deepfakes, we examine human and machine performance across video-level features, and we evaluate the impact of preregistered randomized interventions on deepfake detection. We find that manipulations designed to disrupt visual processing of faces hinder human participants' performance while mostly not affecting the model's performance, suggesting a role for specialized cognitive capacities in explaining human deepfake detection performance.

Caricaturing Shapes in Visual Memory.

When representing high-level stimuli, such as faces and animals, we tend to emphasize salient features-such as a face's prominent cheekbones or a bird's pointed beak. Such mental caricaturing leaves traces in memory, which exaggerates these distinctive qualities. How broadly does this phenomenon extend? Here, in six experiments (N = 700 adults), we explored how memory automatically caricatures basic units of visual processing-simple geometric shapes-even without task-related demands to do so. Participants saw a novel shape and then immediately adjusted a copy of that shape to match what they had seen. Surprisingly, participants reconstructed shapes in exaggerated form, amplifying curvature, enlarging salient parts, and so on. Follow-up experiments generalized this bias to new parameters, ruled out strategic responding, and amplified the effects in serial transmission. Thus, even the most basic stimuli we encounter are remembered as caricatures of themselves.

Performance vs. competence in human-machine comparisons.

Does the human mind resemble the machines that can behave like it? Biologically inspired machine-learning systems approach "human-level" accuracy in an astounding variety of domains, and even predict human brain activity-raising the exciting possibility that such systems represent the world like we do. However, even seemingly intelligent machines fail in strange and "unhumanlike" ways, threatening their status as models of our minds. How can we know when human-machine behavioral differences reflect deep disparities in their underlying capacities, vs. when such failures are only superficial or peripheral? This article draws on a foundational insight from cognitive science-the distinction between performance and competence-to encourage "species-fair" comparisons between humans and machines. The performance/competence distinction urges us to consider whether the failure of a system to behave as ideally hypothesized, or the failure of one creature to behave like another, arises not because the system lacks the relevant knowledge or internal capacities ("competence"), but instead because of superficial constraints on demonstrating that knowledge ("performance"). I argue that this distinction has been neglected by research comparing human and machine behavior, and that it should be essential to any such comparison. Focusing on the domain of image classification, I identify three factors contributing to the species-fairness of human-machine comparisons, extracted from recent work that equates such constraints. Species-fair comparisons level the playing field between natural and artificial intelligence, so that we can separate more superficial differences from those that may be deep and enduring.

Sustained representation of perspectival shape.

Arguably the most foundational principle in perception research is that our experience of the world goes beyond the retinal image; we perceive the distal environment itself, not the proximal stimulation it causes. Shape may be the paradigm case of such "unconscious inference": When a coin is rotated in depth, we infer the circular object it truly is, discarding the perspectival ellipse projected on our eyes. But is this really the fate of such perspectival shapes? Or does a tilted coin retain an elliptical appearance even when we know it's circular? This question has generated heated debate from Locke and Hume to the present; but whereas extant arguments rely primarily on introspection, this problem is also open to empirical test. If tilted coins bear a representational similarity to elliptical objects, then a circular coin should, when rotated, impair search for a distal ellipse. Here, nine experiments demonstrate that this is so, suggesting that perspectival shapes persist in the mind far longer than traditionally assumed. Subjects saw search arrays of three-dimensional "coins," and simply had to locate a distally elliptical coin. Surprisingly, rotated circular coins slowed search for elliptical targets, even when subjects clearly knew the rotated coins were circular. This pattern arose with static and dynamic cues, couldn't be explained by strategic responding or unfamiliarity, generalized across shape classes, and occurred even with sustained viewing. Finally, these effects extended beyond artificial displays to real-world objects viewed in naturalistic, full-cue conditions. We conclude that objects have a remarkably persistent dual character: their objective shape "out there," and their perspectival shape "from here."

When will AI misclassify? Intuiting failures on natural images.

Machine recognition systems now rival humans in their ability to classify natural images. However, their success is accompanied by a striking failure: a tendency to commit bizarre misclassifications on inputs specifically selected to fool them. What do ordinary people know about the nature and prevalence of such classification errors? Here, five experiments exploit the recent discovery of "natural adversarial examples" to ask whether naive observers can predict when and how machines will misclassify natural images. Whereas classical adversarial examples are inputs that have been minimally perturbed to induce misclassifications, natural adversarial examples are simply unmodified natural photographs that consistently fool a wide variety of machine recognition systems. For example, a bird casting a shadow might be misclassified as a sundial, or a beach umbrella made of straw might be misclassified as a broom. In Experiment 1, subjects accurately predicted which natural images machines would misclassify and which they would not. Experiments 2 through 4 extended this ability to how the images would be misclassified, showing that anticipating machine misclassifications goes beyond merely identifying an image as nonprototypical. Finally, Experiment 5 replicated these findings under more ecologically valid conditions, demonstrating that subjects can anticipate misclassifications not only under two-alternative forced-choice conditions (as in Experiments 1-4), but also when the images appear one at a time in a continuous stream-a skill that may be of value to human-machine teams. We suggest that ordinary people can intuit how easy or hard a natural image is to classify, and we discuss the implications of these results for practical and theoretical issues at the interface of biological and artificial vision.

Compositionality in visual perception.

Quilty-Dunn et al.'s wide-ranging defense of the Language of Thought Hypothesis (LoTH) argues that vision traffics in abstract, structured representational formats. We agree: Vision, like language, is compositional - just as words compose into phrases, many visual representations contain discrete constituents that combine in systematic ways. Here, we amass evidence extending this proposal, and explore its implications for how vision interfaces with the rest of the mind.

Melting Ice With Your Mind: Representational Momentum for Physical States.

When a log burns, it transforms from a block of wood into a pile of ash. Such state changes are among the most dramatic ways objects change, going beyond mere changes of position or orientation. How does the mind represent changes of state? A foundational result in visual cognition is that memory extrapolates the positions of moving objects-a distortion called representational momentum. Here, five experiments (N = 400 adults) exploited this phenomenon to investigate mental representations in state space. Participants who viewed objects undergoing state changes (e.g., ice melting, logs burning, or grapes shriveling) remembered them as more changed (e.g., more melted, burned, or shriveled) than they actually were. This pattern extended to several types of state changes, went beyond their low-level properties, and even adhered to their natural trajectories in state space. Thus, mental representations of objects actively incorporate how they change-not only in their relation to their environment, but also in their essential qualities.

Beautiful on the inside: Aesthetic preferences and the skeletal complexity of shapes.

A plain, blank canvas does not look very beautiful; to make it aesthetically appealing requires adding structure and complexity. But how much structure is best? In other words, what is the relationship between beauty and complexity? It has long been hypothesized that complexity and beauty meet at a "sweet spot," such that the most beautiful images are neither too simple nor too complex. Here, we take a novel experimental approach to this question, using an information-theoretic approach to object representation based on an internal "skeletal" structure. We algorithmically generated a library of two-dimensional polygons and manipulated their complexity by gradually smoothing out their features-essentially decreasing the amount of information in the objects. We then stylized these shapes as "paintings" by rendering them with artistic strokes, and "mounted" them on framed canvases hung in a virtual room. Participants were shown pairs of these mounted shapes (which possessed similar structures but varied in skeletal complexity) and chose which shape looked best by previewing each painting on the canvas. Experiment 1 revealed a "Goldilocks" effect: participants preferred paintings that were neither too simple nor too complex, such that moderately complex shapes were chosen as the most attractive paintings. Experiment 2 isolated the role of complexity per se: when the same shapes were scrambled (such that their structural complexity was undermined, while other visual features were preserved), the Goldilocks effect was dramatically diminished. These findings suggest a quadratic relationship between aesthetics and complexity in ways that go beyond previous measures of each and demonstrate the utility of information-theoretic approaches for exploring high-level aspects of visual experience.

Physically Implied Surfaces.

In addition to seeing objects that are directly in view, we also represent objects that are merely implied (e.g., by occlusion, motion, and other cues). What can imply the presence of an object? Here, we explored (in three preregistered experiments; N = 360 adults) the role of physical interaction in creating impressions of objects that are not actually present. After seeing an actor collide with an invisible wall or step onto an invisible box, participants gave facilitated responses to actual, visible surfaces that appeared where the implied wall or box had been-a Stroop-like pattern of facilitation and interference that suggested automatic inferences about the relevant implied surfaces. Follow-up experiments ruled out confounding geometric cues and anticipatory responses. We suggest that physical interactions can trigger representations of the participating surfaces such that we automatically infer the presence of objects implied only by their physical consequences.

Can resources save rationality? "Anti-Bayesian" updating in cognition and perception.

Resource rationality may explain suboptimal patterns of reasoning; but what of "anti-Bayesian" effects where the mind updates in a direction opposite the one it should? We present two phenomena - belief polarization and the size-weight illusion - that are not obviously explained by performance- or resource-based constraints, nor by the authors' brief discussion of reference repulsion. Can resource rationality accommodate them?

Cognition does not affect perception: Evaluating the evidence for "top-down" effects.

What determines what we see? In contrast to the traditional "modular" understanding of perception, according to which visual processing is encapsulated from higher-level cognition, a tidal wave of recent research alleges that states such as beliefs, desires, emotions, motivations, intentions, and linguistic representations exert direct, top-down influences on what we see. There is a growing consensus that such effects are ubiquitous, and that the distinction between perception and cognition may itself be unsustainable. We argue otherwise: None of these hundreds of studies - either individually or collectively - provides compelling evidence for true top-down effects on perception, or "cognitive penetrability." In particular, and despite their variety, we suggest that these studies all fall prey to only a handful of pitfalls. And whereas abstract theoretical challenges have failed to resolve this debate in the past, our presentation of these pitfalls is empirically anchored: In each case, we show not only how certain studies could be susceptible to the pitfall (in principle), but also how several alleged top-down effects actually are explained by the pitfall (in practice). Moreover, these pitfalls are perfectly general, with each applying to dozens of other top-down effects. We conclude by extracting the lessons provided by these pitfalls into a checklist that future work could use to convincingly demonstrate top-down effects on visual perception. The discovery of substantive top-down effects of cognition on perception would revolutionize our understanding of how the mind is organized; but without addressing these pitfalls, no such empirical report will license such exciting conclusions.

Seeing and thinking: Foundational issues and empirical horizons.

The spectacularly varied responses to our target article raised big-picture questions about the nature of seeing and thinking, nitty-gritty experimental design details, and everything in between. We grapple with these issues, including the ready falsifiability of our view, neuroscientific theories that allow everything but demand nothing, cases where seeing and thinking conflict, mental imagery, the free press, an El Greco fallacy fallacy, hallucinogenic drugs, blue bananas, subatomic particles, Boeing 787s, and the racial identities of geometric shapes.

"Top-down" effects where none should be found: the El Greco fallacy in perception research.

A tidal wave of recent research purports to have discovered that higher-level states such as moods, action capabilities, and categorical knowledge can literally and directly affect how things look. Are these truly effects on perception, or might some instead reflect influences on judgment, memory, or response bias? Here, we exploited an infamous art-historical reasoning error (the so-called "El Greco fallacy") to demonstrate that multiple alleged top-down effects (including effects of morality on lightness perception and effects of action capabilities on spatial perception) cannot truly be effects on perception. We suggest that this error may also contaminate several other varieties of top-down effects and that this discovery has implications for debates over the continuity (or lack thereof) of perception and cognition.

"Please tap the shape, anywhere you like": Shape skeletons in human vision revealed by an exceedingly simple measure.

A major challenge for visual recognition is to describe shapes flexibly enough to allow generalization over different views. Computer vision models have championed a potential solution in medial-axis shape skeletons-hierarchically arranged geometric structures that are robust to deformations like bending and stretching. In the experiments reported here, we exploited an old, unheralded, and exceptionally simple paradigm to reveal the presence and nature of shape skeletons in human vision. When participants independently viewed a shape on a touch-sensitive tablet computer and simply tapped the shape anywhere they wished, the aggregated touches formed the shape's medial-axis skeleton. This pattern held across several shape variations, demonstrating profound and predictable influences of even subtle border perturbations and amodally filled-in regions. This phenomenon reveals novel properties of shape representation and demonstrates (in an unusually direct way) how deep and otherwise-hidden visual processes can directly control simple behaviors, even while observers are completely unaware of their existence.

Empirical evidence for perspectival similarity.

When a circular coin is rotated in depth, is there any sense in which it comes to resemble an ellipse? While this question is at the center of a rich and divided philosophical tradition (with some scholars answering affirmatively and some negatively), Morales et al. (2020, 2021) took an empirical approach, reporting 10 experiments whose results favor such perspectival similarity. Recently, Burge and Burge (2022) offered a vigorous critique of this work, objecting to its approach and conclusions on both philosophical and empirical grounds. Here, we answer these objections on both fronts. We show that Burge and Burge's critique rests on misunderstandings of Morales et al.'s claims; of the relation between the data and conclusions; and of the philosophical context in which the work appears. Specifically, Burge and Burge attribute to us a much stronger (and stranger) view than we hold, involving the introduction of "a new entity" located "in some intermediate position(s) between the distal shape and the retinal image." We do not hold this view. Indeed, once properly understood, most of Burge and Burge's objections favor Morales et al.'s claims rather than oppose them. Finally, we discuss several questions that remain unanswered, and reflect on a productive path forward on these issues of foundational scientific and philosophical interest. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Seeing fast and thinking slowThe Border Between Seeing and Thinking Ned Block Oxford University Press, 2023. 560 pp.

A philosopher explores perception and cognition.

Chasing an equation for awarenessPutting Ourselves Back in the Equation George Musser Farrar, Straus and Giroux, 2023. 336 pp.

A writer seeks connections between consciousness and fundamental physics.