Viewpoint adaptation revealed potential representational differences between 2D images and 3D objects.

For convenience and experimental control, cognitive science has relied largely on images as stimuli rather than the real, tangible objects encountered in the real world. Recent evidence suggests that the cognitive processing of images may differ from real objects, especially in the processing of spatial locations and actions, thought to be mediated by the dorsal visual stream. Perceptual and semantic processing in the ventral visual stream, however, has been assumed to be largely unaffected by the realism of objects. Several studies have found that one key difference accounting for differences between real objects and images is actability; however, less research has investigated another potential difference - the three-dimensional nature of real objects as conveyed by cues like binocular disparity. To investigate the extent to which perception is affected by the realism of a stimulus, we compared viewpoint adaptation when stimuli (a face or a kettle) were 2D (flat images without binocular disparity) vs. 3D (i.e., real, tangible objects or stereoscopic images with binocular disparity). For both faces and kettles, adaptation to 3D stimuli induced stronger viewpoint aftereffects than adaptation to 2D images when the adapting orientation was rightward. A computational model suggested that the difference in aftereffects could be explained by broader viewpoint tuning for 3D compared to 2D stimuli. Overall, our finding narrowed the gap between understanding the neural processing of visual images and real-world objects by suggesting that compared to 2D images, real and simulated 3D objects evoke more broadly tuned neural representations, which may result in stronger viewpoint invariance.

Shared representations of human actions across vision and language.

Humans can recognize and communicate about many actions performed by others. How are actions organized in the mind, and is this organization shared across vision and language? We collected similarity judgments of human actions depicted through naturalistic videos and sentences, and tested four models of action categorization, defining actions at different levels of abstraction ranging from specific (action verb) to broad (action target: whether an action is directed towards an object, another person, or the self). The similarity judgments reflected a shared organization of action representations across videos and sentences, determined mainly by the target of actions, even after accounting for other semantic features. Furthermore, language model embeddings predicted the behavioral similarity of action videos and sentences, and captured information about the target of actions alongside unique semantic information. Together, our results show that action concepts are similarly organized in the mind across vision and language, and that this organization reflects socially relevant goals.

Distinct Neural Components of Visually Guided Grasping during Planning and Execution.

Selecting suitable grasps on three-dimensional objects is a challenging visuomotor computation, which involves combining information about an object (e.g., its shape, size, and mass) with information about the actor's body (e.g., the optimal grasp aperture and hand posture for comfortable manipulation). Here, we used functional magnetic resonance imaging to investigate brain networks associated with these distinct aspects during grasp planning and execution. Human participants of either sex viewed and then executed preselected grasps on L-shaped objects made of wood and/or brass. By leveraging a computational approach that accurately predicts human grasp locations, we selected grasp points that disentangled the role of multiple grasp-relevant factors, that is, grasp axis, grasp size, and object mass. Representational Similarity Analysis revealed that grasp axis was encoded along dorsal-stream regions during grasp planning. Grasp size was first encoded in ventral stream areas during grasp planning then in premotor regions during grasp execution. Object mass was encoded in ventral stream and (pre)motor regions only during grasp execution. Premotor regions further encoded visual predictions of grasp comfort, whereas the ventral stream encoded grasp comfort during execution, suggesting its involvement in haptic evaluation. These shifts in neural representations thus capture the sensorimotor transformations that allow humans to grasp objects.SIGNIFICANCE STATEMENT Grasping requires integrating object properties with constraints on hand and arm postures. Using a computational approach that accurately predicts human grasp locations by combining such constraints, we selected grasps on objects that disentangled the relative contributions of object mass, grasp size, and grasp axis during grasp planning and execution in a neuroimaging study. Our findings reveal a greater role of dorsal-stream visuomotor areas during grasp planning, and, surprisingly, increasing ventral stream engagement during execution. We propose that during planning, visuomotor representations initially encode grasp axis and size. Perceptual representations of object material properties become more relevant instead as the hand approaches the object and motor programs are refined with estimates of the grip forces required to successfully lift the object.

Human Neuroimaging Reveals Differences in Activation and Connectivity between Real and Pantomimed Tool Use.

Because the sophistication of tool use is vastly enhanced in humans compared with other species, a rich understanding of its neural substrates requires neuroscientific experiments in humans. Although functional magnetic resonance imaging (fMRI) has enabled many studies of tool-related neural processing, surprisingly few studies have examined real tool use. Rather, because of the many constraints of fMRI, past research has typically used proxies such as pantomiming despite neuropsychological dissociations between pantomimed and real tool use. We compared univariate activation levels, multivariate activation patterns, and functional connectivity when participants used real tools (a plastic knife or fork) to act on a target object (scoring or poking a piece of putty) or pantomimed the same actions with similar movements and timing. During the Execute phase, we found higher activation for real versus pantomimed tool use in sensorimotor regions and the anterior supramarginal gyrus, and higher activation for pantomimed than real tool use in classic tool-selective areas. Although no regions showed significant differences in activation magnitude during the Plan phase, activation patterns differed between real versus pantomimed tool use and motor cortex showed differential functional connectivity. These results reflect important differences between real tool use, a closed-loop process constrained by real consequences, and pantomimed tool use, a symbolic gesture that requires conceptual knowledge of tools but with limited consequences. These results highlight the feasibility and added value of employing natural tool use tasks in functional imaging, inform neuropsychological dissociations, and advance our theoretical understanding of the neural substrates of natural tool use.SIGNIFICANCE STATEMENT The study of tool use offers unique insights into how the human brain synthesizes perceptual, cognitive, and sensorimotor functions to accomplish a goal. We suggest that the reliance on proxies, such as pantomiming, for real tool use has (1) overestimated the contribution of cognitive networks, because of the indirect, symbolic nature of pantomiming; and (2) underestimated the contribution of sensorimotor networks necessary for predicting and monitoring the consequences of real interactions between hand, tool, and the target object. These results enhance our theoretical understanding of the full range of human tool functions and inform our understanding of neuropsychological dissociations between real and pantomimed tool use.

Grasping with a Twist: Dissociating Action Goals from Motor Actions in Human Frontoparietal Circuits.

In daily life, prehension is typically not the end goal of hand-object interactions but a precursor for manipulation. Nevertheless, functional MRI (fMRI) studies investigating manual manipulation have primarily relied on prehension as the end goal of an action. Here, we used slow event-related fMRI to investigate differences in neural activation patterns between prehension in isolation and prehension for object manipulation. Sixteen (seven males and nine females) participants were instructed either to simply grasp the handle of a rotatable dial (isolated prehension) or to grasp and turn it (prehension for object manipulation). We used representational similarity analysis (RSA) to investigate whether the experimental conditions could be discriminated from each other based on differences in task-related brain activation patterns. We also used temporal multivoxel pattern analysis (tMVPA) to examine the evolution of regional activation patterns over time. Importantly, we were able to differentiate isolated prehension and prehension for manipulation from activation patterns in the early visual cortex, the caudal intraparietal sulcus (cIPS), and the superior parietal lobule (SPL). Our findings indicate that object manipulation extends beyond the putative cortical grasping network (anterior intraparietal sulcus, premotor and motor cortices) to include the superior parietal lobule and early visual cortex.SIGNIFICANCE STATEMENT A simple act such as turning an oven dial requires not only that the CNS encode the initial state (starting dial orientation) of the object but also the appropriate posture to grasp it to achieve the desired end state (final dial orientation) and the motor commands to achieve that state. Using advanced temporal neuroimaging analysis techniques, we reveal how such actions unfold over time and how they differ between object manipulation (turning a dial) versus grasping alone. We find that a combination of brain areas implicated in visual processing and sensorimotor integration can distinguish between the complex and simple tasks during planning, with neural patterns that approximate those during the actual execution of the action.

Familiar size affects perception differently in virtual reality and the real world.

The promise of virtual reality (VR) as a tool for perceptual and cognitive research rests on the assumption that perception in virtual environments generalizes to the real world. Here, we conducted two experiments to compare size and distance perception between VR and physical reality (Maltz et al. 2021 J. Vis. 21, 1-18). In experiment 1, we used VR to present dice and Rubik's cubes at their typical sizes or reversed sizes at distances that maintained a constant visual angle. After viewing the stimuli binocularly (to provide vergence and disparity information) or monocularly, participants manually estimated perceived size and distance. Unlike physical reality, where participants relied less on familiar size and more on presented size during binocular versus monocular viewing, in VR participants relied heavily on familiar size regardless of the availability of binocular cues. In experiment 2, we demonstrated that the effects in VR generalized to other stimuli and to a higher quality VR headset. These results suggest that the use of binocular cues and familiar size differs substantially between virtual and physical reality. A deeper understanding of perceptual differences is necessary before assuming that research outcomes from VR will generalize to the real world. This article is part of a discussion meeting issue 'New approaches to 3D vision'.

Familiar size affects the perceived size and distance of real objects even with binocular vision.

Although the familiar size of real-world objects affects size and distance perception, evidence is mixed about whether this is the case when oculomotor cues are available. We examined the familiar size effect (FSE) on both size and distance perception for real objects under two viewing conditions with full or restricted oculomotor cues (binocular viewing, which provides vergence and accommodation cues, and monocular viewing through a 1-mm pinhole, which removes those cues). Familiar objects (a playing die versus a Rubik's cube) were manufactured in their typical (1.6-cm die and 5.7-cm Rubik's cube) and reverse (5.7-cm die and 1.6-cm Rubik's cube) sizes and shown at two distances (25 cm versus 91 cm) in isolation. Small near and large far objects subtended equal retinal angles. Participants provided manual estimates of perceived size and distance. For every combination of size and distance, Rubik's cubes were perceived as larger and farther than the dice, even during binocular viewing at near distances (<1 meter), when oculomotor cues are particularly strong. For size perception but not distance perception, the familiar size effect was significantly stronger under monocular pinhole viewing than binocular viewing. These results suggest that (1) familiar size affects the accuracy of perception, not just the speed; (2) the effect occurs even when oculomotor cues are available; and (3) size and distance perception are not perfectly yoked.

The Treachery of Images: How Realism Influences Brain and Behavior.

Although the cognitive sciences aim to ultimately understand behavior and brain function in the real world, for historical and practical reasons, the field has relied heavily on artificial stimuli, typically pictures. We review a growing body of evidence that both behavior and brain function differ between image proxies and real, tangible objects. We also propose a new framework for immersive neuroscience to combine two approaches: (i) the traditional build-up approach of gradually combining simplified stimuli, tasks, and processes; and (ii) a newer tear-down approach that begins with reality and compelling simulations such as virtual reality to determine which elements critically affect behavior and brain processing.

Grasping performance depends upon the richness of hand feedback.

Although visual feedback of the hand allows fast and accurate grasping actions, little is known about whether the nature of feedback of the hand affects performance. We investigated kinematics during precision grasping (with the index finger and thumb) when participants received different levels of hand feedback, with or without visual feedback of the target. Specifically, we compared performance when participants saw (1) no hand feedback; (2) only the two critical points on the index finger and thumb tips; (3) 21 points on all digit tips and hand joints; (4) 21 points connected by a "skeleton", or (5) full feedback of the hand wearing a glove. When less hand feedback was available, participants took longer to execute the movement because they allowed more time to slow the reach and close the hand. When target feedback was unavailable, participants took longer to plan the movement and reached with higher velocity. We were particularly interested in investigating maximum grip aperture (MGA), which can reflect the margin of error that participants allow to compensate for uncertainty. A trend suggested that MGA was smallest when ample feedback was available (skeleton and full hand feedback, regardless of target feedback) and when only essential information about hand and target was provided (2-point hand feedback + target feedback) but increased when non-essential points were included (21-point feedback). These results suggest that visual feedback of the hand affects grasping performance and that, while more feedback is usually beneficial, this is not necessarily always the case.

The toolish hand illusion: embodiment of a tool based on similarity with the hand.

A tool can function as a body part yet not feel like one: Putting down a fork after dinner does not feel like losing a hand. However, studies show fake body-parts are embodied and experienced as parts of oneself. Typically, embodiment illusions have only been reported when the fake body-part visually resembles the real one. Here we reveal that participants can experience an illusion that a mechanical grabber, which looks scarcely like a hand, is part of their body. We found changes in three signatures of embodiment: the real hand's perceived location, the feeling that the grabber belonged to the body, and autonomic responses to visible threats to the grabber. These findings show that artificial objects can become embodied even though they bear little visual resemblance to the hand.

Manual exploration of objects is related to 7-month-old infants' visual preference for real objects.

The present study examined whether infants' visual preferences for real objects and pictures are related to their manual object exploration skills. Fifty-nine 7-month-old infants were tested in a preferential looking task with a real object and its pictorial counterpart. All of the infants also participated in a manual object exploration task, in which they freely explored five toy blocks. Results revealed a significant positive relationship between infants' haptic scan levels in the manual object exploration task and their gaze behavior in the preferential looking task: The higher infants' haptic scan levels, the longer they looked at real objects compared to pictures. Our findings suggest that the specific exploratory action of haptically scanning an object is associated with infants' visual preference for real objects over pictures.

Decoding motor imagery and action planning in the early visual cortex: Overlapping but distinct neural mechanisms.

Recent evidence points to a role of the primary visual cortex that goes beyond visual processing into high-level cognitive and motor-related functions, including action planning, even in absence of feedforward visual information. It has been proposed that, at the neural level, motor imagery is a simulation based on motor representations, and neuroimaging studies have shown overlapping and shared activity patterns for motor imagery and action execution in frontal and parietal cortices. Yet, the role of the early visual cortex in motor imagery remains unclear. Here we used multivoxel pattern analyses on functional magnetic resonance imaging (fMRI) data to examine whether the content of motor imagery and action intention can be reliably decoded from the activity patterns in the retinotopic location of the target object in the early visual cortex. Further, we investigated whether the discrimination between specific actions generalizes across imagined and intended movements. Eighteen right-handed human participants (11 females) imagined or performed delayed hand actions towards a centrally located object composed of a small shape attached on a large shape. Actions consisted of grasping the large or small shape, and reaching to the center of the object. We found that despite comparable fMRI signal amplitude for different planned and imagined movements, activity patterns in the early visual cortex, as well as dorsal premotor and anterior intraparietal cortex, accurately represented action plans and action imagery. However, movement content is similar irrespective of whether actions are actively planned or covertly imagined in parietal but not early visual or premotor cortex, suggesting a generalized motor representation only in regions that are highly specialized in object directed grasping actions and movement goals. In sum, action planning and imagery have overlapping but non identical neural mechanisms in the cortical action network.

Do infants show knowledge of the familiar size of everyday objects?

The current study aimed to examine the age at which infants exhibit knowledge of the familiar size of common everyday objects. A total of 65 7- and 12-month-old infants were presented with familiar-sized and novel-sized (i.e., larger or smaller than the familiar size) common everyday objects (i.e., pacifiers and sippy cups), which were placed out of their reach. Both 7- and 12-month-olds' first looks were more frequently directed toward physically larger objects irrespective of whether they were familiar- or novel-sized objects. This finding indicates that initial visual orientation is contingent on the magnitude of the absolute physical size of an object. However, when the entire duration of presentation of the objects (i.e., 10 s) was examined, 12-month-olds' mean looking durations were found to be longer for novel-sized objects than for familiar-sized objects. Thus, although infants in both age groups were able to discern the physical sizes of objects, only 12-month-olds could successfully discriminate between the familiar and novel sizes of everyday objects. Notably, 12-month-olds demonstrated knowledge of familiar size even though the test objects were out of their reach and, consequently, unamenable to manual exploration.

Selective Modulation of Early Visual Cortical Activity by Movement Intention.

The primate visual system contains myriad feedback projections from higher- to lower-order cortical areas, an architecture that has been implicated in the top-down modulation of early visual areas during working memory and attention. Here we tested the hypothesis that these feedback projections also modulate early visual cortical activity during the planning of visually guided actions. We show, across three separate human functional magnetic resonance imaging (fMRI) studies involving object-directed movements, that information related to the motor effector to be used (i.e., limb, eye) and action goal to be performed (i.e., grasp, reach) can be selectively decoded-prior to movement-from the retinotopic representation of the target object(s) in early visual cortex. We also find that during the planning of sequential actions involving objects in two different spatial locations, that motor-related information can be decoded from both locations in retinotopic cortex. Together, these findings indicate that movement planning selectively modulates early visual cortical activity patterns in an effector-specific, target-centric, and task-dependent manner. These findings offer a neural account of how motor-relevant target features are enhanced during action planning and suggest a possible role for early visual cortex in instituting a sensorimotor estimate of the visual consequences of movement.

Psychophysical and neuroimaging responses to moving stimuli in a patient with the Riddoch phenomenon due to bilateral visual cortex lesions.

Patients with injury to early visual cortex or its inputs can display the Riddoch phenomenon: preserved awareness for moving but not stationary stimuli. We provide a detailed case report of a patient with the Riddoch phenomenon, MC. MC has extensive bilateral lesions to occipitotemporal cortex that include most early visual cortex and complete blindness in visual field perimetry testing with static targets. Nevertheless, she shows a remarkably robust preserved ability to perceive motion, enabling her to navigate through cluttered environments and perform actions like catching moving balls. Comparisons of MC's structural magnetic resonance imaging (MRI) data to a probabilistic atlas based on controls reveals that MC's lesions encompass the posterior, lateral, and ventral early visual cortex bilaterally (V1, V2, V3A/B, LO1/2, TO1/2, hV4 and VO1 in both hemispheres) as well as more extensive damage to right parietal (inferior parietal lobule) and left ventral occipitotemporal cortex (VO1, PHC1/2). She shows some sparing of anterior occipital cortex, which may account for her ability to see moving targets beyond ~15 degrees eccentricity during perimetry. Most strikingly, functional and structural MRI revealed robust and reliable spared functionality of the middle temporal motion complex (MT+) bilaterally. Moreover, consistent with her preserved ability to discriminate motion direction in psychophysical testing, MC also shows direction-selective adaptation in MT+. A variety of tests did not enable us to discern whether input to MT+ was driven by her spared anterior occipital cortex or subcortical inputs. Nevertheless, MC shows rich motion perception despite profoundly impaired static and form vision, combined with clear preservation of activation in MT+, thus supporting the role of MT+ in the Riddoch phenomenon.

Object complexity modulates the association between action and perception in childhood.

Vision for action and vision for perception both rely on shape representations derived within the visual system. Whether the same psychological and neural mechanisms underlie both forms of behavior remains hotly contested, and whether this arrangement is equivalent in adults and children is controversial as well. To address these outstanding questions, we used an established psychophysical heuristic, Weber's law, which, in adults, has typically been observed for perceptual judgment tasks but not for actions such as grasping. We examined whether this perception-action dissociation in Weber's law was present in childhood as it is in adulthood and whether it was modulated by stimulus complexity. Two major results emerged. First, although adults evinced visuomotor behavior that violated Weber's law, young children (4.5-6.5 years) adhered to Weber's law when they grasped complex objects ("Efron" blocks), which varied along both the graspable and non-graspable dimensions to maintain a constant surface area, but not when they grasped simple objects, which varied only along the graspable dimension. Second, adherence to Weber's law was found across all ages in the context of a perceptual task. Together, these findings suggest that, in early childhood, visuomotor representations are modulated by perceptual representations, particularly when a refined description of object shape is needed.

Functional interaction between human dorsal premotor cortex and the ipsilateral primary motor cortex for grasp plans: a dual-site TMS study.

Recent findings suggest that the dorsal premotor cortex (PMd), a cortical area in the dorsomedial pathway, is involved in grasp control. It is unclear, however, whether human PMd transfers grasp-related information to the primary motor cortex hand area (M1HAND) during action preparation. The present study tested whether ipsilateral cortico-cortical connections between PMd and M1HAND in the left hemisphere are modulated during grasp preparation. Ten participants performed object-directed grasps and reaches with the right hand. Functional connectivity between left PMd and ipsilateral M1HAND was probed with dual-site transcranial magnetic stimulation. We found that PMd-M1HAND functional interactions were facilitated selectively for the muscles involved in the preparation of the upcoming grasps. The PMd-M1HAND interaction was facilitated for first dorsal interosseous muscle for both precision grip and whole-hand grasps and for abductor digiti minimi muscle for whole-hand grasps. We conclude that human dorsomedial PMd-M1HAND circuit encodes handgrip formation during grasp preparation.

Artificial limb representation in amputees.

The human brain contains multiple hand-selective areas, in both the sensorimotor and visual systems. Could our brain repurpose neural resources, originally developed for supporting hand function, to represent and control artificial limbs? We studied individuals with congenital or acquired hand-loss (hereafter one-handers) using functional MRI. We show that the more one-handers use an artificial limb (prosthesis) in their everyday life, the stronger visual hand-selective areas in the lateral occipitotemporal cortex respond to prosthesis images. This was found even when one-handers were presented with images of active prostheses that share the functionality of the hand but not necessarily its visual features (e.g. a 'hook' prosthesis). Further, we show that daily prosthesis usage determines large-scale inter-network communication across hand-selective areas. This was demonstrated by increased resting state functional connectivity between visual and sensorimotor hand-selective areas, proportional to the intensiveness of everyday prosthesis usage. Further analysis revealed a 3-fold coupling between prosthesis activity, visuomotor connectivity and usage, suggesting a possible role for the motor system in shaping use-dependent representation in visual hand-selective areas, and/or vice versa. Moreover, able-bodied control participants who routinely observe prosthesis usage (albeit less intensively than the prosthesis users) showed significantly weaker associations between degree of prosthesis observation and visual cortex activity or connectivity. Together, our findings suggest that altered daily motor behaviour facilitates prosthesis-related visual processing and shapes communication across hand-selective areas. This neurophysiological substrate for prosthesis embodiment may inspire rehabilitation approaches to improve usage of existing substitutionary devices and aid implementation of future assistive and augmentative technologies.

What Role Does "Elongation" Play in "Tool-Specific" Activation and Connectivity in the Dorsal and Ventral Visual Streams?

Images of tools induce stronger activation than images of nontools in a left-lateralized network that includes ventral-stream areas implicated in tool identification and dorsal-stream areas implicated in tool manipulation. Importantly, however, graspable tools tend to be elongated rather than stubby, and so the tool-selective responses in some of these areas may, to some extent, reflect sensitivity to elongation rather than "toolness" per se. Using functional magnetic resonance imaging, we investigated the role of elongation in driving tool-specific activation in the 2 streams and their interconnections. We showed that in some "tool-selective" areas, the coding of toolness and elongation coexisted, but in others, elongation and toolness were coded independently. Psychophysiological interaction analysis revealed that toolness, but not elongation, had a strong modulation of the connectivity between the ventral and dorsal streams. Dynamic causal modeling revealed that viewing tools (either elongated or stubby) increased the connectivity from the ventral- to the dorsal-stream tool-selective areas, but only viewing elongated tools increased the reciprocal connectivity between these areas. Overall, these data disentangle how toolness and elongation affect the activation and connectivity of the tool network and help to resolve recent controversies regarding the relative contribution of "toolness" versus elongation in driving dorsal-stream "tool-selective" areas.