Dependence of behavioral performance on material category in an object-grasping task with monkeys.

Material perception is an essential part of our cognitive function that enables us to properly interact with our complex daily environment. One important aspect of material perception is its multimodal nature. When we see an object, we generally recognize its haptic properties as well as its visual properties. Consequently, one must examine behavior using real objects that are perceived both visually and haptically to fully understand the characteristics of material perception. As a first step, we examined whether there is any difference in the behavioral responses to different materials in monkeys trained to perform an object grasping task in which they saw and grasped rod-shaped real objects made of various materials. We found that the monkeys' behavior in the grasping task, which was measured based on the success rate and the pulling force, differed depending on the material category. Monkeys easily and correctly grasped objects of some materials, such as metal and glass, but failed to grasp objects of other materials. In particular, monkeys avoided grasping fur-covered objects. The differences in the behavioral responses to the material categories cannot be explained solely based on the degree of familiarity with the different materials. These results shed light on the organization of multimodal representation of materials, where their biological significance is an important factor. In addition, a monkey that avoided touching real fur-covered objects readily touched images of the same objects presented on a CRT display. This suggests that employing real objects is important when studying behaviors related to material perception. NEW & NOTEWORTHY We tested monkeys using an object-grasping task in which monkeys saw and grasped rod-shaped real objects made of various materials. We found that the monkeys' behavior differed dramatically across the material categories and that the behavioral differences could not be explained solely based on the degree of familiarity with the different materials. These results shed light on the organization of multimodal representation of materials, where the biological significance of materials is an important factor.

Representation of the material properties of objects in the visual cortex of nonhuman primates.

Information about the material from which objects are made provide rich and useful clues that enable us to categorize and identify those objects, know their state (e.g., ripeness of fruits), and properly act on them. However, despite its importance, little is known about the neural processes that underlie material perception in nonhuman primates. Here we conducted an fMRI experiment in awake macaque monkeys to explore how information about various real-world materials is represented in the visual areas of monkeys, how these neural representations correlate with perceptual material properties, and how they correspond to those in human visual areas that have been studied previously. Using a machine-learning technique, the representation in each visual area was read out from multivoxel patterns of regional activity elicited in response to images of nine real-world material categories (metal, wood, fur, etc.). The congruence of the neural representations with either a measure of low-level image properties, such as spatial frequency content, or with the visuotactile properties of materials, such as roughness, hardness, and warmness, were tested. We show that monkey V1 shares a common representation with human early visual areas reflecting low-level image properties. By contrast, monkey V4 and the posterior inferior temporal cortex represent the visuotactile properties of material, as in human ventral higher visual areas, although there were some interspecies differences in the representational structures. We suggest that, in monkeys, V4 and the posterior inferior temporal cortex are important stages for constructing information about the material properties of objects from their low-level image features.

Perceptual gloss parameters are encoded by population responses in the monkey inferior temporal cortex.

There are neurons localized in the lower bank of the superior temporal sulcus (STS) in the inferior temporal (IT) cortex of the monkey that selectively respond to specific ranges of gloss characterized by combinations of three physical reflectance parameters: specular reflectance (ρs), diffuse reflectance (ρd), and spread of specular reflection (α; Nishio et al., 2012). In the present study, we examined how the activities of these gloss-selective IT neurons are related to perceived gloss. In an earlier psychophysical study, Ferwerda et al. (2001) identified a perceptually uniform gloss space defined by two axes where the c-axis corresponds to a nonlinear combination of ρs and ρd and the d-axis corresponds to 1 - α. In the present study, we tested the responses of gloss-selective neurons to stimuli in the perceptual gloss space defined by the c- and d-axes. We found that gloss-selective neurons systematically changed their responses in the perceptual gloss space, and the distribution of the tuning directions of the population of gloss-selective neurons is biased toward directions in which perceived gloss increases. We also found that a set of perceptual gloss parameters as well as surface albedo can be well explained by the population activities of gloss-selective neurons, and that these parameters are likely encoded by the gloss-selective neurons in this area of the STS to represent various glosses. These results thus provide evidence that the IT cortex represents perceptual gloss space.

Audiovisual integration in the human perception of materials.

Interest in the perception of the material of objects has been growing. While material perception is a critical ability for animals to properly regulate behavioral interactions with surrounding objects (e.g., eating), little is known about its underlying processing. Vision and audition provide useful information for material perception; using only its visual appearance or impact sound, we can infer what an object is made from. However, what material is perceived when the visual appearance of one material is combined with the impact sound of another, and what are the rules that govern cross-modal integration of material information? We addressed these questions by asking 16 human participants to rate how likely it was that audiovisual stimuli (48 combinations of visual appearances of six materials and impact sounds of eight materials) along with visual-only stimuli and auditory-only stimuli fell into each of 13 material categories. The results indicated strong interactions between audiovisual material perceptions; for example, the appearance of glass paired with a pepper sound is perceived as transparent plastic. Rating material-category likelihoods follow a multiplicative integration rule in that the categories judged to be likely are consistent with both visual and auditory stimuli. On the other hand, rating-material properties, such as roughness and hardness, follow a weighted average rule. Despite a difference in their integration calculations, both rules can be interpreted as optimal Bayesian integration of independent audiovisual estimations for the two types of material judgment, respectively.

Neural selectivity and representation of gloss in the monkey inferior temporal cortex.

When we view an object, its appearance depends in large part on specific surface reflectance properties; among these is surface gloss, which provides important information about the material composition of the object and the fine structure of its surface. To study how gloss is represented in the visual cortical areas related to object recognition, we examined the responses of neurons in the inferior temporal (IT) cortex of the macaque monkey to a set of object images exhibiting various combinations of specular reflection, diffuse reflection, and roughness, which are important physical parameters of surface gloss. We found that there are neurons in the lower bank of the superior temporal sulcus that selectively respond to specific gloss. This neuronal selectivity was largely maintained when the shape or illumination of the object was modified and perceived glossiness was unchanged. By contrast, neural responses were significantly altered when the pixels of the images were randomly rearranged, and perceived glossiness was dramatically changed. The stimulus preference of these neurons differed from cell to cell, and, as a population, they systematically represented a variety of surface glosses. We conclude that, within the visual cortex, there are mechanisms operating to integrate local image features and extract information about surface gloss and that this information is systematically represented in the IT cortex, an area playing an important role in object recognition.

Selective responses to specular surfaces in the macaque visual cortex revealed by fMRI.

The surface properties of objects, such as gloss, transparency and texture, provide important information about the material characteristics of objects in our visual environment. However, because there have been few reports on the neuronal responses to surface properties in primates, we still lack information about where and how surface properties are processed in the primate visual cortex. In this study, we used functional magnetic resonance imaging (fMRI) to examine the cortical responses to specular surfaces in the macaque visual cortex. Using computer graphics, we generated images of specular and matte objects and prepared scrambled images by locally randomizing the luminance phases of the images with specular and matte objects. In experiment 1, we contrasted the responses to specular images with those to matte and scrambled images. Activation was observed along the ventral visual pathway, including V1, V2, V3, V4 and the posterior inferior temporal (IT) cortex. In experiment 2, we manipulated the contrasts of images and found that the activation observed in these regions could not be explained solely by the global or local contrasts. These results suggest that image features related to specular surface are processed along the ventral visual pathway from V1 to specific regions in the IT cortex. This is consistent with previous human fMRI experiments that showed surface properties are processed in the ventral visual pathway.

Cerebro-cerebellar interactions in nonhuman primates examined by optogenetic functional magnetic resonance imaging.

Functional magnetic resonance imaging (fMRI) is a promising approach for the simultaneous and extensive scanning of whole-brain activities. Optogenetics is free from electrical and magnetic artifacts and is an ideal stimulation method for combined use with fMRI. However, the application of optogenetics in nonhuman primates (NHPs) remains limited. Recently, we developed an efficient optogenetic intracortical microstimulation method of the primary motor cortex (M1), which successfully induced forelimb movements in macaque monkeys. Here, we aimed to investigate how optogenetic M1 stimulation causes neural modulation in the local and remote brain regions in anesthetized monkeys using 7-tesla fMRI. We demonstrated that optogenetic stimulation of the M1 forelimb and hindlimb regions successfully evoked robust direct and remote fMRI activities. Prominent remote activities were detected in the anterior and posterior lobes in the contralateral cerebellum, which receive projections polysynaptically from the M1. We further demonstrated that the cerebro-cerebellar projections from these M1 regions were topographically organized, which is concordant with the somatotopic map in the cerebellar cortex previously reported in macaques and humans. The present study significantly enhances optogenetic fMRI in NHPs, resulting in profound understanding of the brain network, thereby accelerating the translation of findings from animal models to humans.

Transformation from image-based to perceptual representation of materials along the human ventral visual pathway.

Every object in the world has its own surface quality that is a reflection of the material from which the object is made. We can easily identify and categorize materials (wood, metal, fabric etc.) at a glance, and this ability enables us to decide how to interact appropriately with these objects. Little is known, however, about how materials are represented in the brain, or how that representation is related to material perception or the physical properties of material surface. By combining multivoxel pattern analysis of functional magnetic resonance imaging data with perceptual and image-based physical measures of material properties, we found that the way visual information about materials is coded gradually changes from an image-based representation in early visual areas to a perceptual representation in the ventral higher-order visual areas. We suggest that meaningful information about multimodal aspects of real-world materials reside in the ventral cortex around the fusiform gyrus, where it can be utilized for categorization of materials.

Mechanisms underlying the representation of angles embedded within contour stimuli in area V2 of macaque monkeys.

We previously found that surprisingly many V2 neurons showed selective responses to particular angles embedded within continuous contours [M. Ito & H. Komatsu (2004)Journal of Neuroscience, 24, 3313-3324]. Here, we addressed whether the selectivity is dependent on the presence of individual constituent components or on the unique combination of these components. To reveal roles of constituent half-lines in response to whole angles, we conducted a quantitative model study after the framework of cascade models. Our linear-non-linear summation model implemented a few subunits selective to particular half-lines and was fitted to neuronal responses for each neuron. The study indicates that the best-fitting models well replicate the selectivity in the majority of V2 neurons and that the angle selectivity is dependent on a linear combination of responses to individual half-line components of the angles. The implication is that optimal angles are given by a combination of two preferred half-line components and the selectivity is sharpened by introducing suppression to non-preferred half-line components, rather than a specific facilitatory interaction between two preferred half-line components. The study indicates the participation of the gain control of responsiveness according to the number of half-line components. We also showed that the selectivity to acute angles depends on a combination of responses to one preferred component and weak responses to another component. Therefore, we concluded that the angle selectivity is dependent on selective responses to individual half-line components of the angles rather than a unique combination between them, whereas neurons could be selective to various angle widths at area V2.

Structure and function of neural circuit related to gloss perception in the macaque inferior temporal cortex: a case report.

The inferior temporal (IT) cortex of the macaque monkey plays a pivotal role in the visual recognition of objects. In the IT cortex, a feature-selective network formed by connecting subregions specialized for common visual features seems to be a basic strategy for processing biologically important visual features. Gloss perception plays an important role in the judgment of materials and conditions of objects and is a biologically significant visual function. In the present study, we attempted to determine whether a neural circuit specialized for processing information related to gloss perception exists in the IT cortex in one monkey. We injected retrograde tracer into a gloss-selective subregion in the IT cortex where gloss-selective neurons were clustered in the neural recording experiment, and anatomically examined its neural connections. We observed that retrogradely labeled neurons were densely accumulated in multiple locations in the posterior and anterior IT cortices. Based on the results of this case study, we will discuss the possibility that, together with the injection site, the sites with a dense cluster of labeled neurons form feature-selective neural circuits for the processing of gloss information in the IT cortex.

Distribution of colour-selective activity in the monkey inferior temporal cortex revealed by functional magnetic resonance imaging.

Previous electrophysiological, neuroimaging and lesion studies have suggested that the anterior part of the monkey inferior temporal (IT) cortex, or area TE, plays an important role in colour processing. However, little is known about how colour information is distributed in these cortical regions. Here, we explored the distribution of colour-selective activity in alert macaque monkeys using functional magnetic resonance imaging (fMRI) with two types of stimuli: a multicoloured ('Mondrian') pattern and an isoluminant colour grating. These two types of stimuli are both commonly used in human fMRI studies, but Mondrian stimuli, which contain a richer variety of hues and hence might be more suitable for activating higher-order areas than grating stimuli, have not been used to examine colour-selectivity in higher-order areas in earlier monkey studies. With the Mondrian stimuli, we observed that areas along the ventral pathway, V1, V2/V3, V4 and the IT cortex, responded more strongly to colour stimuli than to luminance stimuli. In the IT cortex, we found that colour-selective activities are not distributed uniformly, but are localized in discrete regions, each extending several millimetres in the anterior or posterior part of the IT cortex. The colour-selective activation in the anterior IT was observed only with the Mondrian stimuli, whereas the colour-selective activation in the posterior IT was observed with both the Mondrian and grating stimuli, with little overlap. These findings suggest that there are multiple subregions with differing stimulus selectivities distributed in the IT cortex, and that colour information is processed in these discrete subregions.

The effects of feature attention on prestimulus cortical activity in the human visual system.

Covert attention affects prestimulus activity in the visual cortex. Although most studies investigating neural mechanisms of attention have focused on the effects of spatial attention, attention can also be directed to particular features. To investigate the spatiotemporal nature of feature attention, we measured subjects' brain activity using magnetoencephalography (MEG) and functional magnetic resonance imaging (fMRI) while subjects attended to color or motion of a stimulus based on a visual cue, which was presented 1 s before the stimulus onset. We used the hierarchical Bayesian method that allows us to estimate cortical currents with MEG and fMRI data in the order of millimeters and milliseconds. When subjects attended to color, activity within the color-sensitive area (fusiform gyrus) was selectively enhanced within the prestimulus period. By contrast, when subjects attended to motion, activity within the motion-sensitive area (middle temporal gyrus) was selectively enhanced during this period. This effect was not seen in frontal, parietal, and lower visual areas. Additionally, this effect was transient rather than sustained, suggesting that it differs from temporal aspects of spatial attention. These results suggest that, although both spatial and feature attention modulate prestimulus activity within specific visual areas, neural mechanisms underlying these effects might be different.

Neural Mechanisms of Material Perception: Quest on Shitsukan.

In recent years, a growing body of research has addressed the nature and mechanism of material perception. Material perception entails perceiving and recognizing a material, surface quality or internal state of an object based on sensory stimuli such as visual, tactile, and/or auditory sensations. This process is ongoing in every aspect of daily life. We can, for example, easily distinguish whether an object is made of wood or metal, or whether a surface is rough or smooth. Judging whether the ground is wet or dry or whether a fish is fresh also involves material perception. Information obtained through material perception can be used to govern actions toward objects and to make decisions about whether to approach an object or avoid it. Because the physical processes leading to sensory signals related to material perception is complicated, it has been difficult to manipulate experimental stimuli in a rigorous manner. However, that situation is now changing thanks to advances in technology and knowledge in related fields. In this article, we will review what is currently known about the neural mechanisms responsible for material perception. We will show that cortical areas in the ventral visual pathway are strongly involved in material perception. Our main focus is on vision, but every sensory modality is involved in material perception. Information obtained through different sensory modalities is closely linked in material perception. Such cross-modal processing is another important feature of material perception, and will also be covered in this review.

Estimation of the timing of human visual perception from magnetoencephalography.

To explore the timing and the underlying neural dynamics of visual perception, we analyzed the relationship between the manual reaction time (RT) to the onset of a visual stimulus and the time course of the evoked neural response simultaneously measured by magnetoencephalography (MEG). The visual stimuli were a transition from incoherent to coherent motion of random dots and an onset of a chromatic grating from a uniform field, which evoke neural responses in different cortical sites. For both stimuli, changes in median RT with changing stimulus strength (motion coherence or chromatic contrast) were accurately predicted, with a stimulus-independent postdetection delay, from the time that the temporally integrated MEG response crossed a threshold (integrator model). In comparison, the prediction of RT was less accurate from the peak MEG latency, or from the time that the nonintegrated MEG response crossed a threshold (level detector model). The integrator model could also account for, at least partially, intertrial changes in RT or in perception (hit/miss) to identical stimuli. Although we examined MEG-RT relationships mainly for data averaged over trials, the integrator model could show some correlations even for single-trial data. The model predictions deteriorated when only early visual responses presumably originating from the striate cortex were used as the input to the integrator model. Our results suggest that the perceptions for visual stimulus appearances are established in extrastriate areas [around MT (middle temporal visual area) for motion and around V4 (fourth visual area) for color] approximately 150-200 ms before subjects manually react to the stimulus.

Anterior and superior lateral occipito-temporal cortex responsible for target motion prediction during overt and covert visual pursuit.

In smooth-pursuit eye movements (SPEM) with gain close to one, SPEM should be controlled mainly by prediction of target motion because retinal slip is nearly zero. We investigated the neural mechanisms of visual-target prediction by the three fMRI experiments. (1) Overt pursuit task: subjects pursued a sinusoidally moving target which blinked (blink condition) or did not blink (continuous condition). (2) Covert pursuit task: subjects covertly pursued the same target with eyes gazed at fixation point. (3) Attend-to-stationary target task: subjects brought attention on a stationary target with eyes gazed at fixation point. In the overt pursuit task, the SPEM gain and the delay in the blink condition were not very different from the continuous condition, indicating good prediction of the blinking target motion. Activities in the dorsolateral prefrontal, precentral, medial superior frontal, intraparietal, and lateral occipito-temporal cortexes increased in the blink-continuous subtraction. The V1 activity decreased for this contrast. In the covert pursuit task, only the anterior/superior LOTC activity remained in the blink-continuous subtraction. In the attend-to-stationary target task, the blink-continuous subtraction elicited no activation. Consequently, the a/sLOTC activity is responsible for target prediction rather than motor commands for eye movements or just target blinking such as visual saliency.

Crossmodal Association of Visual and Haptic Material Properties of Objects in the Monkey Ventral Visual Cortex.

Just by looking at an object, we can recognize its non-visual properties, such as hardness. The visual recognition of non-visual object properties is generally accurate [1], and influences actions toward the object [2]. Recent studies suggest that, in the primate brain, this may involve the ventral visual cortex, which represents objects in a way that reflects not only visual but also non-visual object properties, such as haptic roughness, hardness, and weight [3-7]. This new insight raises a fundamental question: how does the visual cortex come to represent non-visual properties--knowledge that cannot be acquired directly through vision? Here we addressed this unresolved question using fMRI in macaque monkeys. Specifically, we explored whether and how simple visuo-haptic experience--just seeing and touching objects made of various materials--can shape representational content in the visual cortex. We measured brain activity evoked by viewing images of objects before and after the monkeys acquired the visuo-haptic experience and decoded the representational space from the activity patterns [8]. We show that simple long-term visuo-haptic experience greatly impacts representation in the posterior inferior temporal cortex, the higher ventral visual cortex. After the experience, but not before, the activity pattern in this region well reflected the haptic material properties of the experienced objects. Our results suggest that neural representation of non-visual object properties in the visual cortex emerges through long-term crossmodal exposure to objects. This highlights the importance of unsupervised learning of crossmodal associations through everyday experience [9-12] for shaping representation in the visual cortex.

Attentional shifts towards an expected visual target alter the level of alpha-band oscillatory activity in the human calcarine cortex.

Neuronal operations associated with the top-down control process of shifting attention from one locus to another involve a network of cortical regions, and their influence is deemed fundamental to visual perception. However, the extent and nature of these operations within primary visual areas are unknown. In this paper, we used magnetoencephalography (MEG) in combination with magnetic resonance imaging (MRI) to determine whether, prior to the onset of a visual stimulus, neuronal activity within early visual cortex is affected by covert attentional shifts. Time/frequency analyses were used to identify the nature of this activity. Our results show that shifting attention towards an expected visual target results in a late-onset (600 ms postcue onset) depression of alpha activity which persists until the appearance of the target. Independent component analysis (ICA) and dipolar source modeling confirmed that the neuronal changes we observed originated from within the calcarine cortex. Our results further show that the amplitude changes in alpha activity were induced not evoked (i.e., not phase-locked to the cued attentional task). We argue that the decrease in alpha prior to the onset of the target may serve to prime the early visual cortex for incoming sensory information. We conclude that attentional shifts affect activity within the human calcarine cortex by altering the amplitude of spontaneous alpha rhythms and that subsequent modulation of visual input with attentional engagement follows as a consequence of these localized changes in oscillatory activity.

Mirror Neurons in a New World Monkey, Common Marmoset.

Mirror neurons respond when executing a motor act and when observing others' similar act. So far, mirror neurons have been found only in macaques, humans, and songbirds. To investigate the degree of phylogenetic specialization of mirror neurons during the course of their evolution, we determined whether mirror neurons with similar properties to macaques occur in a New World monkey, the common marmoset (Callithrix jacchus). The ventral premotor cortex (PMv), where mirror neurons have been reported in macaques, is difficult to identify in marmosets, since no sulcal landmarks exist in the frontal cortex. We addressed this problem using "in vivo" connection imaging methods. That is, we first identified cells responsive to others' grasping action in a clear landmark, the superior temporal sulcus (STS), under anesthesia, and injected fluorescent tracers into the region. By fluorescence stereomicroscopy, we identified clusters of labeled cells in the ventrolateral frontal cortex, which were confirmed to be within the ventrolateral frontal cortex including PMv after sacrifice. We next implanted electrodes into the ventrolateral frontal cortex and STS and recorded single/multi-units under an awake condition. As a result, we found neurons in the ventrolateral frontal cortex with characteristic "mirror" properties quite similar to those in macaques. This finding suggests that mirror neurons occur in a common ancestor of New and Old World monkeys and its common properties are preserved during the course of primate evolution.

Task-specific disruption of perceptual learning.

For more than a century, the process of stabilization has been a central issue in the research of learning and memory. Namely, after a skill or memory is acquired, it must be consolidated before it becomes resistant to disruption by subsequent learning. Although it is clear that there are many cases in which learning can be disrupted, it is unclear when learning something new disrupts what has already been learned. Herein, we provide two answers to this question with the demonstration that perceptual learning of a visual stimulus disrupts or interferes with the consolidation of a previously learned visual stimulus. In this study, we trained subjects on two different hyperacuity tasks and determined whether learning of the second task disrupted that of the first. We first show that disruption of learning occurs between visual stimuli presented at the same orientation in the same retinotopic location but not for the same stimuli presented at retinotopically disparate locations or different orientations at the same location. Second, we show that disruption from stimuli in the same retinotopic location is ameliorated if the subjects wait for 1 h before training on the second task. These results indicate that disruption, at least in visual learning, is specific to features of the tasks and that a temporal delay of 1 h can stabilize visual learning. This research shows that visual learning is susceptible to disruption and elucidates the processes by which the brain can consolidate learning and thus protect what is learned from being overwritten.

Attentional modulation of oscillatory activity in human visual cortex.

The effects of attentional modulation on activity within the human visual cortex were investigated using magnetoencephalography. Chromatic sinusoidal stimuli were used to evoke activity from the occipital cortex, with attention directed either toward or away from the stimulus using a bar-orientation judgment task. For five observers, global magnetic field power was plotted as a function of time from stimulus onset. The major peak of each function occurred at about 120 ms latency and was well modeled by a current dipole near the calcarine sulcus. Independent component analysis (ICA) on the non-averaged data for each observer also revealed one component of calcarine origin, the location of which matched that of the dipolar source determined from the averaged data. For two observers, ICA revealed a second component near the parieto-occipital sulcus. Although no effects of attention were evident using standard averaging procedures, time-varying spectral analyses of single trials revealed that the main effect of attention was to alter the level of oscillatory activity. Most notably, a sustained increase in alpha-band (7-12 Hz) activity of both calcarine and parieto-occipital origin was evident. In addition, calcarine activity in the range of 13-21 Hz was enhanced, while calcarine activity in the range of 5-6 Hz was reduced. Our results are consistent with the hypothesis that attentional modulation affects neural processing within the calcarine and parieto-occipital cortex by altering the amplitude of alpha-band activity and other natural brain rhythms.