Hierarchical categorization learning is associated with representational changes in the dorsal striatum and posterior frontal and parietal cortex.

Learning and recognition can be improved by sorting novel items into categories and subcategories. Such hierarchical categorization is easy when it can be performed according to learned rules (e.g., "if car, then automatic or stick shift" or "if boat, then motor or sail"). Here, we present results showing that human participants acquire categorization rules for new visual hierarchies rapidly, and that, as they do, corresponding hierarchical representations of the categorized stimuli emerge in patterns of neural activation in the dorsal striatum and in posterior frontal and parietal cortex. Participants learned to categorize novel visual objects into a hierarchy with superordinate and subordinate levels based on the objects' shape features, without having been told the categorization rules for doing so. On each trial, participants were asked to report the category and subcategory of the object, after which they received feedback about the correctness of their categorization responses. Participants trained over the course of a one-hour-long session while their brain activation was measured using functional magnetic resonance imaging. Over the course of training, significant hierarchy learning took place as participants discovered the nested categorization rules, as evidenced by the occurrence of a learning trial, after which performance suddenly increased. This learning was associated with increased representational strength of the newly acquired hierarchical rules in a corticostriatal network including the posterior frontal and parietal cortex and the dorsal striatum. We also found evidence suggesting that reinforcement learning in the dorsal striatum contributed to hierarchical rule learning.

The Architecture of Object-Based Attention.

The allocation of attention to objects raises several intriguing questions: What are objects, how does attention access them, what anatomical regions are involved? Here, we review recent progress in the field to determine the mechanisms underlying object-based attention. First, findings from unconscious priming and cueing suggest that the preattentive targets of object-based attention can be fully developed object representations that have reached the level of identity. Next, the control of object-based attention appears to come from ventral visual areas specialized in object analysis that project downward to early visual areas. How feedback from object areas can accurately target the object's specific locations and features is unknown but recent work in autoencoding has made this plausible. Finally, we suggest that the three classic modes of attention may not be as independent as is commonly considered, and instead could all rely on object-based attention. Specifically, studies show that attention can be allocated to the separated members of a group-without affecting the space between them-matching the defining property of feature-based attention. At the same time, object-based attention directed to a single small item has the properties of space-based attention. We outline the architecture of object-based attention, the novel predictions it brings, and discuss how it works in parallel with other attention pathways.

The knobby ball illusion.

A novel haptic illusion is described where deformations of the fingertip skin lead to subsequent misperceptions of an object's shape.

DeepAction: a MATLAB toolbox for automated classification of animal behavior in video.

The identification of animal behavior in video is a critical but time-consuming task in many areas of research. Here, we introduce DeepAction, a deep learning-based toolbox for automatically annotating animal behavior in video. Our approach uses features extracted from raw video frames by a pretrained convolutional neural network to train a recurrent neural network classifier. We evaluate the classifier on two benchmark rodent datasets and one octopus dataset. We show that it achieves high accuracy, requires little training data, and surpasses both human agreement and most comparable existing methods. We also create a confidence score for classifier output, and show that our method provides an accurate estimate of classifier performance and reduces the time required by human annotators to review and correct automatically-produced annotations. We release our system and accompanying annotation interface as an open-source MATLAB toolbox.

Endogenous attention biases transformational apparent motion based on high-level shape representations.

When two pre-existing, separated squares are connected by the sudden onset of a bar between them, viewers do not perceive the bar to appear all at once. Instead, they see an illusory morphing of the original squares over time. The direction of this transformational apparent motion (TAM) can be influenced by endogenous attention deployed before the appearance of the connecting bar. Here, we investigated whether the influence of endogenous attention on TAM results from operations over high-level feature-independent shape representations, or instead over lower level shape representations defined by specific visual features. To do so, we tested the influence of endogenous attention on TAM in first- and second-order displays, which shared common shapes but had different shape-defining attributes (luminance and texture contrast, respectively). In terms of both the magnitude of directional bias and timing, we found that endogenous attention exerted a similar influence on both first- and second-order objects. These results imply that endogenous attention biases the perceived direction of TAM by operating on high-level shape representations that are invariant to the low-level visual features that define them. Our results support a four-stage model of TAM, where a feature encoding stage passes a features-specific layout to a parsing stage that forms discrete, high-level meta-featural shapes, which are then matched and visually interpolated over time.

Transfer of Tactile Learning from Trained to Untrained Body Parts Supported by Cortical Coactivation in Primary Somatosensory Cortex.

A pioneering study by Volkmann (1858) revealed that training on a tactile discrimination task improved task performance, indicative of tactile learning, and that such tactile learning transferred from trained to untrained body parts. However, the neural mechanisms underlying tactile learning and transfer of tactile learning have remained unclear. We trained groups of human subjects (female and male) in daily sessions on a tactile discrimination task either by stimulating the palm of the right hand or the sole of the right foot. Task performance before training was similar between the palm and sole. Posttraining transfer of tactile learning was greater from the trained right sole to the untrained right palm than from the trained right palm to the untrained right sole. Functional magnetic resonance imaging (fMRI) and multivariate pattern classification analysis revealed that the somatotopic representation of the right palm in contralateral primary somatosensory cortex (SI) was coactivated during tactile stimulation of the right sole. More pronounced coactivation in the cortical representation of the right palm was associated with lower tactile performance for tactile stimulation of the right sole and more pronounced subsequent transfer of tactile learning from the trained right sole to the untrained right palm. In contrast, coactivation of the cortical sole representation during tactile stimulation of the palm was less pronounced and no association with tactile performance and subsequent transfer of tactile learning was found. These results indicate that tactile learning may transfer to untrained body parts that are coactivated to support tactile learning with the trained body part.SIGNIFICANCE STATEMENT Perceptual skills such as the discrimination of tactile cues can improve by means of training, indicative of perceptual learning and sensory plasticity. However, it has remained unclear whether and if so, how such perceptual learning can occur if the training task is very difficult. Here, we show for tactile perceptual learning that the representation of the palm of the hand in primary somatosensory cortex (SI) is coactivated to support learning of a difficult tactile discrimination task with tactile stimulation of the sole of the foot. Such cortical coactivation of an untrained body part to support tactile learning with a trained body part might be critically involved in the subsequent transfer of tactile learning between the trained and untrained body parts.

Smooth pursuit operates over perceived not physical positions of the double-drift stimulus.

The double-drift illusion produces a large deviation in perceived direction that strongly dissociates physical position from perceived position. Surprisingly, saccades do not seem to be affected by the illusion (Lisi & Cavanagh, 2015). When targeting a double-drift stimulus, the saccade system is driven by retinal rather than perceived position. Here, using paired double-drift targets, we test whether the smooth pursuit system is driven by perceived or physical position. Participants (n = 7) smoothly pursued the inferred midpoint (Steinbach, 1976) between two horizontally aligned Gabor patches that were separated by 20° and moving on parallel, oblique paths. On the first half of each trial, the Gabors' internal textures were static while both drifted obliquely downward. On the second half of each trial, while the envelope moved obliquely upward, the internal texture drifted orthogonally to the envelope's motion, producing a large perceived deviation from the downward path even though the upward and downward trajectories always followed the same physical path but in opposite directions. We find that smooth pursuit eye movements accurately followed the nonillusory downward path of the midpoint between the two Gabors, but then followed the illusory rather than the physical trajectory on the upward return. Thus, virtual targets for smooth pursuit are derived from perceived rather than retinal coordinates.

Effects of internal and external velocity on the perceived direction of the double-drift illusion.

In the double-drift illusion, the combination of the internal and external motion vectors produces large misperceptions of both position and direction of motion. Here, we investigate the role that speed plays in determining how these two sources of motion are combined to produce the double-drift illusion. To address this question, we measure the size of the illusion at seven internal speeds combined with six external speeds. We find that the illusion increases with increasing internal speed and decreases with increasing external speed. We model this by combining the external and internal vectors to produce the resulting, illusory direction (Tse & Hsieh, 2006). The relative effect of the two vectors is specified by a constant K in this model and the data reveal that K decreases linearly as external speed increases. This critical role of external speed in modulating the vector combination uncovers new details about how the visual system combines different sources of motion information to produce a global motion percept.

A position anchor sinks the double-drift illusion.

When the internal texture of a Gabor patch moves orthogonally to its envelope's motion, the perceived path, viewed in the periphery, shifts dramatically in position, and direction relative to the true path (the double-drift illusion). Here, we examine positional uncertainty as a critical factor underlying this illusory shift. We presented participants with an anchoring line at different distances from the drifting Gabor's physical path. Our results indicate that placing an anchor (a fixed line) close to the Gabor's path halved the magnitude of the illusion. This suppression was symmetrical for anchors placed on either side of the Gabor. In a second experiment, we used crowding to degrade the anchoring line's position information by embedding it in a set of parallel lines. In this case, despite the presence of the same lines that reduced the illusion when presented in isolation, the illusory shift was now largely restored. We suggest that the adjacent lines crowded each other, reducing their positional certainty, and thus their ability to anchor the location of the moving Gabor. These findings indicate that the positional uncertainty of the equiluminant Gabor patch is critical for the illusory position offset.

The parahippocampal place area and hippocampus encode the spatial significance of landmark objects.

Landmark objects are points of reference that can anchor one's internal cognitive map to the external world while navigating. They are especially useful in indoor environments where other cues such as spatial geometries are often similar across locations. We used functional magnetic resonance imaging (fMRI) and multivariate pattern analysis (MVPA) to understand how the spatial significance of landmark objects is represented in the human brain. Participants learned the spatial layout of a virtual building with arbitrary objects as unique landmarks in each room during a navigation task. They were scanned while viewing the objects before and after learning. MVPA revealed that the neural representation of landmark objects in the right parahippocampal place area (rPPA) and the hippocampus transformed systematically according to their locations. Specifically, objects in different rooms became more distinguishable than objects in the same room. These results demonstrate that rPPA and the hippocampus encode the spatial significance of landmark objects in indoor spaces.

Visual Attention Modulates Glutamate-Glutamine Levels in Vestibular Cortex: Evidence from Magnetic Resonance Spectroscopy.

Attending to a stimulus enhances the neuronal responses to it, while responses to nonattended stimuli are not enhanced and may even be suppressed. Although the neural mechanisms of response enhancement for attended stimuli have been intensely studied, the neural mechanisms underlying attentional suppression remain largely unknown. It is uncertain whether attention acts to suppress the processing in sensory cortical areas that would otherwise process the nonattended stimulus or the subcortical input to these cortical areas. Moreover, the neurochemical mechanisms inducing a reduction or suppression of neuronal responses to nonattended stimuli are as yet unknown. Here, we investigated how attention directed toward visual processing cross-modally acts to suppress vestibular responses in the human brain. By using functional magnetic resonance spectroscopy in a group of female and male subjects, we find that attention to visual motion downregulates in a load-dependent manner the concentration of excitatory neurotransmitter (glutamate and its precursor glutamine, referred to together as Glx) within the parietoinsular vestibular cortex (PIVC), a core cortical area of the vestibular system, while leaving the concentration of inhibitory neurotransmitter (GABA) in PIVC unchanged. This makes PIVC less responsive to excitatory thalamic vestibular input, as corroborated by functional magnetic resonance imaging. Together, our results suggest that attention acts to suppress the processing of nonattended sensory cues cortically by neurochemically rendering the core cortical area of the nonattended sensory modality less responsive to excitatory thalamic input.SIGNIFICANCE STATEMENT Here, we address a fundamental problem that has eluded attention research for decades, namely, how the brain ignores irrelevant stimuli. To date, three classes of solutions to this problem have been proposed: (1) enhancement of GABAergic interneuron activity in cortex, (2) downregulation of glutamatergic cell activity in cortex; and (3) downregulation of neural activity in thalamic projection areas, which would then provide the cortex with less input. Here, we use magnetic resonance spectroscopy in humans and find support for the second hypothesis, implying that attention to one sensory modality involves the suppression of irrelevant stimuli of another sensory modality by downregulating glutamate in the cortex.

Attentional tracking takes place over perceived rather than veridical positions.

Illusions can induce striking differences between perception and retinal input. For instance, a static Gabor with a moving internal texture appears to be shifted in the direction of its internal motion, a shift that increases dramatically when the Gabor itself is also in motion. Here, we ask whether attention operates on the perceptual or physical location of this stimulus. To do so, we generated an attentional tracking task where participants (N = 15) had to keep track of a single target among three Gabors that rotated around a common center in the periphery. During tracking, the illusion was used to make three Gabors appear either shifted away from or toward one another while maintaining the same physical separation. Because tracking performance depends in part on target to distractor spacing, if attention selects targets from perceived positions, performance should be better when the Gabors appear further apart and worse when they appear closer together. We find that tracking performance is superior with greater perceived separation, implying that attentional tracking operates over perceived rather than physical positions.

Abutting Objects Warp the Three-Dimensional Curvature of Modally Completing Surfaces.

Binocular disparity can give rise to the perception of open surfaces or closed curved surfaces (volumes) that appear to vary smoothly across discrete depths. Here I build on my recent papers by providing examples where modally completing surfaces not only fill in from one depth layer's visible contours to another layer's visible contours within virtual contours in an analog manner, but where modally completing surface curvature is altered by the interpolation of an abutting object perceived to be connected to or embedded within that modally completing surface. Seemingly minor changes in such an abutting object can flip the interpretation of distal regions, for example, turning a distant edge (where a surface ends) into rim (where a surface bends to occlude itself) or turning an open surface into a closed one. In general, the interpolated modal surface appears to deform, warp, or bend in three-dimensions to accommodate the abutting object. These demonstrations cannot be easily explained by existing models of visual processing or modal completion and drive home the implausibility of localistic accounts of modal or amodal completion that are based, for example, solely on extending contours in space until they meet behind an occluder or in front of "pacmen." These demonstrations place new constraints on the holistic surface and volume generation processes that construct our experience of a three-dimensional world of surfaces and objects under normal viewing conditions.

Pop-out for illusory rather than veridical trajectories with double-drift stimuli.

If a patch of texture drifts in one direction while its internal texture drifts in the orthogonal direction, the perceived direction of this double-drift stimulus (also known as the infinite regress and curveball illusions) deviates strongly from its physical direction. Here, we use double-drift stimuli to construct two types of search arrays: The first had an oddball target in terms of the physical trajectories, but no oddball for the perceived trajectory, whereas the second had a perceptual oddball, but no physical oddball. We used these two arrays to determine whether pop-out operates over physical or perceived trajectories. Participants reported the location of the odd double-drift stimulus that had either a unique physical or perceived trajectory in a set of four or eight items. When the distractors all shared one perceived trajectory, but the target had an odd perceived trajectory, it popped out even though the physical trajectories of the stimuli were mixed: Accuracy rates were at ceiling, and response times decreased with increasing set size. In contrast, participants were significantly less accurate and slower at finding the physical oddball when all the paths had a common perceived trajectory. Moreover, responses became less accurate and slower with increasing set size. Our findings suggest that, at least for this type of stimulus, perceptual features can be processed rapidly, whereas the search for physical features is very inefficient.

Apparent Motion Is Computed in Perceptual Coordinates.

When a Gabor moves in one direction in the visual periphery while its internal texture moves in the orthogonal direction, its perceived direction can deviate from its physical direction by as much as 45° or more. Lisi et al. showed that immediate saccades go to the physical location of double-drift targets, whereas delayed saccades primarily go to their perceived locations. Here, we investigated whether the apparent motion seen from the offset of a double-drift stimulus to the onset of a later target probe originates from the perceived or physical location of the double-drift stimulus. We find that apparent motion proceeds away from the perceived position of the double-drift stimulus at all temporal delays. This suggests that apparent motion is computed in perceptual rather than retinotopic coordinates.

The vector combination underlying the double-drift illusion is based on motion in world coordinates: Evidence from smooth pursuit.

If a Gabor pattern drifts in one direction while its internal texture drifts in the orthogonal direction, its perceived direction deviates strongly from its true direction and is instead some combination of its real external motion and its internal motion (Tse & Hsieh, 2006). In the first experiment, we confirm that, for the stimuli used in our experiment, the direction shifts on a gray background were explained by a vector combination of the internal and external motions whereas for the Gabor on a black background, we find no illusory shifts. These results suggest that the internal motion contributes to the perceived direction but only when the Gabor's positional uncertainty is high. Next, we test whether the vector combination is based on motions on the retina or motions in the world. When participants track a fixation point that moves in tandem with the Gabor, keeping it roughly stable on the retina, the illusion is undiminished. This finding indicates that the vector combination of internal and external motion that produces the double-drift illusion must happen after the eye movement signals have been factored into the stimulus motions to recover motions in the world, in particular, in areas V3A, V6, MSTd, and VIP.

Neural Correlates of the Conscious Perception of Visual Location Lie Outside Visual Cortex.

When perception differs from the physical stimulus, as it does for visual illusions and binocular rivalry, the opportunity arises to localize where perception emerges in the visual processing hierarchy. Representations prior to that stage differ from the eventual conscious percept even though they provide input to it. Here, we investigate where and how a remarkable misperception of position emerges in the brain. This "double-drift" illusion causes a dramatic mismatch between retinal and perceived location, producing a perceived motion path that can differ from its physical path by 45° or more. The deviations in the perceived trajectory can accumulate over at least a second, whereas other motion-induced position shifts accumulate over 80-100 ms before saturating. Using fMRI and multivariate pattern analysis, we find that the illusory path does not share activity patterns with a matched physical path in any early visual areas. In contrast, a whole-brain searchlight analysis reveals a shared representation in anterior regions of the brain. These higher-order areas would have the longer time constants required to accumulate the small moment-to-moment position offsets that presumably originate in early visual cortical areas and then transform these sensory inputs into a final conscious percept. The dissociation between perception and the activity in early sensory cortex suggests that consciously perceived position does not emerge in what is traditionally regarded as the visual system but instead emerges at a higher level.

Learning efficient visual search for stimuli containing diagnostic spatial configurations and color-shape conjunctions.

Visual search is often slow and difficult for complex stimuli such as feature conjunctions. Search efficiency, however, can improve with training. Search for stimuli that can be identified by the spatial configuration of two elements (e.g., the relative position of two colored shapes) improves dramatically within a few hundred trials of practice. Several recent imaging studies have identified neural correlates of this learning, but it remains unclear what stimulus properties participants learn to use to search efficiently. Influential models, such as reverse hierarchy theory, propose two major possibilities: learning to use information contained in low-level image statistics (e.g., single features at particular retinotopic locations) or in high-level characteristics (e.g., feature conjunctions) of the task-relevant stimuli. In a series of experiments, we tested these two hypotheses, which make different predictions about the effect of various stimulus manipulations after training. We find relatively small effects of manipulating low-level properties of the stimuli (e.g., changing their retinotopic location) and some conjunctive properties (e.g., color-position), whereas the effects of manipulating other conjunctive properties (e.g., color-shape) are larger. Overall, the findings suggest conjunction learning involving such stimuli might be an emergent phenomenon that reflects multiple different learning processes, each of which capitalizes on different types of information contained in the stimuli. We also show that both targets and distractors are learned, and that reversing learned target and distractor identities impairs performance. This suggests that participants do not merely learn to discriminate target and distractor stimuli, they also learn stimulus identity mappings that contribute to performance improvements.

Meridian interference reveals neural locus of motion-induced position shifts.

When a Gabor patch moves along a path in one direction while its internal texture drifts orthogonally to this path, it can appear to deviate from its physical path by 45° or more. This double-drift illusion is different from other motion-induced position shift effects in several ways: it has an integration period of over a second; the illusory displacement that accumulates over a second or more is orthogonal to rather than along the motion path; the perceptual deviations are much larger; and they have little or no effect on eye movements to the target. In this study we investigated the underlying neural mechanisms of the motion integration and position processing for this double-drift stimulus by testing possible anatomical constraints on its magnitude. We found that the illusion was reduced at the vertical and horizontal meridians when the perceptual path would cross or be driven toward the meridian, but not at other locations or other motion directions. The disruption of the accumulation of the position error at both the horizontal and vertical meridians suggests a central role of quadrantic areas in the generation of this type of motion-induced position shift. NEW & NOTEWORTHY The remarkably strong double-drift illusion is disrupted at both the vertical and horizontal meridians. We propose that this finding is the behavioral consequence of the anatomical gaps at both meridians, suggesting that neural areas with quadrantic representations (e.g., V2, V3) are the initial locus of this motion-induced position shift. This result rules out V1 as the source of the illusion because it has an anatomical break only at the vertical meridian.

Long Time No See: Enduring Behavioral and Neuronal Changes in Perceptual Learning of Motion Trajectories 3 Years After Training.

Here, we report on the long-term stability of changes in behavior and brain activity following perceptual learning of conjunctions of simple motion features. Participants were trained for 3 weeks on a visual search task involving the detection of a dot moving in a "v"-shaped target trajectory among inverted "v"-shaped distractor trajectories. The first and last training sessions were carried out during functional magnetic resonance imaging (fMRI). Learning stability was again examined behaviorally and using fMRI 3 years after the end of training. Results show that acquired behavioral improvements were remarkably stable over time and that these changes were specific to trained target and distractor trajectories. A similar pattern was observed on the neuronal level, when the representation of target and distractor stimuli was examined in early retinotopic visual cortex (V1-V3): training enhanced activity for the target relative to the surrounding distractors in the search array and this enhancement persisted after 3 years. However, exchanging target and distractor trajectories abolished both neuronal and behavioral effects, suggesting that training-induced changes in stimulus representation are specific to trained stimulus identities.