Convolutional neural network models applied to neuronal responses in macaque V1 reveal limited nonlinear processing.

Computational models of the primary visual cortex (V1) have suggested that V1 neurons behave like Gabor filters followed by simple nonlinearities. However, recent work employing convolutional neural network (CNN) models has suggested that V1 relies on far more nonlinear computations than previously thought. Specifically, unit responses in an intermediate layer of VGG-19 were found to best predict macaque V1 responses to thousands of natural and synthetic images. Here, we evaluated the hypothesis that the poor performance of lower layer units in VGG-19 might be attributable to their small receptive field size rather than to their lack of complexity per se. We compared VGG-19 with AlexNet, which has much larger receptive fields in its lower layers. Whereas the best-performing layer of VGG-19 occurred after seven nonlinear steps, the first convolutional layer of AlexNet best predicted V1 responses. Although the predictive accuracy of VGG-19 was somewhat better than that of standard AlexNet, we found that a modified version of AlexNet could match the performance of VGG-19 after only a few nonlinear computations. Control analyses revealed that decreasing the size of the input images caused the best-performing layer of VGG-19 to shift to a lower layer, consistent with the hypothesis that the relationship between image size and receptive field size can strongly affect model performance. We conducted additional analyses using a Gabor pyramid model to test for nonlinear contributions of normalization and contrast saturation. Overall, our findings suggest that the feedforward responses of V1 neurons can be well explained by assuming only a few nonlinear processing stages.

Morality or competence is more important? The effect of evaluation dimensions on ERP responses to neutral faces depends on contextual valence and self-relevance.

Recent studies have shown that the processing of neutral facial expressions could be modulated by the valence and self-relevance of preceding verbal evaluations. However, these studies have not distinguished the dimension (i.e., morality and competence) from verbal evaluations. In fact, there is a hot controversy about whether morality or competence receives more weight. Therefore, using the ERP technique, the current study aimed to address this issue by comparing the influence of morality and competence evaluations on behavioral and neural responses to neutral facial expressions when these evaluations varied with contextual valence and self-relevance. Our ERP results revealed that the early EPN amplitudes were larger for neutral faces after receiving evaluations about self relative to evaluations about senders. Moreover, the EPN was more negative after a competence evaluation relative to a morality evaluation when these evaluations were positive, while this effect was absent when these evaluations were negative. The late LPP was larger after a morality evaluation compared to a competence evaluation when these evaluations were negative and directed to self. However, no significant LPP effect between morality and competence evaluations was observed when these evaluations were positive. The present study extended previous studies by showing that early and late processing stages of faces are affected by the evaluation dimension in a top-down manner and further modulated by contextual valence and self-relevance.

Neural correlates of individual differences in multimodal emotion recognition ability.

Studies have reported substantial variability in emotion recognition ability (ERA) - an important social skill - but possible neural underpinnings for such individual differences are not well understood. This functional magnetic resonance imaging (fMRI) study investigated neural responses during emotion recognition in young adults (N = 49) who were selected for inclusion based on their performance (high or low) during previous testing of ERA. Participants were asked to judge brief video recordings in a forced-choice emotion recognition task, wherein stimuli were presented in visual, auditory and multimodal (audiovisual) blocks. Emotion recognition rates during brain scanning confirmed that individuals with high (vs low) ERA received higher accuracy for all presentation blocks. fMRI-analyses focused on key regions of interest (ROIs) involved in the processing of multimodal emotion expressions, based on previous meta-analyses. In neural response to emotional stimuli contrasted with neutral stimuli, individuals with high (vs low) ERA showed higher activation in the following ROIs during the multimodal condition: right middle superior temporal gyrus (mSTG), right posterior superior temporal sulcus (PSTS), and right inferior frontal cortex (IFC). Overall, results suggest that individual variability in ERA may be reflected across several stages of decisional processing, including extraction (mSTG), integration (PSTS) and evaluation (IFC) of emotional information.

When experience with scenes foils attentional orienting: ERP evidence against flexible target-context mapping in visual search.

Visual search is speeded when a target is repeatedly presented in an invariant scene context of nontargets (contextual cueing), demonstrating observers' capability for using statistical long-term memory (LTM) to make predictions about upcoming sensory events, thus improving attentional orienting. In the current study, we investigated whether expectations arising from individual, learned environmental structures can encompass multiple target locations. We recorded event-related potentials (ERPs) while participants performed a contextual cueing search task with repeated and non-repeated spatial item configurations. Notably, a given search display could be associated with either a single target location (standard contextual cueing) or two possible target locations. Our result showed that LTM-guided attention was always limited to only one target position in single- but also in the dual-target displays, as evidenced by expedited reaction times (RTs) and enhanced N1pc and N2pc deflections contralateral to one ("dominant") target of up to two repeating target locations. This contrasts with the processing of non-learned ("minor") target positions (in dual-target displays), which revealed slowed RTs alongside an initial N1pc "misguidance" signal that then vanished in the subsequent N2pc. This RT slowing was accompanied by enhanced N200 and N400 waveforms over fronto-central electrodes, suggesting that control mechanisms regulate the competition between dominant and minor targets. Our study thus reveals a dissociation in processing dominant versus minor targets: While LTM templates guide attention to dominant targets, minor targets necessitate control processes to overcome the automatic bias towards previously learned, dominant target locations.

Auditory change detection and visual selective attention: association between MMN and N2pc.

While the auditory and visual systems each provide distinct information to our brain, they also work together to process and prioritize input to address ever-changing conditions. Previous studies highlighted the trade-off between auditory change detection and visual selective attention; however, the relationship between them is still unclear. Here, we recorded electroencephalography signals from 106 healthy adults in three experiments. Our findings revealed a positive correlation at the population level between the amplitudes of event-related potential indices associated with auditory change detection (mismatch negativity) and visual selective attention (posterior contralateral N2) when elicited in separate tasks. This correlation persisted even when participants performed a visual task while disregarding simultaneous auditory stimuli. Interestingly, as visual attention demand increased, participants whose posterior contralateral N2 amplitude increased the most exhibited the largest reduction in mismatch negativity, suggesting a within-subject trade-off between the two processes. Taken together, our results suggest an intimate relationship and potential shared mechanism between auditory change detection and visual selective attention. We liken this to a total capacity limit that varies between individuals, which could drive correlated individual differences in auditory change detection and visual selective attention, and also within-subject competition between the two, with task-based modulation of visual attention causing within-participant decrease in auditory change detection sensitivity.

Eye movement intervention facilitates concurrent perception and memory processing.

A widely used psychotherapeutic treatment for post-traumatic stress disorder (PTSD) involves performing bilateral eye movement (EM) during trauma memory retrieval. However, how this treatment-described as eye movement desensitization and reprocessing (EMDR)-alleviates trauma-related symptoms is unclear. While conventional theories suggest that bilateral EM interferes with concurrently retrieved trauma memories by taxing the limited working memory resources, here, we propose that bilateral EM actually facilitates information processing. In two EEG experiments, we replicated the bilateral EM procedure of EMDR, having participants engaging in continuous bilateral EM or receiving bilateral sensory stimulation (BS) as a control while retrieving short- or long-term memory. During EM or BS, we presented bystander images or memory cues to probe neural representations of perceptual and memory information. Multivariate pattern analysis of the EEG signals revealed that bilateral EM enhanced neural representations of simultaneously processed perceptual and memory information. This enhancement was accompanied by heightened visual responses and increased neural excitability in the occipital region. Furthermore, bilateral EM increased information transmission from the occipital to the frontoparietal region, indicating facilitated information transition from low-level perceptual representation to high-level memory representation. These findings argue for theories that emphasize information facilitation rather than disruption in the EMDR treatment.

The influence of electrode types to the visually induced gamma oscillations in mouse primary visual cortex.

The local field potential (LFP) is an extracellular electrical signal associated with neural ensemble input and dendritic signaling. Previous studies have linked gamma band oscillations of the LFP in cortical circuits to sensory stimuli encoding, attention, memory, and perception. Inconsistent results regarding gamma tuning for visual features were reported, but it remains unclear whether these discrepancies are due to variations in electrode properties. Specifically, the surface area and impedance of the electrode are important characteristics in LFP recording. To comprehensively address these issues, we conducted an electrophysiological study in the V1 region of lightly anesthetized mice using two types of electrodes: one with higher impedance (1 MΩ) and a sharp tip (10 µm), while the other had lower impedance (100 KΩ) but a thicker tip (200 µm). Our findings demonstrate that gamma oscillations acquired by sharp-tip electrodes were significantly stronger than those obtained from thick-tip electrodes. Regarding size tuning, most gamma power exhibited surround suppression at larger gratings when recorded from sharp-tip electrodes. However, the majority showed enhanced gamma power at larger gratings when recorded from thick-tip electrodes. Therefore, our study suggests that microelectrode parameters play a significant role in accurately recording gamma oscillations and responsive tuning to sensory stimuli.

Oculomotor routines for perceptual judgments.

In everyday life we frequently make simple visual judgments about object properties, for example, how big or wide is a certain object? Our goal is to test whether there are also task-specific oculomotor routines that support perceptual judgments, similar to the well-established exploratory routines for haptic perception. In a first study, observers saw different scenes with two objects presented in a photorealistic virtual reality environment. Observers were asked to judge which of two objects was taller or wider while gaze was tracked. All tasks were performed with the same set of virtual objects in the same scenes, so that we can compare spatial characteristics of exploratory gaze behavior to quantify oculomotor routines for each task. Width judgments showed fixations around the center of the objects with larger horizontal spread. In contrast, for height judgments, gaze was shifted toward the top of the objects with larger vertical spread. These results suggest specific strategies in gaze behavior that presumably are used for perceptual judgments. To test the causal link between oculomotor behavior and perception, in a second study, observers could freely gaze at the object or we introduced a gaze-contingent setup forcing observers to fixate specific positions on the object. Discrimination performance was similar between free-gaze and the gaze-contingent conditions for width and height judgments. These results suggest that although gaze is adapted for different tasks, performance seems to be based on a perceptual strategy, independent of potential cues that can be provided by the oculomotor system.

Effect of spatial context on perceived walking direction.

Contextual modulation occurs for many aspects of high-level vision but is relatively unexplored for the perception of walking direction. In a recent study, we observed an effect of the temporal context on perceived walking direction. Here, we examined the spatial contextual modulation by measuring the perceived direction of a target point-light walker in the presence of two flanker walkers, one on each side. Experiment 1 followed a within-subjects design. Participants (n = 30) completed a spatial context task by judging the walking direction of the target in 13 different conditions: a walker alone in the center or with two flanking walkers either intact or scrambled at a flanker deviation of ±15°, ±30°, or ±45°. For comparison, participants completed an adaptation task where they reported the walking direction of a target after adaptation to ±30° walking direction. We found the expected repulsive effects in the adaptation task but attractive effects in the spatial context task. In Experiment 2 (n = 40), we measured the tuning of spatial contextual modulation across a wide range of flanker deviation magnitudes ranging from 15° to 165° in 15° intervals. Our results showed significant attractive effects across a wide range of flanker walking directions with the peak effect at around 30°. The assimilative versus repulsive effects of spatial contextual modulation and temporal adaptation suggest dissociable neural mechanisms, but they may operate on the same population of sensory channels coding for walking direction, as evidenced by similarity in the peak tuning across the walking direction of the inducers.

Multiple object tracking in the presence of a goal: Attentional anticipation and suppression.

In previous studies, we found that tracking multiple objects involves anticipatory attention, especially in the linear direction, even when a target bounced against a wall. We also showed that active involvement, in which the wall was replaced by a controllable paddle, resulted in increased allocation of attention to the bounce direction. In the current experiments, we wanted to further investigate the potential influence of the valence of the heading of an object. In Experiments 1 and 2, participants were instructed to catch targets with a movable goal. In Experiment 3, participants were instructed to manipulate the permeability of a static wall in order to let targets either approach goals (i.e., green goals) or avoid goals (i.e., red goals). The results of Experiment 1 showed that probe detection ahead of a target that moved in the direction of the goal was higher as compared to probe detection in the direction of a no-goal area. Experiment 2 provided further evidence that the attentional highlighting found in the first experiment depends on the movement direction toward the goal. In Experiment 3, we found that not so much the positive (or neutral) valence (here, the green and no-goal areas) led to increased allocation of attention but rather a negative valence (here the red goals) led to a decreased allocation of attention.

Serial dependence requires visual awareness: Evidence from continuous flash suppression.

The visual system often undergoes a relatively stable perception even in a noisy visual environment. This crucial function was reflected in a visual perception phenomenon-serial dependence, in which recent stimulus history systematically biases current visual decisions. Although serial dependence effects have been revealed in numerous studies, few studies examined whether serial dependence would require visual awareness. By using the continuous flash suppression (CFS) technique to render grating stimuli invisible, we investigated whether serial dependence effects could emerge at the unconscious levels. In an orientation adjustment task, subjects viewed a randomly oriented grating and reported their orientation perception via an adjustment response. Subjects performed a series of three type trial pairs. The first two trial pairs, in which subjects were instructed to make a response or no response toward the first trial of the pairs, respectively, were used to measure serial dependence at the conscious levels; the third trial pair, in which the grating stimulus in the first trial of the pair was masked by a CFS stimulus, was used to measure the serial dependence at the unconscious levels. One-back serial dependence effects for the second trial of the pairs were evaluated. We found significant serial dependence effects at the conscious levels, whether absence (Experiment 1) or presence (Experiment 2) of CFS stimuli, but failed to find the effects at the unconscious levels, corroborating the view that serial dependence requires visual awareness.

Perceptual dimensions of wood materials.

Materials exhibit an extraordinary range of visual appearances. Characterizing and quantifying appearance is important not only for basic research on perceptual mechanisms but also for computer graphics and a wide range of industrial applications. Although methods exist for capturing and representing the optical properties of materials and how they vary across surfaces (Haindl & Filip, 2013), the representations are typically very high-dimensional, and how these representations relate to subjective perceptual impressions of material appearance remains poorly understood. Here, we used a data-driven approach to characterizing the perceived appearance characteristics of 30 samples of wood veneer using a "visual fingerprint" that describes each sample as a multidimensional feature vector, with each dimension capturing a different aspect of the appearance. Fifty-six crowd-sourced participants viewed triplets of movies depicting different wood samples as the sample rotated. Their task was to report which of the two match samples was subjectively most similar to the test sample. In another online experiment, 45 participants rated 10 wood-related appearance characteristics for each of the samples. The results reveal a consistent embedding of the samples across both experiments and a set of nine perceptual dimensions capturing aspects including the roughness, directionality, and spatial scale of the surface patterns. We also showed that a weighted linear combination of 11 image statistics, inspired by the rating characteristics, predicts perceptual dimensions well.

Multidimensional neural representations of social features during movie viewing.

The social world is dynamic and contextually embedded. Yet, most studies utilize simple stimuli that do not capture the complexity of everyday social episodes. To address this, we implemented a movie viewing paradigm and investigated how everyday social episodes are processed in the brain. Participants watched one of two movies during an MRI scan. Neural patterns from brain regions involved in social perception, mentalization, action observation and sensory processing were extracted. Representational similarity analysis results revealed that several labeled social features (including social interaction, mentalization, the actions of others, characters talking about themselves, talking about others and talking about objects) were represented in the superior temporal gyrus (STG) and middle temporal gyrus (MTG). The mentalization feature was also represented throughout the theory of mind network, and characters talking about others engaged the temporoparietal junction (TPJ), suggesting that listeners may spontaneously infer the mental state of those being talked about. In contrast, we did not observe the action representations in the frontoparietal regions of the action observation network. The current findings indicate that STG and MTG serve as key regions for social processing, and that listening to characters talk about others elicits spontaneous mental state inference in TPJ during natural movie viewing.

Non-invasive electrical brain stimulation modulates human conscious perception of mental representation.

Mental representation is a key concept in cognitive science; nevertheless, its neural foundations remain elusive. We employed non-invasive electrical brain stimulation and functional magnetic resonance imaging to address this. During this process, participants perceived flickering red and green visual stimuli, discerning them either as distinct, non-fused colours or as a mentally generated, fused colour (orange). The application of transcranial alternating current stimulation to the medial prefrontal region (a key node of the default-mode network) suppressed haemodynamic activation in higher-order subthalamic and central executive networks associated with the perception of fused colours. This implies that higher-order thalamocortical and default-mode networks are crucial in humans' conscious perception of mental representation.

Toward a push-pull strategy against invasive snails using chemical and visual stimuli.

Four invasive Mediterranean snails, i.e., Theba pisana (Müller, 1774), Cernuella virgata (da Costa, 1778), Cochlicella acuta (Müller, 1774) and Cochlicella barbara (Linnaeus, 1758) cost $170 million yearly to the grain industry in Australia. Their impact is mainly due to their estivation behavior: snails climb on cereal and legume stalks to rest during summer, which coincides with harvest, causing grain contamination issues in crops such as wheat, barley and canola. Diverse management methods have been developed to regulate snail populations, with limited success. Our study investigates the potential for a push-pull strategy to divert invasive snails from cultivated fields. A "push" part (i.e. using a repellent stimuli) was based on the use of a chemical deterrent repelling snails from the cultivated field, and a "pull" part (i.e. using an attractive stimuli) was based on offering attractive estivation supports for snails to aggregate outside the cultivated field. First, artificial estivation supports of different colors were tested under laboratory and field conditions and showed that red supports were the most attractive for these snails. Second, different substances were tested as potential snail deterrents (garlic, coffee, coffee grounds, copper). Garlic extracts were the most powerful snail deterrent and were shown to effectively protect an estivation support and food source from snails under laboratory conditions. These results, which were highly consistent for the four species, illustrate the potential of a push-pull strategy against invasive snails in Australia. It is the first attempt to develop a push-pull strategy relying on both visual and chemical stimuli to achieve results, as well as manipulating the estivation behavior of a pest.

Learning to Choose: Behavioral Dynamics Underlying the Initial Acquisition of Decision-Making.

Current theories of decision-making propose that decisions arise through competition between choice options. Computational models of the decision process estimate how quickly information about choice options is integrated and how much information is needed to trigger a choice. Experiments using this approach typically report data from well-trained participants. As such, we do not know how the decision process evolves as a decision-making task is learned for the first time. To address this gap, we used a behavioral design separating learning the value of choice options from learning to make choices. We trained male rats to respond to single visual stimuli with different reward values. Then, we trained them to make choices between pairs of stimuli. Initially, the rats responded more slowly when presented with choices. However, as they gained experience in making choices, this slowing reduced. Response slowing on choice trials persisted throughout the testing period. We found that it was specifically associated with increased exponential variability when the rats chose the higher value stimulus. Additionally, our analysis using drift diffusion modeling revealed that the rats required less information to make choices over time. These reductions in the decision threshold occurred after just a single session of choice learning. These findings provide new insights into the learning process of decision-making tasks. They suggest that the value of choice options and the ability to make choices are learned separately and that experience plays a crucial role in improving decision-making performance.

Virtual dynamic interaction games reveal impaired multisensory integration in women with migraine.

OBJECTIVE: In this cross-sectional observational study, we aimed to investigate sensory profiles and multisensory integration processes in women with migraine using virtual dynamic interaction systems. BACKGROUND: Compared to studies on unimodal sensory processing, fewer studies show that multisensory integration differs in patients with migraine. Multisensory integration of visual, auditory, verbal, and haptic modalities has not been evaluated in migraine. METHODS: A 12-min virtual dynamic interaction game consisting of four parts was played by the participants. During the game, the participants were exposed to either visual stimuli only or multisensory stimuli in which auditory, verbal, and haptic stimuli were added to the visual stimuli. A total of 78 women participants (28 with migraine without aura and 50 healthy controls) were enrolled in this prospective exploratory study. Patients with migraine and healthy participants who met the inclusion criteria were randomized separately into visual and multisensory groups: Migraine multisensory (14 adults), migraine visual (14 adults), healthy multisensory (25 adults), and healthy visual (25 adults). The Sensory Profile Questionnaire was utilized to assess the participants' sensory profiles. The game scores and survey results were analyzed. RESULTS: In visual stimulus, the gaming performance scores of patients with migraine without aura were similar to the healthy controls, at a median (interquartile range [IQR]) of 81.8 (79.5-85.8) and 80.9 (77.1-84.2) (p = 0.149). Error rate of visual stimulus in patients with migraine without aura were comparable to healthy controls, at a median (IQR) of 0.11 (0.08-0.13) and 0.12 (0.10-0.14), respectively (p = 0,166). In multisensory stimulation, average gaming score was lower in patients with migraine without aura compared to healthy individuals (median [IQR] 82.2 [78.8-86.3] vs. 78.6 [74.0-82.4], p = 0.028). In women with migraine, exposure to new sensory modality upon visual stimuli in the fourth, seventh, and tenth rounds (median [IQR] 78.1 [74.1-82.0], 79.7 [77.2-82.5], 76.5 [70.2-82.1]) exhibited lower game scores compared to visual stimuli only (median [IQR] 82.3 [77.9-87.8], 84.2 [79.7-85.6], 80.8 [79.0-85.7], p = 0.044, p = 0.049, p = 0.016). According to the Sensory Profile Questionnaire results, sensory sensitivity, and sensory avoidance scores of patients with migraine (median [IQR] score 45.5 [41.0-54.7] and 47.0 [41.5-51.7]) were significantly higher than healthy participants (median [IQR] score 39.0 [34.0-44.2] and 40.0 [34.0-48.0], p < 0.001, p = 0.001). CONCLUSION: The virtual dynamic game approach showed for the first time that the gaming performance of patients with migraine without aura was negatively affected by the addition of auditory, verbal, and haptic stimuli onto visual stimuli. Multisensory integration of sensory modalities including haptic stimuli is disturbed even in the interictal period in women with migraine. Virtual games can be employed to assess the impact of sensory problems in the course of the disease. Also, sensory training could be a potential therapy target to improve multisensory processing in migraine.

What Fechner could not do: Separating perceptual encoding and decoding with difference scaling.

A key question in perception research is how stimulus variations translate into perceptual magnitudes, that is, the perceptual encoding process. As experimenters, we cannot probe perceptual magnitudes directly, but infer the encoding process from responses obtained in a psychophysical experiment. The most prominent experimental technique to measure perceptual appearance is matching, where observers adjust a probe stimulus to match a target in its appearance along the dimension of interest. The resulting data quantify the perceived magnitude of the target in physical units of the probe, and are thus an indirect expression of the underlying encoding process. In this paper, we show analytically and in simulation that data from matching tasks do not sufficiently constrain perceptual encoding functions, because there exist an infinite number of pairs of encoding functions that generate the same matching data. We use simulation to demonstrate that maximum likelihood conjoint measurement (Ho, Landy, & Maloney, 2008; Knoblauch & Maloney, 2012) does an excellent job of recovering the shape of ground truth encoding functions from data that were generated with these very functions. Finally, we measure perceptual scales and matching data for White's effect (White, 1979) and show that the matching data can be predicted from the estimated encoding functions, down to individual differences.

The Parameters Governing the Anti-Myopia Efficacy of Chromatically Simulated Myopic Defocus in Tree Shrews.

Purpose: We previously showed that exposing tree shrews (Tupaia belangeri, small diurnal mammals closely related to primates) to chromatically simulated myopic defocus (CSMD) counteracted small-cage myopia and instead induced hyperopia (approximately +4 diopters [D]). Here, we explored the parameters of this effect. Methods: Tree shrews were exposed to the following interventions for 11 days: (1) rearing in closed (n = 7) or open (n = 6) small cages; (2) exposed to a video display of Maltese cross images with CSMD combined with overhead lighting (n = 4); (3) exposed to a video display of Maltese cross images with zero blue contrast ("flat blue," n = 8); and (4) exposed to a video display of black and white grayscale tree images with different spatial filtering (blue pixels lowpass <1 and <2 cycles per degree [CPD]) for the CSMD. Results: (1) Tree shrews kept in closed cages, but not open cages, developed myopia. (2) Overhead illumination reduced the hyperopia induced by CSMD. (3) Zero-blue contrast produced hyperopia but slightly less than the CSMD. (4) Both of the CSMD tree images counteracted small cage myopia, but the one low pass filtering blue <1 CPD was more effective at inducing hyperopia. Conclusions: Any pattern with reduced blue contrast at and below approximately 1 CPD counteracts myopia/promotes hyperopia, but maximal effectiveness may require that the video display be the brightest object in the environment. Translational Relevance: Chromatically simulated myopic blur might be a powerful anti-myopia therapy in children, but the parameter selection could be critical. Issues for translation to humans are discussed.