Posterior Parietal Cortex Drives Inferotemporal Activations During Three-Dimensional Object Vision.

The primate visual system consists of a ventral stream, specialized for object recognition, and a dorsal visual stream, which is crucial for spatial vision and actions. However, little is known about the interactions and information flow between these two streams. We investigated these interactions within the network processing three-dimensional (3D) object information, comprising both the dorsal and ventral stream. Reversible inactivation of the macaque caudal intraparietal area (CIP) during functional magnetic resonance imaging (fMRI) reduced fMRI activations in posterior parietal cortex in the dorsal stream and, surprisingly, also in the inferotemporal cortex (ITC) in the ventral visual stream. Moreover, CIP inactivation caused a perceptual deficit in a depth-structure categorization task. CIP-microstimulation during fMRI further suggests that CIP projects via posterior parietal areas to the ITC in the ventral stream. To our knowledge, these results provide the first causal evidence for the flow of visual 3D information from the dorsal stream to the ventral stream, and identify CIP as a key area for depth-structure processing. Thus, combining reversible inactivation and electrical microstimulation during fMRI provides a detailed view of the functional interactions between the two visual processing streams.

Caudal Intraparietal Sulcus and three-dimensional vision: A combined functional magnetic resonance imaging and single-cell study.

The cortical network processing three-dimensional (3D) object structure defined by binocular disparity spans both the ventral and dorsal visual streams. However, very little is known about the neural representation of 3D structure at intermediate levels of the visual hierarchy. Here, we investigated the neural selectivity for 3D surfaces in the macaque Posterior Intraparietal area (PIP) in the medial bank of the caudal intraparietal sulcus (IPS). We first identified a region sensitive to depth-structure information in the medial bank of the caudal IPS using functional Magnetic Resonance Imaging (fMRI), and then recorded single-cell activity within this fMRI activation in the same animals. Most PIP neurons were selective for the 3D orientation of planar surfaces (first-order disparity) at very short latencies, whereas a very small fraction of PIP neurons were selective for curved surfaces (second-order disparity). A linear support vector machine classifier could reliably identify the direction of the disparity gradient in planar and curved surfaces based on the responses of a population of disparity-selective PIP neurons. These results provide the first detailed account of the neuronal properties in area PIP, which occupies an intermediate position in the hierarchy of visual areas involved in processing depth structure from disparity.

Functional architecture for disparity in macaque inferior temporal cortex and its relationship to the architecture for faces, color, scenes, and visual field.

Binocular disparity is a powerful depth cue for object perception. The computations for object vision culminate in inferior temporal cortex (IT), but the functional organization for disparity in IT is unknown. Here we addressed this question by measuring fMRI responses in alert monkeys to stimuli that appeared in front of (near), behind (far), or at the fixation plane. We discovered three regions that showed preferential responses for near and far stimuli, relative to zero-disparity stimuli at the fixation plane. These "near/far" disparity-biased regions were located within dorsal IT, as predicted by microelectrode studies, and on the posterior inferotemporal gyrus. In a second analysis, we instead compared responses to near stimuli with responses to far stimuli and discovered a separate network of "near" disparity-biased regions that extended along the crest of the superior temporal sulcus. We also measured in the same animals fMRI responses to faces, scenes, color, and checkerboard annuli at different visual field eccentricities. Disparity-biased regions defined in either analysis did not show a color bias, suggesting that disparity and color contribute to different computations within IT. Scene-biased regions responded preferentially to near and far stimuli (compared with stimuli without disparity) and had a peripheral visual field bias, whereas face patches had a marked near bias and a central visual field bias. These results support the idea that IT is organized by a coarse eccentricity map, and show that disparity likely contributes to computations associated with both central (face processing) and peripheral (scene processing) visual field biases, but likely does not contribute much to computations within IT that are implicated in processing color.

Choice-related Activity in the Anterior Intraparietal Area during 3-D Structure Categorization.

The anterior intraparietal area (AIP) of macaques contains neurons that signal the depth structure of disparity-defined 3-D shapes. Previous studies have suggested that AIP's depth information is used for sensorimotor transformations related to the efficient grasping of 3-D objects. We trained monkeys to categorize disparity-defined 3-D shapes and examined whether neuronal activity in AIP may also underlie pure perceptual categorization behavior. We first show that neurons with a similar 3-D shape preference cluster in AIP. We then demonstrate that the monkeys' 3-D shape discrimination performance depends on the position in depth of the stimulus and that this performance difference is reflected in the activity of AIP neurons. We further reveal correlations between the neuronal activity in AIP and the subject's subsequent choices and RTs during 3-D shape categorization. Our findings propose AIP as an important processing stage for 3-D shape perception.

Synchronization between the end stages of the dorsal and the ventral visual stream.

The end stage areas of the ventral (IT) and the dorsal (AIP) visual streams encode the shape of disparity-defined three-dimensional (3D) surfaces. Recent anatomical tracer studies have found direct reciprocal connections between the 3D-shape selective areas in IT and AIP. Whether these anatomical connections are used to facilitate 3D-shape perception is still unknown. We simultaneously recorded multi-unit activity (MUA) and local field potentials in IT and AIP while monkeys discriminated between concave and convex 3D shapes and measured the degree to which the activity in IT and AIP synchronized during the task. We observed strong beta-band synchronization between IT and AIP preceding stimulus onset that decreased shortly after stimulus onset and became modulated by stereo-signal strength and stimulus contrast during the later portion of the stimulus period. The beta-coherence modulation was unrelated to task-difficulty, regionally specific, and dependent on the MUA selectivity of the pairs of sites under study. The beta-spike-field coherence in AIP predicted the upcoming choice of the monkey. Several convergent lines of evidence suggested AIP as the primary source of the AIP-IT synchronized activity. The synchronized beta activity seemed to occur during perceptual anticipation and when the system has stabilized to a particular perceptual state but not during active visual processing. Our findings demonstrate for the first time that synchronized activity exists between the end stages of the dorsal and ventral stream during 3D-shape discrimination.

Binary CorNET: Accelerator for HR Estimation From Wrist-PPG.

Research on heart rate (HR) estimation using wrist-worn photoplethysmography (PPG) sensors have progressed rapidly owing to the prominence of commercial sensing modules, used widely for lifestyle monitoring. Reported methodologies have been fairly successful in mitigating the effect of motion artifacts (MA) in ambulatory environment for HR estimation. Recently, a learning framework, CorNET, employing two-layer convolution neural networks (CNN) and two-layer long short-term network (LSTM) was successfully reported for estimating HR from MA-induced PPG signals. However, such a network topology with large number of parameters presents a challenge, towards low-complexity hardware implementation aimed at on-node processing. In this paper, we demonstrate a fully binarized network (bCorNET) topology and its corresponding algorithm-to-architecture mapping and energy-efficient implementation for HR estimation. The proposed framework achieves a MAE of 6.67 ± 5.49 bpm when evaluated on 22 IEEE SPC subjects. The design, synthesized with ST65 nm technology library achieving 3 GOPS @ 1 MHz, consumes 56.1 µJ per window with occupied 1634K NAND2 equivalent cell area and had a latency of 32 ms when estimating HR every 2 s from PPG signals.

Stimulus similarity-contingent neural adaptation can be time and cortical area dependent.

Repetition of a stimulus results in decreased responses in many cortical areas. This so-called adaptation or repetition suppression has been used in several human functional magnetic resonance imaging studies to deduce the stimulus selectivity of neuronal populations. We tested in macaque monkeys whether the degree of neural adaptation depends on the similarity between the adapter and test stimulus. To manipulate similarity, we varied stimulus size. We recorded the responses of single neurons to different-sized shapes in inferior temporal (IT) and prefrontal cortical (PFC) areas while the animals were engaged in a size or shape discrimination task. The degree of response adaptation in IT decreased with increasing size differences between the adapter and the test stimuli in both tasks, but the dependence of adaptation on the degree of similarity between the adapter and test stimuli was limited mainly to the early phase of the neural response in IT. PFC neurons showed only weak size-contingent repetition effects, despite strong size selectivity observed with the same stimuli. Thus, based on the repetition effects in PFC, one would have erroneously concluded that PFC shows weak or no size selectivity in such tasks. These findings are relevant for the interpretation of functional magnetic resonance adaptation data: they support the conjecture that the degree of adaptation scales with the similarity between adapter and test stimuli. However, they also show that the temporal evolution of adaptation during the course of the response, and differences in the way individual regions react to stimulus repetition, may complicate the inference of neuronal tuning from functional magnetic resonance adaptation.

BioTranslator: Inferring R-Peaks from Ambulatory Wrist-Worn PPG Signal.

Advancements in wireless sensor networks (WSN) technology and miniaturization of wearable sensors have enabled long-term continuous pervasive biomedical signal monitoring. Wrist-worn photoplethysmography (PPG) sensors have gained popularity given their form factor. However the signal quality suffers due to motion artifacts when used in ambulatory settings, making vital parameter estimation a challenging task. In this paper, we present a novel deep learning framework, BioTranslator, for computing the instantaneous heart rate (IHR), using wrist-worn PPG signals collected during physical activity. Using one-dimensional Convolution-Deconvolution Network, we translate a single channel PPG signal to an electrocardiogram(ECG)-like time series signal, from which relevant R-peak information can be inferred enabling IHR measures. The proposed network configuration was evaluated on 12 subjects of the TROIKA dataset, involved in physical activity. The proposed network identifies 92.8% of R-peaks, besides achieving a mean absolute error of 51±6.3ms with respect to reference ECG-derived IHR.

CorNET: Deep Learning Framework for PPG-Based Heart Rate Estimation and Biometric Identification in Ambulant Environment.

Advancements in wireless sensor network technologies have enabled the proliferation of miniaturized body-worn sensors, capable of long-term pervasive biomedical signal monitoring. Remote cardiovascular monitoring has been one of the beneficiaries of this development, resulting in non-invasive, photoplethysmography (PPG) sensors being used in ambulatory settings. Wrist-worn PPG, although a popular alternative to electrocardiogram, suffers from motion artifacts inherent in daily life. Hence, in this paper, we present a novel deep learning framework (CorNET) to efficiently estimate heart rate (HR) information and perform biometric identification (BId) using only a wrist-worn, single-channel PPG signal collected in ambulant environment. We have formulated a completely personalized data-driven approach, using a four-layer deep neural network. Two convolution neural network layers are used in conjunction with two long short-term memory layers, followed by a dense output layer for modeling the temporal sequence inherent within the pulsatile signal representative of cardiac activity. The final dense layer is customized with respect to the application, functioning as: regression layer-having a single neuron to predict HR; classification layer-two neurons that identify a subject among a group. The proposed network was evaluated on the TROIKA dataset having 22 PPG records collected during various physical activities. We achieve a mean absolute error of 1.47 ± 3.37 beats per minute for HR estimation and an average accuracy of 96% for BId on 20 subjects. CorNET was further evaluated successfully in an ambulant use-case scenario with custom sensors for two subjects.

Functional interactions between the macaque dorsal and ventral visual pathways during three-dimensional object vision.

The division of labor between the dorsal and the ventral visual stream in the primate brain has inspired numerous studies on the visual system in humans and in nonhuman primates. However, how and under which circumstances the two visual streams interact is still poorly understood. Here we review evidence from anatomy, modelling, electrophysiology, electrical microstimulation (EM), reversible inactivation and functional imaging in the macaque monkey aimed at clarifying at which levels in the hierarchy of visual areas the two streams interact, and what type of information might be exchanged between the two streams during three-dimensional (3D) object viewing. Neurons in both streams encode 3D structure from binocular disparity, synchronized activity between parietal and inferotemporal areas is present during 3D structure categorization, and clusters of 3D structure-selective neurons in parietal cortex are anatomically connected to ventral stream areas. In addition, caudal intraparietal cortex exerts a causal influence on 3D-structure related activations in more anterior parietal cortex and in inferotemporal cortex. Thus, both anatomical and functional evidence indicates that the dorsal and the ventral visual stream interact during 3D object viewing.

Attention-related changes in correlated neuronal activity arise from normalization mechanisms.

Attention is believed to enhance perception by altering the activity-level correlations between pairs of neurons. How attention changes neuronal activity correlations is unknown. Using multielectrodes in monkey visual cortex, we measured spike-count correlations when single or multiple stimuli were presented and when stimuli were attended or unattended. When stimuli were unattended, adding a suppressive, nonpreferred stimulus beside a preferred stimulus increased spike-count correlations between pairs of similarly tuned neurons but decreased spike-count correlations between pairs of oppositely tuned neurons. A stochastic normalization model containing populations of oppositely tuned, mutually suppressive neurons explains these changes and also explains why attention decreased or increased correlations: as an indirect consequence of attention-related changes in the inputs to normalization mechanisms. Our findings link normalization mechanisms to correlated neuronal activity and attention, showing that normalization mechanisms shape response correlations and that these correlations change when attention biases normalization mechanisms.

Attention operates uniformly throughout the classical receptive field and the surround.

Shifting attention among visual stimuli at different locations modulates neuronal responses in heterogeneous ways, depending on where those stimuli lie within the receptive fields of neurons. Yet how attention interacts with the receptive-field structure of cortical neurons remains unclear. We measured neuronal responses in area V4 while monkeys shifted their attention among stimuli placed in different locations within and around neuronal receptive fields. We found that attention interacts uniformly with the spatially-varying excitation and suppression associated with the receptive field. This interaction explained the large variability in attention modulation across neurons, and a non-additive relationship among stimulus selectivity, stimulus-induced suppression and attention modulation that has not been previously described. A spatially-tuned normalization model precisely accounted for all observed attention modulations and for the spatial summation properties of neurons. These results provide a unified account of spatial summation and attention-related modulation across both the classical receptive field and the surround.

Binocular depth processing in the ventral visual pathway.

One of the most powerful forms of depth perception capitalizes on the small relative displacements, or binocular disparities, in the images projected onto each eye. The brain employs these disparities to facilitate various computations, including sensori-motor transformations (reaching, grasping), scene segmentation and object recognition. In accordance with these different functions, disparity activates a large number of regions in the brain of both humans and monkeys. Here, we review how disparity processing evolves along different regions of the ventral visual pathway of macaques, emphasizing research based on both correlational and causal techniques. We will discuss the progression in the ventral pathway from a basic absolute disparity representation to a more complex three-dimensional shape code. We will show that, in the course of this evolution, the underlying neuronal activity becomes progressively more bound to the global perceptual experience. We argue that these observations most probably extend beyond disparity processing per se, and pertain to object processing in the ventral pathway in general. We conclude by posing some important unresolved questions whose answers may significantly advance the field, and broaden its scope.This article is part of the themed issue 'Vision in our three-dimensional world'.

Stereopsis after anterior temporal lobectomy.

Brain areas critical for stereopsis have been investigated in non-human primates but are largely unknown in the human brain. Microelectrode recordings and functional MRI (fMRI) studies in monkeys have shown that in monkeys the inferior temporal cortex is critically involved in 3D shape categorization. Furthermore, some human fMRI studies similarly suggest an involvement of visual areas in the temporal lobe in depth perception. We aimed to investigate the role of the human anterior temporal neocortex in stereopsis by assessing stereoscopic depth perception before and after anterior temporal lobectomy. Eighteen epilepsy surgery patients were tested, pre- and postoperatively, in 3 different depth discrimination tasks. Sensitivity for local and global disparity was tested in a near-far discrimination task and sensitivity for 3D curvature was assessed in a convex-concave discrimination task, where 3D shapes were presented at different positions in depth. We found no evidence that temporal lobe epilepsy surgery has a significant effect on stereopsis. In contrast with earlier findings, we conclude that local as well as global stereopsis is maintained after unilateral resection of the temporal pole in epilepsy surgery patients. Our findings, together with previous studies, suggest that in humans more posterior visual regions underlie depth perception.

Effects of Microstimulation in the Anterior Intraparietal Area during Three-Dimensional Shape Categorization.

The anterior intraparietal area (AIP) of rhesus monkeys is part of the dorsal visual stream and contains neurons whose visual response properties are commensurate with a role in three-dimensional (3D) shape perception. Neuronal responses in AIP signal the depth structure of disparity-defined 3D shapes, reflect the choices of monkeys while they categorize 3D shapes, and mirror the behavioral variability across different stimulus conditions during 3D-shape categorization. However, direct evidence for a role of AIP in 3D-shape perception has been lacking. We trained rhesus monkeys to categorize disparity-defined 3D shapes and examined AIP's contribution to 3D-shape categorization by microstimulating in clusters of 3D-shape selective AIP neurons during task performance. We find that microstimulation effects on choices (monkey M1) and reaction times (monkey M1 and M2) depend on the 3D-shape preference of the stimulated site. Moreover, electrical stimulation of the same cells, during either the 3D-shape-categorization task or a saccade task, could affect behavior differently. Interestingly, in one monkey we observed a strong correlation between the strength of choice-related AIP activity (choice probabilities) and the influence of microstimulation on 3D-shape-categorization behavior (choices and reaction time). These findings propose AIP as part of the network responsible for 3D-shape perception. The results also show that the anterior intraparietal cortex contains cells with different tuning properties, i.e. 3D-shape- or saccade-related, that can be dynamically read out depending on the requirements of the task at hand.

Approaches for drug delivery with intracortical probes.

Intracortical microprobes allow the precise monitoring of electrical and chemical signaling and are widely used in neuroscience. Microelectromechanical system (MEMS) technologies have greatly enhanced the integration of multifunctional probes by facilitating the combination of multiple recording electrodes and drug delivery channels in a single probe. Depending on the neuroscientific application, various assembly strategies are required in addition to the microprobe fabrication itself. This paper summarizes recent advances in the fabrication and assembly of micromachined silicon probes for drug delivery achieved within the EU-funded research project NeuroProbes. The described fabrication process combines a two-wafer silicon bonding process with deep reactive ion etching, wafer grinding, and thin film patterning and offers a maximum in design flexibility. By applying this process, three general comb-like microprobe designs featuring up to four 8-mm-long shafts, cross sections from 150×200 to 250×250 µm², and different electrode and fluidic channel configurations are realized. Furthermore, we discuss the development and application of different probe assemblies for acute, semichronic, and chronic applications, including comb and array assemblies, floating microprobe arrays, as well as the complete drug delivery system NeuroMedicator for small animal research.

Inferotemporal cortex subserves three-dimensional structure categorization.

We perceive real-world objects as three-dimensional (3D), yet it is unknown which brain area underlies our ability to perceive objects in this way. The macaque inferotemporal (IT) cortex contains neurons that respond selectively to 3D structures defined by binocular disparity. To examine the causal role of IT in the categorization of 3D structures, we electrically stimulated clusters of IT neurons with a similar 3D-structure preference while monkeys performed a 3D-structure categorization task. Microstimulation of 3D-structure-selective IT clusters caused monkeys to choose the preferred structure of the 3D-structure-selective neurons considerably more often. Microstimulation in IT also accelerated the monkeys' choice for the preferred structure, while delaying choices corresponding to the nonpreferred structure of a given site. These findings reveal that 3D-structure-selective neurons in IT contribute to the categorization of 3D objects.

Contribution of inferior temporal and posterior parietal activity to three-dimensional shape perception.

One of the fundamental goals of neuroscience is to understand how perception arises from the activity of neurons in the brain. Stereopsis is a type of three-dimensional (3D) perception that relies on two slightly different projections of the world onto the retinas of the two eyes, i.e., binocular disparity. Neurons selective for curved surfaces defined by binocular disparity may contribute to the perception of an object's 3D structure. Such neurons have been observed in both the anterior lower bank of the superior temporal sulcus (TEs, part of the inferior temporal cortex [IT]) and the anterior intraparietal area (AIP). However, the specific contributions of IT and AIP to depth perception remain unknown. We simultaneously recorded multiunit activity in IT and AIP while monkeys discriminated between concave and convex 3D shapes. We observed a correlation between the neural activity and behavioral choice that arose early and during perceptual decision formation in IT but later and after perceptual decision formation in AIP. These results suggest a role for IT, but not AIP, in 3D shape discrimination. Furthermore, the results demonstrate that similar neuronal stimulus selectivities in two areas do not imply a similar function.