Neither hype nor gloom do DNNs justice.

Neither the hype exemplified in some exaggerated claims about deep neural networks (DNNs), nor the gloom expressed by Bowers et al. do DNNs as models in vision science justice: DNNs rapidly evolve, and today's limitations are often tomorrow's successes. In addition, providing explanations as well as prediction and image-computability are model desiderata; one should not be favoured at the expense of the other.

Robust and data-efficient generalization of self-supervised machine learning for diagnostic imaging.

Machine-learning models for medical tasks can match or surpass the performance of clinical experts. However, in settings differing from those of the training dataset, the performance of a model can deteriorate substantially. Here we report a representation-learning strategy for machine-learning models applied to medical-imaging tasks that mitigates such 'out of distribution' performance problem and that improves model robustness and training efficiency. The strategy, which we named REMEDIS (for 'Robust and Efficient Medical Imaging with Self-supervision'), combines large-scale supervised transfer learning on natural images and intermediate contrastive self-supervised learning on medical images and requires minimal task-specific customization. We show the utility of REMEDIS in a range of diagnostic-imaging tasks covering six imaging domains and 15 test datasets, and by simulating three realistic out-of-distribution scenarios. REMEDIS improved in-distribution diagnostic accuracies up to 11.5% with respect to strong supervised baseline models, and in out-of-distribution settings required only 1-33% of the data for retraining to match the performance of supervised models retrained using all available data. REMEDIS may accelerate the development lifecycle of machine-learning models for medical imaging.

Interhemispheric transfer of working memories.

Visual working memory (WM) storage is largely independent between the left and right visual hemifields/cerebral hemispheres, yet somehow WM feels seamless. We studied how WM is integrated across hemifields by recording neural activity bilaterally from lateral prefrontal cortex. An instructed saccade during the WM delay shifted the remembered location from one hemifield to the other. Before the shift, spike rates and oscillatory power showed clear signatures of memory laterality. After the shift, the lateralization inverted, consistent with transfer of the memory trace from one hemisphere to the other. Transferred traces initially used different neural ensembles from feedforward-induced ones, but they converged at the end of the delay. Around the time of transfer, synchrony between the two prefrontal hemispheres peaked in theta and beta frequencies, with a directionality consistent with memory trace transfer. This illustrates how dynamics between the two cortical hemispheres can stitch together WM traces across visual hemifields.

Generalized Shape Metrics on Neural Representations.

Understanding the operation of biological and artificial networks remains a difficult and important challenge. To identify general principles, researchers are increasingly interested in surveying large collections of networks that are trained on, or biologically adapted to, similar tasks. A standardized set of analysis tools is now needed to identify how network-level covariates-such as architecture, anatomical brain region, and model organism-impact neural representations (hidden layer activations). Here, we provide a rigorous foundation for these analyses by defining a broad family of metric spaces that quantify representational dissimilarity. Using this framework, we modify existing representational similarity measures based on canonical correlation analysis and centered kernel alignment to satisfy the triangle inequality, formulate a novel metric that respects the inductive biases in convolutional layers, and identify approximate Euclidean embeddings that enable network representations to be incorporated into essentially any off-the-shelf machine learning method. We demonstrate these methods on large-scale datasets from biology (Allen Institute Brain Observatory) and deep learning (NAS-Bench-101). In doing so, we identify relationships between neural representations that are interpretable in terms of anatomical features and model performance.

Laminar recordings in frontal cortex suggest distinct layers for maintenance and control of working memory.

All of the cerebral cortex has some degree of laminar organization. These different layers are composed of neurons with distinct connectivity patterns, embryonic origins, and molecular profiles. There are little data on the laminar specificity of cognitive functions in the frontal cortex, however. We recorded neuronal spiking/local field potentials (LFPs) using laminar probes in the frontal cortex (PMd, 8A, 8B, SMA/ACC, DLPFC, and VLPFC) of monkeys performing working memory (WM) tasks. LFP power in the gamma band (50-250 Hz) was strongest in superficial layers, and LFP power in the alpha/beta band (4-22 Hz) was strongest in deep layers. Memory delay activity, including spiking and stimulus-specific gamma bursting, was predominately in superficial layers. LFPs from superficial and deep layers were synchronized in the alpha/beta bands. This was primarily unidirectional, with alpha/beta bands in deep layers driving superficial layer activity. The phase of deep layer alpha/beta modulated superficial gamma bursting associated with WM encoding. Thus, alpha/beta rhythms in deep layers may regulate the superficial layer gamma bands and hence maintenance of the contents of WM.

How thoughts arise from sights: inferotemporal and prefrontal contributions to vision.

We are rapidly approaching a comprehensive understanding of the neural mechanisms behind object recognition. How we use this knowledge of the visual world to plan and act is comparatively mysterious. To fill this gap, we must understand how visual representations are transformed within cognitive regions, and how these cognitive representations of visual information act back upon earlier sensory representations. Here, we summarize our current understanding of visual representation in inferotemporal cortex (IT) and prefrontal cortex (PFC), and the interactions between them. We emphasize the apparent consistency of visual representation in PFC across tasks, and suggest ways to leverage advances in our understanding of high-level vision to better understand cognitive processing.

Persistent Single-Neuron Activity during Working Memory in the Human Medial Temporal Lobe.

Working memory is an essential component of human cognition. Persistent activity related to working memory has been reported in many brain areas, including the inferior temporal and prefrontal cortex [1-8]. The medial temporal lobe (MTL) contains "concept cells" that respond invariantly to specific individuals or places whether presented as images, text, or speech [9, 10]. It is unknown, however, whether the MTL also participates in working memory processes. We thus sought to determine whether human MTL neurons respond to images held in working memory. We recorded from patients with chronically intractable epilepsy as they performed a task that required them to remember three or four sequentially presented pictures across a brief delay. 48% of visually selective neurons continued to carry image-specific information after image offset, but most ceased to encode previously presented images after a subsequent presentation of a different image. However, 8% of visually selective neurons encoded previously presented images during a final maintenance period, despite presentation of further images in the intervening interval. Population activity of stimulus-selective neurons predicted behavioral outcome in terms of correct and incorrect responses. These findings indicate that the MTL is part of a brain-wide network for working memory.

Scene-selective coding by single neurons in the human parahippocampal cortex.

Imaging, electrophysiological, and lesion studies have shown a relationship between the parahippocampal cortex (PHC) and the processing of spatial scenes. Our present knowledge of PHC, however, is restricted to the macroscopic properties and dynamics of bulk tissue; the behavior and selectivity of single parahippocampal neurons remains largely unknown. In this study, we analyzed responses from 630 parahippocampal neurons in 24 neurosurgical patients during visual stimulus presentation. We found a spatially clustered subpopulation of scene-selective units with an associated event-related field potential. These units form a population code that is more distributed for scenes than for other stimulus categories, and less sparse than elsewhere in the medial temporal lobe. Our electrophysiological findings provide insight into how individual units give rise to the population response observed with functional imaging in the parahippocampal place area.

Stimulus Load and Oscillatory Activity in Higher Cortex.

Exploring and exploiting a rich visual environment requires perceiving, attending, and remembering multiple objects simultaneously. Recent studies have suggested that this mental "juggling" of multiple objects may depend on oscillatory neural dynamics. We recorded local field potentials from the lateral intraparietal area, frontal eye fields, and lateral prefrontal cortex while monkeys maintained variable numbers of visual stimuli in working memory. Behavior suggested independent processing of stimuli in each hemifield. During stimulus presentation, higher-frequency power (50-100 Hz) increased with the number of stimuli (load) in the contralateral hemifield, whereas lower-frequency power (8-50 Hz) decreased with the total number of stimuli in both hemifields. During the memory delay, lower-frequency power increased with contralateral load. Load effects on higher frequencies during stimulus encoding and lower frequencies during the memory delay were stronger when neural activity also signaled the location of the stimuli. Like power, higher-frequency synchrony increased with load, but beta synchrony (16-30 Hz) showed the opposite effect, increasing when power decreased (stimulus presentation) and decreasing when power increased (memory delay). Our results suggest roles for lower-frequency oscillations in top-down processing and higher-frequency oscillations in bottom-up processing.

Custom-fit radiolucent cranial implants for neurophysiological recording and stimulation.

BACKGROUND: Recording and manipulating neural activity in awake behaving animal models requires long-term implantation of cranial implants that must address a variety of design considerations, which include preventing infection, minimizing tissue damage, mechanical strength of the implant, and MRI compatibility. NEW METHOD: Here we address these issues by designing legless, custom-fit cranial implants using structural MRI-based reconstruction of the skull and that are made from carbon-reinforced PEEK. RESULTS: We report several novel custom-fit radiolucent implant designs, which include a legless recording chamber, a legless stimulation chamber, a multi-channel microdrive and a head post. The fit to the skull was excellent in all cases, with no visible gaps between the base of the implants and the skull. The wound margin was minimal in size and showed no sign of infection or skin recession. COMPARISON WITH EXISTING METHODS: Cranial implants used for neurophysiological investigation in awake behaving animals often employ methyl methacrylate (MMA) to serve as a bonding agent to secure the implant to the skull. Other designs rely on radially extending legs to secure the implant. Both of these methods have significant drawbacks. MMA is toxic to bone and frequently leads to infection while radially extending legs cause the skin to recede away from the implant, ultimately exposing bone and proliferating granulation tissue. CONCLUSIONS: These radiolucent implants constitute a set of technologies suitable for reliable long-term recording, which minimize infection and tissue damage.

A network for scene processing in the macaque temporal lobe.

Spatial navigation is a complex process, but one that is essential for any mobile organism. We localized a region in the macaque occipitotemporal sulcus that responds preferentially to images of scenes. Single-unit recording revealed that this region, which we term the lateral place patch (LPP), contained a large concentration of scene-selective single units. These units were not modulated by spatial layout alone but were instead modulated by a combination of spatial and nonspatial factors, with individual units coding specific scene parts. We further demonstrate by microstimulation that LPP is connected with extrastriate visual areas V4V and DP and a scene-selective medial place patch in the parahippocampal gyrus, revealing a ventral network for visual scene processing in the macaque.

A category-specific response to animals in the right human amygdala.

The amygdala is important in emotion, but it remains unknown whether it is specialized for certain stimulus categories. We analyzed responses recorded from 489 single neurons in the amygdalae of 41 neurosurgical patients and found a categorical selectivity for pictures of animals in the right amygdala. This selectivity appeared to be independent of emotional valence or arousal and may reflect the importance that animals held throughout our evolutionary past.

Latency and selectivity of single neurons indicate hierarchical processing in the human medial temporal lobe.

Neurons in the temporal lobe of both monkeys and humans show selective responses to classes of visual stimuli and even to specific individuals. In this study, we investigate the latency and selectivity of visually responsive neurons recorded from microelectrodes in the parahippocampal cortex, entorhinal cortex, hippocampus, and amygdala of human subjects during a visual object presentation task. During 96 experimental sessions in 35 subjects, we recorded from a total of 3278 neurons. Of these units, 398 responded selectively to one or more of the presented stimuli. Mean response latencies were substantially larger than those reported in monkeys. We observed a highly significant correlation between the latency and the selectivity of these neurons: the longer the latency the greater the selectivity. Particularly, parahippocampal neurons were found to respond significantly earlier and less selectively than those in the other three regions. Regional analysis showed significant correlations between latency and selectivity within the parahippocampal cortex, entorhinal cortex, and hippocampus, but not within the amygdala. The later and more selective responses tended to be generated by cells with sparse baseline firing rates and vice versa. Our results provide direct evidence for hierarchical processing of sensory information at the interface between the visual pathway and the limbic system, by which increasingly refined and specific representations of stimulus identity are generated over time along the anatomic pathways of the medial temporal lobe.