Visual perception of highly memorable images is mediated by a distributed network of ventral visual regions that enable a late memorability response.

Behavioral and neuroscience studies in humans and primates have shown that memorability is an intrinsic property of an image that predicts its strength of encoding into and retrieval from memory. While previous work has independently probed when or where this memorability effect may occur in the human brain, a description of its spatiotemporal dynamics is missing. Here, we used representational similarity analysis (RSA) to combine functional magnetic resonance imaging (fMRI) with source-estimated magnetoencephalography (MEG) to simultaneously measure when and where the human cortex is sensitive to differences in image memorability. Results reveal that visual perception of High Memorable images, compared to Low Memorable images, recruits a set of regions of interest (ROIs) distributed throughout the ventral visual cortex: a late memorability response (from around 300 ms) in early visual cortex (EVC), inferior temporal cortex, lateral occipital cortex, fusiform gyrus, and banks of the superior temporal sulcus. Image memorability magnitude results are represented after high-level feature processing in visual regions and reflected in classical memory regions in the medial temporal lobe (MTL). Our results present, to our knowledge, the first unified spatiotemporal account of visual memorability effect across the human cortex, further supporting the levels-of-processing theory of perception and memory.

Predicting memorability of face photographs with deep neural networks.

With the advent of social media in our daily life, we are exposed to a plethora of images, particularly face photographs, every day. Recent behavioural studies have shown that some of these photographs stick in the mind better than others. Previous research have shown that memorability is an intrinsic property of an image, hence the memorability of an image can be computed from that image. Moreover, various works found that the memorability of an image is highly consistent across people and also over time. Recently, researchers employed deep neural networks to predict image memorability. Here, we show although those models perform well on scene and object images, they perform poorly on photographs of human faces. We demonstrate and explain why generic memorability models do not result in an acceptable performance on face photographs and propose seven different models to estimate the memorability of face images. In addition, we show that these models outperform the previous classical methods, which were used for predicting face memorability.

Memoir study: Investigating image memorability across developmental stages.

Images have been shown to consistently differ in terms of their memorability in healthy adults: some images stick in one's mind while others are forgotten quickly. Studies have suggested that memorability is an intrinsic, continuous property of a visual stimulus that can be both measured and manipulated. Memory literature suggests that important developmental changes occur throughout adolescence that have an impact on recognition memory, yet the effect that these changes have on image memorability has not yet been investigated. In the current study, we recruited adolescents ages 11-18 (n = 273, mean = 16) to an online visual memory experiment to explore the effects of developmental changes throughout adolescence on image memorability, and determine if memorability findings in adults can be generalized to the adolescent age group. We used the online experiment to calculate adolescent memorability scores for 1,000 natural images, and compared the results to the MemCat dataset-a memorability dataset that is annotated with adult memorability scores (ages 19-27). Our study finds that memorability scores in adolescents and adults are strongly and significantly correlated (Spearman's rank correlation, r = 0.76, p < 0.001). This correlation persists even when comparing adults with developmentally different sub-groups of adolescents (ages 11-14: r = 0.67, p < 0.001; ages 15-18: r = 0.60, p < 0.001). Moreover, the rankings of image categories by mean memorability scores were identical in both adolescents and adults (including the adolescent sub-groups), indicating that broadly, certain image categories are more memorable for both adolescents and adults. Interestingly, however, adolescents experienced significantly higher false alarm rates than adults, supporting studies that show increased impulsivity and reward-seeking behaviour in adolescents. Our results reveal that the memorability of images remains consistent across individuals at different stages of development. This consistency aligns with and strengthens prior research, indicating that memorability is an intrinsic property of images. Our findings open new pathways for applying memorability studies in adolescent populations, with profound implications in fields such as education, marketing, and psychology. Our work paves the way for innovative approaches in these domains, leveraging the consistent nature of image memorability across age groups.

Effects of neuromodulation-inspired mechanisms on the performance of deep neural networks in a spatial learning task.

In recent years, the biological underpinnings of adaptive learning have been modeled, leading to faster model convergence and various behavioral benefits in tasks including spatial navigation and cue-reward association. Furthermore, studies have investigated how the neuromodulatory system, a major driver of synaptic plasticity and state-dependent changes in the brain neuronal activities, plays a role in training deep neural networks (DNNs). In this study, we extended previous studies on neuromodulation-inspired DNNs and explored the effects of neuromodulatory components on learning and single unit activities in a spatial learning task. Under the multiscale neuromodulatory framework, plastic components, dropout probability modulation, and learning rate decay were added to the single unit, layer, and whole network levels of DNN models, respectively. We observed behavioral benefits including faster learning and smaller error of ambulation. We then concluded that neuromodulatory components can affect learning trajectories, outcomes, and single unit activities, in a component- and hyperparameter-dependent manner.

Machine learning-based predictive modeling of resilience to stressors in pregnant women during COVID-19: A prospective cohort study.

During the COVID-19 pandemic, pregnant women have been at high risk for psychological distress. Lifestyle factors may be modifiable elements to help reduce and promote resilience to prenatal stress. We used Machine-Learning (ML) algorithms applied to questionnaire data obtained from an international cohort of 804 pregnant women to determine whether physical activity and diet were resilience factors against prenatal stress, and whether stress levels were in turn predictive of sleep classes. A support vector machine accurately classified perceived stress levels in pregnant women based on physical activity behaviours and dietary behaviours. In turn, we classified hours of sleep based on perceived stress levels. This research adds to a developing consensus concerning physical activity and diet, and the association with prenatal stress and sleep in pregnant women. Predictive modeling using ML approaches may be used as a screening tool and to promote positive health behaviours for pregnant women.

Prenatal and postpartum maternal mental health and neonatal motor outcomes during the COVID-19 pandemic.

Background: Rates of prenatal and postpartum stress and depression in pregnant individuals have increased during the COVID-19 pandemic. Perinatal maternal mental health has been linked to worse motor development in offspring, with motor deficits appearing in infancy and early childhood. We aimed to evaluate the relationship between prenatal and postpartum stress and depression and motor outcome in infants born during the COVID-19 pandemic. Methods: One hundred and seventeen participants completed an online prospective survey study at two timepoints: during pregnancy and within 2 months postpartum. Depression was self-reported using the Edinburgh Perinatal/Postpartum Depression Scale (EPDS), and stress via the Perceived Stress Scale (PSS). Mothers reported total infant motor ability (fine and gross) using the interRAI 0-3 Developmental Domains questionnaire. Results: Prenatal (EPDS median=10.0, interquartile range[IQR]=6.0 - 14.0, B=-0.035, 95%CI=-0.062 to -0.007, p = 0.014) and postpartum maternal depression outcomes (median=7, IQR=4-12, B=-0.037, 95%CI= -0.066 to -0.008, p = 0.012) were significantlynegatively associated with total infant motor ability. Neither pregnancy nor postpartum perceived stress was associated with infant motor function. A cluster analysis revealed that preterm and low-birth weight infants whose mothers reported elevated depressive symptoms during pregnancy and in the postpartum period had the poorest motor outcomes. Conclusions: Prenatal and postpartum depression, but not stress, was associated with early infant motor abilities. Preterm and low-birth weight infants whose mothers reported elevated depressive symptoms maybe at-risk of experiencing poor motor outcomes. These results highlight the importance of identifying pre- and postnatal maternal mental health issues, especially during the ongoing COVID-19 pandemic.

Depression in pregnant women with and without COVID-19.

Evidence suggests that pregnant women who test positive for COVID-19 may develop more severe illness than non-pregnant women and may be at greater risk for psychological distress. The relationship between COVID-19 status (positive, negative, never tested) and symptoms of depression was examined in a survey study (May to September 2020) of pregnant women (n = 869). Pregnant women who reported testing positive for COVID-19 were significantly more likely to report depressive symptoms compared with women who tested negative (P = 0.027) and women who were never tested (P = 0.005). Findings indicate that pregnant women who test positive for COVID-19 should be screened and monitored for depressive symptoms.

Reconstructing feedback representations in the ventral visual pathway with a generative adversarial autoencoder.

While vision evokes a dense network of feedforward and feedback neural processes in the brain, visual processes are primarily modeled with feedforward hierarchical neural networks, leaving the computational role of feedback processes poorly understood. Here, we developed a generative autoencoder neural network model and adversarially trained it on a categorically diverse data set of images. We hypothesized that the feedback processes in the ventral visual pathway can be represented by reconstruction of the visual information performed by the generative model. We compared representational similarity of the activity patterns in the proposed model with temporal (magnetoencephalography) and spatial (functional magnetic resonance imaging) visual brain responses. The proposed generative model identified two segregated neural dynamics in the visual brain. A temporal hierarchy of processes transforming low level visual information into high level semantics in the feedforward sweep, and a temporally later dynamics of inverse processes reconstructing low level visual information from a high level latent representation in the feedback sweep. Our results append to previous studies on neural feedback processes by presenting a new insight into the algorithmic function and the information carried by the feedback processes in the ventral visual pathway.

Cochlea to categories: The spatiotemporal dynamics of semantic auditory representations.

How does the auditory system categorize natural sounds? Here we apply multimodal neuroimaging to illustrate the progression from acoustic to semantically dominated representations. Combining magnetoencephalographic (MEG) and functional magnetic resonance imaging (fMRI) scans of observers listening to naturalistic sounds, we found superior temporal responses beginning ∼55 ms post-stimulus onset, spreading to extratemporal cortices by ∼100 ms. Early regions were distinguished less by onset/peak latency than by functional properties and overall temporal response profiles. Early acoustically-dominated representations trended systematically toward category dominance over time (after ∼200 ms) and space (beyond primary cortex). Semantic category representation was spatially specific: Vocalizations were preferentially distinguished in frontotemporal voice-selective regions and the fusiform; scenes and objects were distinguished in parahippocampal and medial place areas. Our results are consistent with real-world events coded via an extended auditory processing hierarchy, in which acoustic representations rapidly enter multiple streams specialized by category, including areas typically considered visual cortex.

Emergence of Visual Center-Periphery Spatial Organization in Deep Convolutional Neural Networks.

Research at the intersection of computer vision and neuroscience has revealed hierarchical correspondence between layers of deep convolutional neural networks (DCNNs) and cascade of regions along human ventral visual cortex. Recently, studies have uncovered emergence of human interpretable concepts within DCNNs layers trained to identify visual objects and scenes. Here, we asked whether an artificial neural network (with convolutional structure) trained for visual categorization would demonstrate spatial correspondences with human brain regions showing central/peripheral biases. Using representational similarity analysis, we compared activations of convolutional layers of a DCNN trained for object and scene categorization with neural representations in human brain visual regions. Results reveal a brain-like topographical organization in the layers of the DCNN, such that activations of layer-units with central-bias were associated with brain regions with foveal tendencies (e.g. fusiform gyrus), and activations of layer-units with selectivity for image backgrounds were associated with cortical regions showing peripheral preference (e.g. parahippocampal cortex). The emergence of a categorical topographical correspondence between DCNNs and brain regions suggests these models are a good approximation of the perceptual representation generated by biological neural networks.

Reliability and Generalizability of Similarity-Based Fusion of MEG and fMRI Data in Human Ventral and Dorsal Visual Streams.

To build a representation of what we see, the human brain recruits regions throughout the visual cortex in cascading sequence. Recently, an approach was proposed to evaluate the dynamics of visual perception in high spatiotemporal resolution at the scale of the whole brain. This method combined functional magnetic resonance imaging (fMRI) data with magnetoencephalography (MEG) data using representational similarity analysis and revealed a hierarchical progression from primary visual cortex through the dorsal and ventral streams. To assess the replicability of this method, we here present the results of a visual recognition neuro-imaging fusion experiment and compare them within and across experimental settings. We evaluated the reliability of this method by assessing the consistency of the results under similar test conditions, showing high agreement within participants. We then generalized these results to a separate group of individuals and visual input by comparing them to the fMRI-MEG fusion data of Cichy et al (2016), revealing a highly similar temporal progression recruiting both the dorsal and ventral streams. Together these results are a testament to the reproducibility of the fMRI-MEG fusion approach and allows for the interpretation of these spatiotemporal dynamic in a broader context.

Population response magnitude variation in inferotemporal cortex predicts image memorability.

Most accounts of image and object encoding in inferotemporal cortex (IT) focus on the distinct patterns of spikes that different images evoke across the IT population. By analyzing data collected from IT as monkeys performed a visual memory task, we demonstrate that variation in a complementary coding scheme, the magnitude of the population response, can largely account for how well images will be remembered. To investigate the origin of IT image memorability modulation, we probed convolutional neural network models trained to categorize objects. We found that, like the brain, different natural images evoked different magnitude responses from these networks, and in higher layers, larger magnitude responses were correlated with the images that humans and monkeys find most memorable. Together, these results suggest that variation in IT population response magnitude is a natural consequence of the optimizations required for visual processing, and that this variation has consequences for visual memory.

Beyond core object recognition: Recurrent processes account for object recognition under occlusion.

Core object recognition, the ability to rapidly recognize objects despite variations in their appearance, is largely solved through the feedforward processing of visual information. Deep neural networks are shown to achieve human-level performance in these tasks, and explain the primate brain representation. On the other hand, object recognition under more challenging conditions (i.e. beyond the core recognition problem) is less characterized. One such example is object recognition under occlusion. It is unclear to what extent feedforward and recurrent processes contribute in object recognition under occlusion. Furthermore, we do not know whether the conventional deep neural networks, such as AlexNet, which were shown to be successful in solving core object recognition, can perform similarly well in problems that go beyond the core recognition. Here, we characterize neural dynamics of object recognition under occlusion, using magnetoencephalography (MEG), while participants were presented with images of objects with various levels of occlusion. We provide evidence from multivariate analysis of MEG data, behavioral data, and computational modelling, demonstrating an essential role for recurrent processes in object recognition under occlusion. Furthermore, the computational model with local recurrent connections, used here, suggests a mechanistic explanation of how the human brain might be solving this problem.

The perceptual neural trace of memorable unseen scenes.

Some scenes are more memorable than others: they cement in minds with consistencies across observers and time scales. While memory mechanisms are traditionally associated with the end stages of perception, recent behavioral studies suggest that the features driving these memorability effects are extracted early on, and in an automatic fashion. This raises the question: is the neural signal of memorability detectable during early perceptual encoding phases of visual processing? Using the high temporal resolution of magnetoencephalography (MEG), during a rapid serial visual presentation (RSVP) task, we traced the neural temporal signature of memorability across the brain. We found an early and prolonged memorability related signal under a challenging ultra-rapid viewing condition, across a network of regions in both dorsal and ventral streams. This enhanced encoding could be the key to successful storage and recognition.

Ultra-Rapid serial visual presentation reveals dynamics of feedforward and feedback processes in the ventral visual pathway.

Human visual recognition activates a dense network of overlapping feedforward and recurrent neuronal processes, making it hard to disentangle processing in the feedforward from the feedback direction. Here, we used ultra-rapid serial visual presentation to suppress sustained activity that blurs the boundaries of processing steps, enabling us to resolve two distinct stages of processing with MEG multivariate pattern classification. The first processing stage was the rapid activation cascade of the bottom-up sweep, which terminated early as visual stimuli were presented at progressively faster rates. The second stage was the emergence of categorical information with peak latency that shifted later in time with progressively faster stimulus presentations, indexing time-consuming recurrent processing. Using MEG-fMRI fusion with representational similarity, we localized recurrent signals in early visual cortex. Together, our findings segregated an initial bottom-up sweep from subsequent feedback processing, and revealed the neural signature of increased recurrent processing demands for challenging viewing conditions.

Decoding the orientation of contrast edges from MEG evoked and induced responses.

Visual gamma oscillations have been proposed to subserve perceptual binding, but their strong modulation by diverse stimulus features confounds interpretations of their precise functional role. Overcoming this challenge necessitates a comprehensive account of the relationship between gamma responses and stimulus features. Here we used multivariate pattern analyses on human MEG data to characterize the relationships between gamma responses and one basic stimulus feature, the orientation of contrast edges. Our findings confirmed we could decode orientation information from induced responses in two dominant frequency bands at 24-32 Hz and 50-58 Hz. Decoding was higher for cardinal than oblique orientations, with similar results also obtained for evoked MEG responses. In contrast to multivariate analyses, orientation information was mostly absent in univariate signals: evoked and induced responses in early visual cortex were similar in all orientations, with only exception an inverse oblique effect observed in induced responses, such that cardinal orientations produced weaker oscillatory signals than oblique orientations. Taken together, our results showed multivariate methods are well suited for the analysis of gamma oscillations, with multivariate patterns robustly encoding orientation information and predominantly discriminating cardinal from oblique stimuli.

A State Space Model for Spatial Updating of Remembered Visual Targets during Eye Movements.

In the oculomotor system, spatial updating is the ability to aim a saccade toward a remembered visual target position despite intervening eye movements. Although this has been the subject of extensive experimental investigation, there is still no unifying theoretical framework to explain the neural mechanism for this phenomenon, and how it influences visual signals in the brain. Here, we propose a unified state-space model (SSM) to account for the dynamics of spatial updating during two types of eye movement; saccades and smooth pursuit. Our proposed model is a non-linear SSM and implemented through a recurrent radial-basis-function neural network in a dual Extended Kalman filter (EKF) structure. The model parameters and internal states (remembered target position) are estimated sequentially using the EKF method. The proposed model replicates two fundamental experimental observations: continuous gaze-centered updating of visual memory-related activity during smooth pursuit, and predictive remapping of visual memory activity before and during saccades. Moreover, our model makes the new prediction that, when uncertainty of input signals is incorporated in the model, neural population activity and receptive fields expand just before and during saccades. These results suggest that visual remapping and motor updating are part of a common visuomotor mechanism, and that subjective perceptual constancy arises in part from training the visual system on motor tasks.

Gaussian kernel width optimization for sparse Bayesian learning.

Sparse kernel methods have been widely used in regression and classification applications. The performance and the sparsity of these methods are dependent on the appropriate choice of the corresponding kernel functions and their parameters. Typically, the kernel parameters are selected using a cross-validation approach. In this paper, a learning method that is an extension of the relevance vector machine (RVM) is presented. The proposed method can find the optimal values of the kernel parameters during the training procedure. This algorithm uses an expectation-maximization approach for updating kernel parameters as well as other model parameters; therefore, the speed of convergence and computational complexity of the proposed method are the same as the standard RVM. To control the convergence of this fully parameterized model, the optimization with respect to the kernel parameters is performed using a constraint on these parameters. The proposed method is compared with the typical RVM and other competing methods to analyze the performance. The experimental results on the commonly used synthetic data, as well as benchmark data sets, demonstrate the effectiveness of the proposed method in reducing the performance dependency on the initial choice of the kernel parameters.

The relevance sample-feature machine: a sparse Bayesian learning approach to joint feature-sample selection.

This paper introduces a novel sparse Bayesian machine-learning algorithm for embedded feature selection in classification tasks. Our proposed algorithm, called the relevance sample feature machine (RSFM), is able to simultaneously choose the relevance samples and also the relevance features for regression or classification problems. We propose a separable model in feature and sample domains. Adopting a Bayesian approach and using Gaussian priors, the learned model by RSFM is sparse in both sample and feature domains. The proposed algorithm is an extension of the standard RVM algorithm, which only opts for sparsity in the sample domain. Experimental comparisons on synthetic as well as benchmark data sets show that RSFM is successful in both feature selection (eliminating the irrelevant features) and accurate classification. The main advantages of our proposed algorithm are: less system complexity, better generalization and avoiding overfitting, and less computational cost during the testing stage.