Critical Point-Finding Methods Reveal Gradient-Flat Regions of Deep Network Losses.

Despite the fact that the loss functions of deep neural networks are highly nonconvex, gradient-based optimization algorithms converge to approximately the same performance from many random initial points. One thread of work has focused on explaining this phenomenon by numerically characterizing the local curvature near critical points of the loss function, where the gradients are near zero. Such studies have reported that neural network losses enjoy a no-bad-local-minima property, in disagreement with more recent theoretical results. We report here that the methods used to find these putative critical points suffer from a bad local minima problem of their own: they often converge to or pass through regions where the gradient norm has a stationary point. We call these gradient-flat regions, since they arise when the gradient is approximately in the kernel of the Hessian, such that the loss is locally approximately linear, or flat, in the direction of the gradient. We describe how the presence of these regions necessitates care in both interpreting past results that claimed to find critical points of neural network losses and in designing second-order methods for optimizing neural networks.

Selectivity and robustness of sparse coding networks.

We investigate how the population nonlinearities resulting from lateral inhibition and thresholding in sparse coding networks influence neural response selectivity and robustness. We show that when compared to pointwise nonlinear models, such population nonlinearities improve the selectivity to a preferred stimulus and protect against adversarial perturbations of the input. These findings are predicted from the geometry of the single-neuron iso-response surface, which provides new insight into the relationship between selectivity and adversarial robustness. Inhibitory lateral connections curve the iso-response surface outward in the direction of selectivity. Since adversarial perturbations are orthogonal to the iso-response surface, adversarial attacks tend to be aligned with directions of selectivity. Consequently, the network is less easily fooled by perceptually irrelevant perturbations to the input. Together, these findings point to benefits of integrating computational principles found in biological vision systems into artificial neural networks.

Spontaneous activations follow a common developmental course across primary sensory areas in mouse neocortex.

Spontaneous propagation of spiking within the local neocortical circuits of mature primary sensory areas is highly nonrandom, engaging specific sets of interconnected and functionally related neurons. These spontaneous activations promise insight into neocortical structure and function, but their properties in the first 2 wk of perinatal development are incompletely characterized. Previously, we have found that there is a minimal numerical sample, on the order of 400 cells, necessary to fully capture mature neocortical circuit dynamics. Therefore we maximized our numerical sample by using two-photon calcium imaging to observe spontaneous activity in populations of up to 1,062 neurons spanning multiple columns and layers in 52 acute coronal slices of mouse neocortex at each day from postnatal day (PND) 3 to PND 15. Slices contained either primary auditory cortex (A1) or somatosensory barrel field (S1BF), which allowed us to compare sensory modalities with markedly different developmental timelines. Between PND 3 and PND 8, populations in both areas exhibited activations of anatomically compact subgroups on the order of dozens of cells. Between PND 9 and PND 13, the spatiotemporal structure of the activity diversified to include spatially distributed activations encompassing hundreds of cells. Sparse activations covering the entire field of view dominated in slices taken on or after PND 14. These and other findings demonstrate that the developmental progression of spontaneous activations from active local modules in the first postnatal week to sparse, intermingled groups of neurons at the beginning of the third postnatal week generalizes across primary sensory areas, consistent with an intrinsic developmental trajectory independent of sensory input.

Robust, automated sleep scoring by a compact neural network with distributional shift correction.

Studying the biology of sleep requires the accurate assessment of the state of experimental subjects, and manual analysis of relevant data is a major bottleneck. Recently, deep learning applied to electroencephalogram and electromyogram data has shown great promise as a sleep scoring method, approaching the limits of inter-rater reliability. As with any machine learning algorithm, the inputs to a sleep scoring classifier are typically standardized in order to remove distributional shift caused by variability in the signal collection process. However, in scientific data, experimental manipulations introduce variability that should not be removed. For example, in sleep scoring, the fraction of time spent in each arousal state can vary between control and experimental subjects. We introduce a standardization method, mixture z-scoring, that preserves this crucial form of distributional shift. Using both a simulated experiment and mouse in vivo data, we demonstrate that a common standardization method used by state-of-the-art sleep scoring algorithms introduces systematic bias, but that mixture z-scoring does not. We present a free, open-source user interface that uses a compact neural network and mixture z-scoring to allow for rapid sleep scoring with accuracy that compares well to contemporary methods. This work provides a set of computational tools for the robust automation of sleep scoring.

Acute effects of MDMA on autonomic cardiac activity and their relation to subjective prosocial and stimulant effects.

MDMA is a stimulant with unique "prosocial" effects, the physiological and pharmacological mechanisms of which are unknown. Here, we examine the relationship of measures of parasympathetic and sympathetic nervous system activity to the prosocial effects of MDMA. Parasympathetic activity was measured using respiratory sinus arrhythmia (RSA) and sympathetic activity using pre-ejection period (PEP). Over three sessions, 33 healthy volunteers received placebo, 0.75 mg/kg, and 1.5 mg/kg MDMA under counterbalanced, double-blind conditions, while we measured subjective feelings, RSA, and PEP. RSA and PEP data were available for 26 and 21 participants, respectively. MDMA increased prosocial and stimulated feelings, decreased RSA, and decreased PEP. At 1.5 mg/kg, subjective prosocial effects correlated with stimulated feelings and PEP, but not RSA. This suggests sympathetic, rather than parasympathetic, effects relate to the prosocial effects of MDMA.

MDMA decreases the effects of simulated social rejection.

3-4-Methylenedioxymethamphetamine (MDMA) increases self-reported positive social feelings and decreases the ability to detect social threat in faces, but its effects on experiences of social acceptance and rejection have not been determined. We examined how an acute dose of MDMA affects subjective and autonomic responses to simulated social acceptance and rejection. We predicted that MDMA would decrease subjective responses to rejection. On an exploratory basis, we also examined the effect of MDMA on respiratory sinus arrhythmia (RSA), a measure of parasympathetic cardiac control often thought to index social engagement and emotional regulation. Over three sessions, healthy adult volunteers with previous MDMA experience (N=36) received capsules containing placebo, 0.75 or 1.5 mg/kg of MDMA under counter-balanced double-blind conditions. During expected peak drug effect, participants played two rounds of a virtual social simulation task called "Cyberball" during which they experienced acceptance in one round and rejection in the other. During the task we also obtained electrocardiograms (ECGs), from which we calculated RSA. After each round, participants answered questionnaires about their mood and self-esteem. As predicted, MDMA decreased the effect of simulated social rejection on self-reported mood and self-esteem and decreased perceived intensity of rejection, measured as the percent of ball tosses participants reported receiving. Consistent with its sympathomimetic properties, MDMA decreased RSA as compared to placebo. Our finding that MDMA decreases perceptions of rejection in simulated social situations extends previous results indicating that MDMA reduces perception of social threat in faces. Together these findings suggest a cognitive mechanism by which MDMA might produce pro-social behavior and feelings and how the drug might function as an adjunct to psychotherapy. These phenomena merit further study in non-simulated social environments.