Neural systems for choice and valuation with counterfactual learning signals.

The purpose of this experiment was to test a computational model of reinforcement learning with and without fictive prediction error (FPE) signals to investigate how counterfactual consequences contribute to acquired representations of action-specific expected value, and to determine the functional neuroanatomy and neuromodulator systems that are involved. 80 male participants underwent dietary depletion of either tryptophan or tyrosine/phenylalanine to manipulate serotonin (5HT) and dopamine (DA), respectively. They completed 80 rounds (240 trials) of a strategic sequential investment task that required accepting interim losses in order to access a lucrative state and maximize long-term gains, while being scanned. We extended the standard Q-learning model by incorporating both counterfactual gains and losses into separate error signals. The FPE model explained the participants' data significantly better than a model that did not include counterfactual learning signals. Expected value from the FPE model was significantly correlated with BOLD signal change in the ventromedial prefrontal cortex (vmPFC) and posterior orbitofrontal cortex (OFC), whereas expected value from the standard model did not predict changes in neural activity. The depletion procedure revealed significantly different neural responses to expected value in the vmPFC, caudate, and dopaminergic midbrain in the vicinity of the substantia nigra (SN). Differences in neural activity were not evident in the standard Q-learning computational model. These findings demonstrate that FPE signals are an important component of valuation for decision making, and that the neural representation of expected value incorporates cortical and subcortical structures via interactions among serotonergic and dopaminergic modulator systems.

Neurotransmitter systems in alcohol dependence.

An interplay of different neurotransmitter systems has been implicated in the development and maintenance of alcohol dependence. Here we focus on neuroadaptations in reward-related neurotransmitter systems and their impact on central processing of alcohol-associated and reward-indicating stimuli. We discuss genotype effects on cue-induced neuronal activation and present new computational methods based on machine learning to deal with complex genotype-phenotype interactions, e.g. between brain atrophy and genes associated with glutamatergic and dopaminergic neurotransmission.

Synergistic effects of the dopaminergic and glutamatergic system on hippocampal volume in alcohol-dependent patients.

Several genes of the dopaminergic and glutamatergic neurotransmitter systems have been found to be associated with alcohol disease and related intermediate phenotypes. Here, we evaluated genetic variants of the catechol-O-methyltransferase (COMT) and the metabotropic glutamate receptor 3 (mGluR3) genes in alcohol-dependent patients and their association with volumetric measurements of brain structures. By combined analysis of imaging data and genotyping results, large numbers of variables are produced that overstrain conventional statistical methods based on tests for group differences. Limitations in assessment of epistatic effects and multiple testing problems are encountered. Therefore, we introduce a novel method for detecting associations between a set of genetic markers and phenotypical measurements based on machine learning techniques. Hippocampal volume was found to be associated with epistatic effects of the COMT-mGluR3 genes in alcohol-dependent patients but not in controls. These data are in line with prior studies supporting a role for dopamine-glutamate interaction in modulation of alcohol disease.

Visualization of synaptic markers in the optic neuropils of Drosophila using a new constrained deconvolution method.

The fruitfly Drosophila melanogaster offers compelling genetic advantages for the analysis of its nervous system, but cell size precludes immunocytochemical analysis of wild-type structure and mutant phenotypes beyond the level of neuronal arborizations. For many antibodies, especially when immunoelectron microscopy is not feasible, it would therefore be desirable to extend the resolution limit of confocal microscopy as far as possible. Because high-resolution confocal microscopy suffers from considerable blurring, so-called deconvolution algorithms are needed to remove, at least partially, the blur introduced by the microscope and by the specimen itself. Here, we present the establishment and application of a new deconvolution method to visualize synaptic markers in Drosophila optic neuropils at the resolution limit of light. We ascertained all necessary parameters experimentally and verified them by deconvolving injected fluorescent microspheres in immunostained optic lobe tissue. The resulting deconvolution method was used to analyze colocalization between the synaptic vesicle marker neuronal synaptobrevin and synaptic and putative synaptic markers in photoreceptor terminals. We report differential localization of these near the resolution limit of light, which could not be distinguished without deconvolution.

Contrast adaptation and infomax in visual cortical neurons.

In the primary visual cortex (V1) the contrast response function of many neurons saturates at high contrast and adapts depending on the visual stimulus. We propose that both effects - contrast saturation and adaptation - can be explained by a fast and a slow component in the synaptic dynamics. In our model the saturation is an effect of fast synaptic depression with a recovery time constant of about 200 ms. Fast synaptic depression leads to a contrast response function with a high gain for only a limited range of contrast values. Furthermore, we propose that slow adaptation of the transmitter release probability at the geniculocortical synapses is the underlying neural mechanism that accounts for contrast adaptation on a time scale of about 7 sec. For the functional role of contrast adaptation we make the hypothesis that it serves to achieve the best visual cortical representation of the geniculate input. This representation should maximize the mutual information between the cortical activity and the geniculocortical input by increasing the release probability in a low contrast environment. We derive an adaptation rule for the transmitter release probability based on this infomax principle. We show that changes in the transmitter release probability may compensate for changes in the variance of the geniculate inputs - an essential requirement for contrast adaptation. Also, we suggest that increasing the release probability in a low contrast environment is beneficial for signal extraction, because neurons remain sensitive only to an increase in the presynaptic activity if it is synchronous and, therefore, likely to be stimulus related. Our hypotheses are tested in numerical simulations of a network of integrate-and-fire neurons for one column of V1 using fast synaptic depression and slow synaptic adaptation. The simulations show that changing the synaptic release probability of the geniculocortical synapses is a better model for contrast adaptation than the adaptation of the synaptic weights: only in the case of changing the transmitter release probability does our model reproduce the experimental finding that the average membrane potential (DC component) adapts much more strongly than the stimulus modulated component (F1 component). In the case of changing the synaptic weights, however, the average membrane potential (DC) as well as the stimulus modulated component (F1 component) would adapt. Furthermore, changing the release probability at the recurrent cortical synapses cannot account for contrast adaptation, but could be responsible for establishing oscillatory activity often observed in recordings from visual cortical cells.

Unspecific long-term potentiation can evoke functional segregation in a model of area 17.

It has been shown recently in rat hippocampus that the synapse specificity of Hebbian long-term potentiation breaks down at short distances (< 100 microm). Using a neural network model we show that this unspecific component of long-term potentiation can be responsible for the robust formation and maintainance of cortical organization during activity-driven development. When the model is applied to the formation of orientation and ocular dominance in visual cortex, addition of an unspecific component to standard Hebbian learning, in combination with a tendency of left-eye and right-eye driven synapses to initially group together on the postsynaptic neuron, induces the simultaneous emergence and stabilization of ocular dominance and of segregated, oriented ON/OFF subfields. Since standard Hebbian learning cannot account for the simultaneous stabilization of both structures, unspecific LTP thus induces a qualitatively new behaviour. Since unspecific LTP only acts between synapses which are locally clustered in space, our results imply that details of the local grouping of synapses on the dendritic arbors of postsynaptic cells can considerably influence the formation of the cortical functional organization at the systems level.

Correction methods for three-dimensional reconstructions from confocal images: I. Tissue shrinking and axial scaling.

We show here, using locust wholemount ganglia as an example, that scaling artifacts in three-dimensional reconstructions from confocal microscopic images due to refractive index mismatch in the light path and tissue shrinking, can account for dramatic errors in measurements of morphometric values. Refractive index mismatch leads to considerable alteration of the axial dimension, and true dimensions must be restored by rescaling the Z-axis of the image stack. The appropriate scaling factor depends on the refractive indices of the media in the light path and the numerical aperture of the objective used and can be determined by numerical simulations, as we show here. In addition, different histochemical procedures were tested in regard to their effect on tissue dimensions. Reconstructions of scans at different stages of these protocols show that shrinking can be avoided prior to clearing when dehydrating ethanol series are carefully applied. Fixation and mismatching buffer osmolarity have no effect. We demonstrate procedures to reduce artifacts during mounting and clearing in methyl salicylate, such that only isometric shrinkage occurs, which can easily be corrected by rescaling the image dimensions. Glycerol-based clearing agents produced severe anisometric and nonlinear shrinkage and we could not find a way to overcome this.

Neural pattern formation via a competitive Hebbian mechanism.

In this contribution we investigate a simple pattern formation process [9,10] based on Hebbian learning and competitive interactions within cortex. This process generates spatial representations of afferent (sensory) information which strongly resemble patterns of response properties of neurons commonly called brain maps. For one of the most thoroughly studied phenomena in cortical development, the formation of topographic maps, orientation and ocular dominance columns in macaque striate cortex, the process, for example, generates the observed patterns of receptive field properties including the recently described correlations between orientation preference and ocular dominance. Competitive Hebbian learning has not only proven to be a useful concept in the understanding of development and plasticity in several brain areas, but the underlying principles have have been successfully applied to problems in machine learning [22]. The model's universality, simplicity, predictive power, and usefulness warrants a closer investigation.

A model for the depth-dependence of receptive field size and contrast sensitivity of cells in layer 4C of macaque striate cortex.

A model of LGN-input to layer 4C of macaque primary visual cortex has been used to test the hypothesis that feedforward convergence of P- and M-inputs onto layer 4C spiny stellate neurons is sufficient to explain the observed gradual change in receptive field size and contrast sensitivity with depth in the layer. Overlap of dendrites of postsynaptic neurons between M- and P-input zones proved sufficient to explain change in the lower two-thirds of layer 4C, while more rapid change in upper 4C was matched by proposing two different M-inputs with partial overlap in upper 4C alpha.

Blind signal separation from optical imaging recordings with extended spatial decorrelation.

Optical imaging is the video recording of two-dimensional patterns of changes in light reflectance from cortical tissue evoked by stimulation. We derived a method, extended spatial decorrelation (ESD), that uses second-order statistics in space for separating the intrinsic signals into the stimulus related components and the nonspecific variations. The performance of ESD on model data is compared to independent component analysis algorithms using statistics of fourth and higher order. Robustness against sensor noise is scored. When applied to optical images, ESD separates the stimulus specific signal well from biological noise and artifacts.

Soft nearest prototype classification.

We propose a new method for the construction of nearest prototype classifiers which is based on a Gaussian mixture ansatz and which can be interpreted as an annealed version of learning vector quantization (LVQ). The algorithm performs a gradient descent on a cost-function minimizing the classification error on the training set. We investigate the properties of the algorithm and assess its performance for several toy data sets and for an optical letter classification task. Results show 1) that annealing in the dispersion parameter of the Gaussian kernels improves classification accuracy; 2) that classification results are better than those obtained with standard learning vector quantization (LVQ 2.1, LVQ 3) for equal numbers of prototypes; and 3) that annealing of the width parameter improved the classification capability. Additionally, the principled approach provides an explanation of a number of features of the (heuristic) LVQ methods.

The effect of novelty on reinforcement learning.

Recent research suggests that novelty has an influence on reward-related learning. Here, we showed that novel stimuli presented from a pre-familiarized category can accelerate or decelerate learning of the most rewarding category, depending on the condition. The extent of this influence depended on the individual trait of novelty seeking. Different reinforcement learning models were developed to quantify subjects' choices. We introduced a bias parameter to model explorative behavior toward novel stimuli and characterize individual variation in novelty response. The theoretical framework allowed us to test different assumptions, concerning the motivational value of novelty. The best fitting-model combined all novelty components and had a significant positive correlation with both the experimentally measured novelty bias and the independent novelty-seeking trait. Altogether, we have not only shown that novelty by itself enhances behavioral responses underlying reward processing, but also that novelty has a direct influence on reward-dependent learning processes, consistently with computational predictions.

Activity driven adaptive stochastic resonance.

Cortical neurons in vivo show fluctuations in their membrane potential of the order of several milli-volts. Using simple and biophysically realistic models of a single neuron we demonstrate that noise induced fluctuations can be used to adaptively optimize the sensitivity of the neuron's output to ensembles of subthreshold inputs of different average strengths. Optimal information transfer is achieved by changing the strength of the noise such that the neuron's average firing rate remains constant. Adaptation is fast, because only crude estimates of the output rate are required at any time.

The effect of intracortical competition on the formation of topographic maps in models of Hebbian learning.

Correlation-based learning (CBL) models and self-organizing maps (SOM) are two classes of Hebbian models that have both been proposed to explain the activity-driven formation of cortical maps. Both models differ significantly in the way lateral cortical interactions are treated, leading to different predictions for the formation of receptive fields. The linear CBL models predict that receptive field profiles are determined by the average values and the spatial correlations of the second order of the afferent activity patterns, whereas SOM models map stimulus features. Here, we investigate a class of models which are characterized by a variable degree of lateral competition and which have the CBL and SOM models as limit cases. We show that there exists a critical value for intracortical competition below which the model exhibits CBL properties and above which feature mapping sets in. The class of models is then analyzed with respect to the formation of topographic maps between two layers of neurons. For Gaussian input stimuli we find that localized receptive fields and topographic maps emerge above the critical value for intracortical competition, and we calculate this value as a function of the size of the input stimuli and the range of the lateral interaction function. Additionally, we show that the learning rule can be derived via the optimization of a global cost function in a framework of probabilistic output neurons which represent a set of input stimuli by a sparse code.

Simulation of scanning laser techniques for optical imaging of blood-related intrinsic signals.

Optical imaging of intrinsic signals detects neural activation patterns by taking video images of the local activity-related changes in the light intensity reflected from neural tissue (intrinsic signals). At red light (605 nm), these signals are caused mainly by local variations of the tissue absorption following deoxygenation of blood. We characterize the image generation process during optical imaging by Monte Carlo simulations of light propagation through a homogeneous model tissue equipped with a local absorber. Conventional video imaging and scanning laser imaging are compared. We find that, compared with video imaging, scanning laser techniques drastically increase both the contrast and the lateral resolution of optical recordings. Also, the maximum depth up to which the signals can be detected is increased by roughly a factor of 2 when scanning laser optical imaging is used. Further, the radial profile of the diffuse-reflectance pattern for each pixel is subject to changes that correlate with the depth of the absorber within the tissue. We suggest a detection geometry for the online measurement of these radial profiles that can be realized by modifying a standard scanning laser ophthalmoscope.

A stochastic self-organizing map for proximity data.

We derive an efficient algorithm for topographic mapping of proximity data (TMP), which can be seen as an extension of Kohonen's self-organizing map to arbitrary distance measures. The TMP cost function is derived in a Baysian framework of folded Markov chains for the description of autoencoders. It incorporates the data by a dissimilarity matrix D and the topographic neighborhood by a matrix H of transition probabilities. From the principle of maximum entropy, a nonfactorizing Gibbs distribution is obtained, which is approximated in a mean-field fashion. This allows for maximum likelihood estimation using an expectation-maximization algorithm. In analogy to the transition from topographic vector quantization to the self-organizing map, we suggest an approximation to TMP that is computationally more efficient. In order to prevent convergence to local minima, an annealing scheme in the temperature parameter is introduced, for which the critical temperature of the first phase transition is calculated in terms of D and H. Numerical results demonstrate the working of the algorithm and confirm the analytical results. Finally, the algorithm is used to generate a connection map of areas of the cat's cerebral cortex.