Neural systems for choice and valuation with counterfactual learning signals.

The purpose of this experiment was to test a computational model of reinforcement learning with and without fictive prediction error (FPE) signals to investigate how counterfactual consequences contribute to acquired representations of action-specific expected value, and to determine the functional neuroanatomy and neuromodulator systems that are involved. 80 male participants underwent dietary depletion of either tryptophan or tyrosine/phenylalanine to manipulate serotonin (5HT) and dopamine (DA), respectively. They completed 80 rounds (240 trials) of a strategic sequential investment task that required accepting interim losses in order to access a lucrative state and maximize long-term gains, while being scanned. We extended the standard Q-learning model by incorporating both counterfactual gains and losses into separate error signals. The FPE model explained the participants' data significantly better than a model that did not include counterfactual learning signals. Expected value from the FPE model was significantly correlated with BOLD signal change in the ventromedial prefrontal cortex (vmPFC) and posterior orbitofrontal cortex (OFC), whereas expected value from the standard model did not predict changes in neural activity. The depletion procedure revealed significantly different neural responses to expected value in the vmPFC, caudate, and dopaminergic midbrain in the vicinity of the substantia nigra (SN). Differences in neural activity were not evident in the standard Q-learning computational model. These findings demonstrate that FPE signals are an important component of valuation for decision making, and that the neural representation of expected value incorporates cortical and subcortical structures via interactions among serotonergic and dopaminergic modulator systems.

The effect of novelty on reinforcement learning.

Recent research suggests that novelty has an influence on reward-related learning. Here, we showed that novel stimuli presented from a pre-familiarized category can accelerate or decelerate learning of the most rewarding category, depending on the condition. The extent of this influence depended on the individual trait of novelty seeking. Different reinforcement learning models were developed to quantify subjects' choices. We introduced a bias parameter to model explorative behavior toward novel stimuli and characterize individual variation in novelty response. The theoretical framework allowed us to test different assumptions, concerning the motivational value of novelty. The best fitting-model combined all novelty components and had a significant positive correlation with both the experimentally measured novelty bias and the independent novelty-seeking trait. Altogether, we have not only shown that novelty by itself enhances behavioral responses underlying reward processing, but also that novelty has a direct influence on reward-dependent learning processes, consistently with computational predictions.

Neurotransmitter systems in alcohol dependence.

An interplay of different neurotransmitter systems has been implicated in the development and maintenance of alcohol dependence. Here we focus on neuroadaptations in reward-related neurotransmitter systems and their impact on central processing of alcohol-associated and reward-indicating stimuli. We discuss genotype effects on cue-induced neuronal activation and present new computational methods based on machine learning to deal with complex genotype-phenotype interactions, e.g. between brain atrophy and genes associated with glutamatergic and dopaminergic neurotransmission.

Synergistic effects of the dopaminergic and glutamatergic system on hippocampal volume in alcohol-dependent patients.

Several genes of the dopaminergic and glutamatergic neurotransmitter systems have been found to be associated with alcohol disease and related intermediate phenotypes. Here, we evaluated genetic variants of the catechol-O-methyltransferase (COMT) and the metabotropic glutamate receptor 3 (mGluR3) genes in alcohol-dependent patients and their association with volumetric measurements of brain structures. By combined analysis of imaging data and genotyping results, large numbers of variables are produced that overstrain conventional statistical methods based on tests for group differences. Limitations in assessment of epistatic effects and multiple testing problems are encountered. Therefore, we introduce a novel method for detecting associations between a set of genetic markers and phenotypical measurements based on machine learning techniques. Hippocampal volume was found to be associated with epistatic effects of the COMT-mGluR3 genes in alcohol-dependent patients but not in controls. These data are in line with prior studies supporting a role for dopamine-glutamate interaction in modulation of alcohol disease.

Activity driven adaptive stochastic resonance.

Cortical neurons in vivo show fluctuations in their membrane potential of the order of several milli-volts. Using simple and biophysically realistic models of a single neuron we demonstrate that noise induced fluctuations can be used to adaptively optimize the sensitivity of the neuron's output to ensembles of subthreshold inputs of different average strengths. Optimal information transfer is achieved by changing the strength of the noise such that the neuron's average firing rate remains constant. Adaptation is fast, because only crude estimates of the output rate are required at any time.

Soft nearest prototype classification.

We propose a new method for the construction of nearest prototype classifiers which is based on a Gaussian mixture ansatz and which can be interpreted as an annealed version of learning vector quantization (LVQ). The algorithm performs a gradient descent on a cost-function minimizing the classification error on the training set. We investigate the properties of the algorithm and assess its performance for several toy data sets and for an optical letter classification task. Results show 1) that annealing in the dispersion parameter of the Gaussian kernels improves classification accuracy; 2) that classification results are better than those obtained with standard learning vector quantization (LVQ 2.1, LVQ 3) for equal numbers of prototypes; and 3) that annealing of the width parameter improved the classification capability. Additionally, the principled approach provides an explanation of a number of features of the (heuristic) LVQ methods.

Analysis of calcium imaging signals from the honeybee brain by nonlinear models.

Recent Ca(2+)-imaging studies on the antennal lobe of the honeybee (Apis mellifera) have shown that olfactory stimuli evoke complex spatiotemporal changes of the intracellular Ca(2+) concentration, in which stimulus-dependent subsets of glomeruli are highlighted. In this work we use nonlinear models for the quantitative identification of the spatial and temporal properties of the Ca(2+)-dependent fluorescence signal. This technique describes time series of the Ca(2+) signal as a superposition of biophysically motivated model functions for photobleaching and Ca(2+) dynamics and provides optimal estimates of their amplitudes (signal strengths) and time constants together with error measures. Using this method, we can reliably identify two different stimulus-dependent signal components. Their delays and rise times, delta(c1) = (0.4 +/- 0.3) s, tau(c1) = (3.8 +/- 1.2) s for the fast component and delta(c2) = (2.4 +/- 0.6) s, tau(c2) = (10.3 +/- 3.2) s for the slow component, are constant over space and across different odors and animals. In chronological experiments, the amplitude of the fast (slow) component often decreases (increases) with time. The pattern of the Ca(2+) dynamics in space and time can be reliably described as a superposition of only two spatiotemporally separable patterns based on the fast and slow components. However, the distributions of both components over space turn out to differ from each other, and more work has to be done in order to specify their relationship with neuronal activity.

Visualization of synaptic markers in the optic neuropils of Drosophila using a new constrained deconvolution method.

The fruitfly Drosophila melanogaster offers compelling genetic advantages for the analysis of its nervous system, but cell size precludes immunocytochemical analysis of wild-type structure and mutant phenotypes beyond the level of neuronal arborizations. For many antibodies, especially when immunoelectron microscopy is not feasible, it would therefore be desirable to extend the resolution limit of confocal microscopy as far as possible. Because high-resolution confocal microscopy suffers from considerable blurring, so-called deconvolution algorithms are needed to remove, at least partially, the blur introduced by the microscope and by the specimen itself. Here, we present the establishment and application of a new deconvolution method to visualize synaptic markers in Drosophila optic neuropils at the resolution limit of light. We ascertained all necessary parameters experimentally and verified them by deconvolving injected fluorescent microspheres in immunostained optic lobe tissue. The resulting deconvolution method was used to analyze colocalization between the synaptic vesicle marker neuronal synaptobrevin and synaptic and putative synaptic markers in photoreceptor terminals. We report differential localization of these near the resolution limit of light, which could not be distinguished without deconvolution.

An analysis of orientation and ocular dominance patterns in the visual cortex of cats and ferrets.

We report an analysis of orientation and ocular dominance maps that were recorded optically from area 17 of cats and ferrets. Similar to a recent study performed in primates (Obermayer & Blasdel, 1997), we find that 80% (for cats and ferrets) of orientation singularities that are nearest neighbors have opposite sign and that the spatial distribution of singularities deviates from a random distribution of points, because the average distances between nearest neighbors are significantly larger than expected for a random distribution. Orientation maps of normally raised cats and ferrets show approximately the same typical wavelength; however, the density of singularities is higher in ferrets than in cats. Also, we find the well-known overrepresentation of cardinal versus oblique orientations in young ferrets (Chapman & Bonhoeffer, 1998; Coppola, White, Fitzpatrick, & Purves, 1998) but only a weak, not quite significant overrepresentation of cardinal orientations in cats, as has been reported previously (Bonhoeffer & Grinvald, 1993). Orientation and ocular dominance slabs in cats exhibit a tendency of being orthogonal to each other (Hubener, Shoham, Grinvald, & Bonhoeffer, 1997), albeit less pronounced, as has been reported for primates (Obermayer & Blasdel, 1993). In chronic recordings from single animals, a decrease of the singularity density and an increase of the ocular dominance wavelength with age but no change of the orientation wavelengths were found. Orientation maps are compared with two pattern models for orientation preference maps: bandpass-filtered white noise and the field analogy model. Bandpass-filtered white noise predicts sign correlations between orientation singularities, but the correlations are significantly stronger (87% opposite sign pairs) than what we have found in the data. Also, bandpass-filtered noise predicts a deviation of the spatial distribution of singularities from a random dot pattern. The field analogy model can account for the structure of certain local patches but not for the whole orientation map. Differences between the predictions of the field analogy model and experimental data are smaller than what has been reported for primates (Obermayer & Blasdel, 1997), which can be explained by the smaller size of the imaged areas in cats and ferrets.

Correction methods for three-dimensional reconstructions from confocal images: I. Tissue shrinking and axial scaling.

We show here, using locust wholemount ganglia as an example, that scaling artifacts in three-dimensional reconstructions from confocal microscopic images due to refractive index mismatch in the light path and tissue shrinking, can account for dramatic errors in measurements of morphometric values. Refractive index mismatch leads to considerable alteration of the axial dimension, and true dimensions must be restored by rescaling the Z-axis of the image stack. The appropriate scaling factor depends on the refractive indices of the media in the light path and the numerical aperture of the objective used and can be determined by numerical simulations, as we show here. In addition, different histochemical procedures were tested in regard to their effect on tissue dimensions. Reconstructions of scans at different stages of these protocols show that shrinking can be avoided prior to clearing when dehydrating ethanol series are carefully applied. Fixation and mismatching buffer osmolarity have no effect. We demonstrate procedures to reduce artifacts during mounting and clearing in methyl salicylate, such that only isometric shrinkage occurs, which can easily be corrected by rescaling the image dimensions. Glycerol-based clearing agents produced severe anisometric and nonlinear shrinkage and we could not find a way to overcome this.

Blind signal separation from optical imaging recordings with extended spatial decorrelation.

Optical imaging is the video recording of two-dimensional patterns of changes in light reflectance from cortical tissue evoked by stimulation. We derived a method, extended spatial decorrelation (ESD), that uses second-order statistics in space for separating the intrinsic signals into the stimulus related components and the nonspecific variations. The performance of ESD on model data is compared to independent component analysis algorithms using statistics of fourth and higher order. Robustness against sensor noise is scored. When applied to optical images, ESD separates the stimulus specific signal well from biological noise and artifacts.

Principal component analysis and blind separation of sources for optical imaging of intrinsic signals.

The analysis of data sets from optical imaging of intrinsic signals requires the separation of signals, which accurately reflect stimulated neuronal activity (mapping signal), from signals related to background activity. Here we show that blind separation of sources by extended spatial decorrelation (ESD) is a powerful method for the extraction of the mapping signal from the total recorded signal. ESD is based on the assumptions (i) that each signal component varies smoothly across space and (ii) that every component has zero cross-correlation functions with the other components. In contrast to the standard analysis of optical imaging data, the proposed method (i) is applicable to nonorthogonal stimulus-conditions, (ii) can remove the global signal, blood-vessel patterns, and movement artifacts, (iii) works without ad hoc assumptions about the data structure in the frequency domain, and (iv) provides a confidence measure for the signals (Z score). We first demonstrate on orientation maps from cat and ferret visual cortex, that principal component analysis, which acts as a preprocessing step to ESD, can already remove global signals from image stacks, as long as data stacks for at least two-not necessarily orthogonal-stimulus conditions are available. We then show that the full ESD analysis can further reduce global signal components and-finally-concentrate the mapping signal within a single component both for differential image stacks and for image stacks recorded during presentation of a single stimulus.

A mean-field model for orientation tuning, contrast saturation, and contextual effects in the primary visual cortex.

Orientation-selective cells in the primary visual cortex of monkeys and cats are often characterized by an orientation-tuning width that is invariant under stimulus contrast. At the same time their contrast response function saturates or even super-saturates for high values of contrast. When two bar stimuli are presented within their classical receptive field, the neuronal response decreases with the intersection angle. When two stimuli are presented inside and outside the classical receptive field, the response of the cell increases with the intersection angle. Both cats and monkeys show iso-orientation suppression, which has sometimes been reported to be combined with cross-orientation facilitation. This property has previously been described as sensitivity to orientation contrast. We address the emergence of these effects with a model that describes the processing of geniculocortical signals through cortical circuitry. We hypothesize that short intracortical fibers mediate the classical receptive field effects, whereas long-range collaterals evoke contextual effects such as sensitivity to orientation contrast. We model this situation by setting up a mean-field description of two neighboring cortical hypercolumns, which can process a nonoverlapping center and a (nonclassical) surround stimulus. Both hypercolumns interact via idealized long-range connections. For an isolated model hypercolumn, we find that either contrast saturation or contrast-invariant orientation tuning emerges, depending on the strength of the lateral excitation. There is no parameter regime, however, where both phenomena emerge simultaneously. In the regime where contrast saturation is found, the model also correctly reproduces suppression due to a second, cross-oriented grid within the classical receptive field. If two model hypercolumns are mutually coupled by long-range connections that are iso-orientation specific, nonclassical surround stimuli show either suppression or facilitation for all surround orientations. Sensitivity to orientation contrast is not observed. This property requires excitatory-to-excitatory long-range couplings that are less orientation specific than those targeting inhibitory neurons.

The effect of intracortical competition on the formation of topographic maps in models of Hebbian learning.

Correlation-based learning (CBL) models and self-organizing maps (SOM) are two classes of Hebbian models that have both been proposed to explain the activity-driven formation of cortical maps. Both models differ significantly in the way lateral cortical interactions are treated, leading to different predictions for the formation of receptive fields. The linear CBL models predict that receptive field profiles are determined by the average values and the spatial correlations of the second order of the afferent activity patterns, whereas SOM models map stimulus features. Here, we investigate a class of models which are characterized by a variable degree of lateral competition and which have the CBL and SOM models as limit cases. We show that there exists a critical value for intracortical competition below which the model exhibits CBL properties and above which feature mapping sets in. The class of models is then analyzed with respect to the formation of topographic maps between two layers of neurons. For Gaussian input stimuli we find that localized receptive fields and topographic maps emerge above the critical value for intracortical competition, and we calculate this value as a function of the size of the input stimuli and the range of the lateral interaction function. Additionally, we show that the learning rule can be derived via the optimization of a global cost function in a framework of probabilistic output neurons which represent a set of input stimuli by a sparse code.

Contrast adaptation and infomax in visual cortical neurons.

In the primary visual cortex (V1) the contrast response function of many neurons saturates at high contrast and adapts depending on the visual stimulus. We propose that both effects - contrast saturation and adaptation - can be explained by a fast and a slow component in the synaptic dynamics. In our model the saturation is an effect of fast synaptic depression with a recovery time constant of about 200 ms. Fast synaptic depression leads to a contrast response function with a high gain for only a limited range of contrast values. Furthermore, we propose that slow adaptation of the transmitter release probability at the geniculocortical synapses is the underlying neural mechanism that accounts for contrast adaptation on a time scale of about 7 sec. For the functional role of contrast adaptation we make the hypothesis that it serves to achieve the best visual cortical representation of the geniculate input. This representation should maximize the mutual information between the cortical activity and the geniculocortical input by increasing the release probability in a low contrast environment. We derive an adaptation rule for the transmitter release probability based on this infomax principle. We show that changes in the transmitter release probability may compensate for changes in the variance of the geniculate inputs - an essential requirement for contrast adaptation. Also, we suggest that increasing the release probability in a low contrast environment is beneficial for signal extraction, because neurons remain sensitive only to an increase in the presynaptic activity if it is synchronous and, therefore, likely to be stimulus related. Our hypotheses are tested in numerical simulations of a network of integrate-and-fire neurons for one column of V1 using fast synaptic depression and slow synaptic adaptation. The simulations show that changing the synaptic release probability of the geniculocortical synapses is a better model for contrast adaptation than the adaptation of the synaptic weights: only in the case of changing the transmitter release probability does our model reproduce the experimental finding that the average membrane potential (DC component) adapts much more strongly than the stimulus modulated component (F1 component). In the case of changing the synaptic weights, however, the average membrane potential (DC) as well as the stimulus modulated component (F1 component) would adapt. Furthermore, changing the release probability at the recurrent cortical synapses cannot account for contrast adaptation, but could be responsible for establishing oscillatory activity often observed in recordings from visual cortical cells.

A model for the intracortical origin of orientation preference and tuning in macaque striate cortex.

We report results of numerical simulations for a model of generation of orientation selectivity in macaque striate cortex. In contrast to previous models, where the initial orientation bias is generated by convergent geniculate input to simple cells and subsequently sharpened by lateral circuits, our approach is based on anisotropic intracortical excitatory connections which provide both the initial orientation bias and its subsequent amplification. Our study shows that the emerging response properties are similar to the response properties that are observed experimentally, hence the hypothesis of an intracortical generation of orientation bias is a sensible alternative to the notion of an afferent bias by convergent geniculocortical projection patterns. In contrast to models based on an afferent orientation bias, however, the "intracortical hypothesis" predicts that orientation tuning gradually evolves from an initially nonoriented response and a complete loss of orientation tuning when the recurrent excitation is blocked, but new experiments must be designed to unambiguously decide between both hypotheses.

A model for the depth-dependence of receptive field size and contrast sensitivity of cells in layer 4C of macaque striate cortex.

A model of LGN-input to layer 4C of macaque primary visual cortex has been used to test the hypothesis that feedforward convergence of P- and M-inputs onto layer 4C spiny stellate neurons is sufficient to explain the observed gradual change in receptive field size and contrast sensitivity with depth in the layer. Overlap of dendrites of postsynaptic neurons between M- and P-input zones proved sufficient to explain change in the lower two-thirds of layer 4C, while more rapid change in upper 4C was matched by proposing two different M-inputs with partial overlap in upper 4C alpha.

A stochastic self-organizing map for proximity data.

We derive an efficient algorithm for topographic mapping of proximity data (TMP), which can be seen as an extension of Kohonen's self-organizing map to arbitrary distance measures. The TMP cost function is derived in a Baysian framework of folded Markov chains for the description of autoencoders. It incorporates the data by a dissimilarity matrix D and the topographic neighborhood by a matrix H of transition probabilities. From the principle of maximum entropy, a nonfactorizing Gibbs distribution is obtained, which is approximated in a mean-field fashion. This allows for maximum likelihood estimation using an expectation-maximization algorithm. In analogy to the transition from topographic vector quantization to the self-organizing map, we suggest an approximation to TMP that is computationally more efficient. In order to prevent convergence to local minima, an annealing scheme in the temperature parameter is introduced, for which the critical temperature of the first phase transition is calculated in terms of D and H. Numerical results demonstrate the working of the algorithm and confirm the analytical results. Finally, the algorithm is used to generate a connection map of areas of the cat's cerebral cortex.

Simulation of scanning laser techniques for optical imaging of blood-related intrinsic signals.

Optical imaging of intrinsic signals detects neural activation patterns by taking video images of the local activity-related changes in the light intensity reflected from neural tissue (intrinsic signals). At red light (605 nm), these signals are caused mainly by local variations of the tissue absorption following deoxygenation of blood. We characterize the image generation process during optical imaging by Monte Carlo simulations of light propagation through a homogeneous model tissue equipped with a local absorber. Conventional video imaging and scanning laser imaging are compared. We find that, compared with video imaging, scanning laser techniques drastically increase both the contrast and the lateral resolution of optical recordings. Also, the maximum depth up to which the signals can be detected is increased by roughly a factor of 2 when scanning laser optical imaging is used. Further, the radial profile of the diffuse-reflectance pattern for each pixel is subject to changes that correlate with the depth of the absorber within the tissue. We suggest a detection geometry for the online measurement of these radial profiles that can be realized by modifying a standard scanning laser ophthalmoscope.

Unspecific long-term potentiation can evoke functional segregation in a model of area 17.

It has been shown recently in rat hippocampus that the synapse specificity of Hebbian long-term potentiation breaks down at short distances (< 100 microm). Using a neural network model we show that this unspecific component of long-term potentiation can be responsible for the robust formation and maintainance of cortical organization during activity-driven development. When the model is applied to the formation of orientation and ocular dominance in visual cortex, addition of an unspecific component to standard Hebbian learning, in combination with a tendency of left-eye and right-eye driven synapses to initially group together on the postsynaptic neuron, induces the simultaneous emergence and stabilization of ocular dominance and of segregated, oriented ON/OFF subfields. Since standard Hebbian learning cannot account for the simultaneous stabilization of both structures, unspecific LTP thus induces a qualitatively new behaviour. Since unspecific LTP only acts between synapses which are locally clustered in space, our results imply that details of the local grouping of synapses on the dendritic arbors of postsynaptic cells can considerably influence the formation of the cortical functional organization at the systems level.