Dynamic reinforcement learning reveals time-dependent shifts in strategy during reward learning.

Different brain systems have been hypothesized to subserve multiple "experts" that compete to generate behavior. In reinforcement learning, two general processes, one model-free (MF) and one model-based (MB), are often modeled as a mixture of agents (MoA) and hypothesized to capture differences between automaticity vs. deliberation. However, shifts in strategy cannot be captured by a static MoA. To investigate such dynamics, we present the mixture-of-agents hidden Markov model (MoA-HMM), which simultaneously learns inferred action values from a set of agents and the temporal dynamics of underlying "hidden" states that capture shifts in agent contributions over time. Applying this model to a multi-step,reward-guided task in rats reveals a progression of within-session strategies: a shift from initial MB exploration to MB exploitation, and finally to reduced engagement. The inferred states predict changes in both response time and OFC neural encoding during the task, suggesting that these states are capturing real shifts in dynamics.

Value representations in the rodent orbitofrontal cortex drive learning, not choice.

Humans and animals make predictions about the rewards they expect to receive in different situations. In formal models of behavior, these predictions are known as value representations, and they play two very different roles. Firstly, they drive choice: the expected values of available options are compared to one another, and the best option is selected. Secondly, they support learning: expected values are compared to rewards actually received, and future expectations are updated accordingly. Whether these different functions are mediated by different neural representations remains an open question. Here, we employ a recently developed multi-step task for rats that computationally separates learning from choosing. We investigate the role of value representations in the rodent orbitofrontal cortex, a key structure for value-based cognition. Electrophysiological recordings and optogenetic perturbations indicate that these representations do not directly drive choice. Instead, they signal expected reward information to a learning process elsewhere in the brain that updates choice mechanisms.

Deep Reinforcement Learning and Its Neuroscientific Implications.

The emergence of powerful artificial intelligence (AI) is defining new research directions in neuroscience. To date, this research has focused largely on deep neural networks trained using supervised learning in tasks such as image classification. However, there is another area of recent AI work that has so far received less attention from neuroscientists but that may have profound neuroscientific implications: deep reinforcement learning (RL). Deep RL offers a comprehensive framework for studying the interplay among learning, representation, and decision making, offering to the brain sciences a new set of research tools and a wide range of novel hypotheses. In the present review, we provide a high-level introduction to deep RL, discuss some of its initial applications to neuroscience, and survey its wider implications for research on brain and behavior, concluding with a list of opportunities for next-stage research.

Habits without values.

Habits form a crucial component of behavior. In recent years, key computational models have conceptualized habits as arising from model-free reinforcement learning mechanisms, which typically select between available actions based on the future value expected to result from each. Traditionally, however, habits have been understood as behaviors that can be triggered directly by a stimulus, without requiring the animal to evaluate expected outcomes. Here, we develop a computational model instantiating this traditional view, in which habits develop through the direct strengthening of recently taken actions rather than through the encoding of outcomes. We demonstrate that this model accounts for key behavioral manifestations of habits, including insensitivity to outcome devaluation and contingency degradation, as well as the effects of reinforcement schedule on the rate of habit formation. The model also explains the prevalent observation of perseveration in repeated-choice tasks as an additional behavioral manifestation of the habit system. We suggest that mapping habitual behaviors onto value-free mechanisms provides a parsimonious account of existing behavioral and neural data. This mapping may provide a new foundation for building robust and comprehensive models of the interaction of habits with other, more goal-directed types of behaviors and help to better guide research into the neural mechanisms underlying control of instrumental behavior more generally. (PsycINFO Database Record (c) 2019 APA, all rights reserved).

Author Correction: Dorsal hippocampus contributes to model-based planning.

In the version of this article initially published, the green label in Fig. 1c read "rightward choices" instead of "leftward choices." The error has been corrected in the HTML and PDF versions of the article.

Silicon waveguide optical switch with embedded phase change material.

Phase-change materials (PCMs) have emerged as promising active elements in silicon (Si) photonic systems. In this work, we design, fabricate, and characterize a hybrid Si-PCM optical switch. By integrating vanadium dioxide (a PCM) within a Si photonic waveguide, in a non-resonant geometry, we achieve ~10 dB broadband optical contrast with a PCM length of 500 nm using thermal actuation.

Dorsal hippocampus contributes to model-based planning.

Planning can be defined as action selection that leverages an internal model of the outcomes likely to follow each possible action. Its neural mechanisms remain poorly understood. Here we adapt recent advances from human research for rats, presenting for the first time an animal task that produces many trials of planned behavior per session, making multitrial rodent experimental tools available to study planning. We use part of this toolkit to address a perennially controversial issue in planning: the role of the dorsal hippocampus. Although prospective hippocampal representations have been proposed to support planning, intact planning in animals with damaged hippocampi has been repeatedly observed. Combining formal algorithmic behavioral analysis with muscimol inactivation, we provide causal evidence directly linking dorsal hippocampus with planning behavior. Our results and methods open the door to new and more detailed investigations of the neural mechanisms of planning in the hippocampus and throughout the brain.

Effects of Early Smoking Habits on Young Adult Female Voices in Greece.

Cigarette use is a preventable cause of mortality and diseases. The World Health Organization states that Europe and especially Greece has the highest occurrence of smoking among adults. The prevalence of smoking among women in Greece was estimated to be over 30% in 2012. Smoking is a risk factor for many diseases. Studies have demonstrated the association between smoking and laryngeal pathologies as well as changes in voice characteristics. The purpose of this study was to estimate the effect of early smoking habit on young adult female voices and if they perceive any vocal changes using two assessment methods. The Voice Handicap Index and the acoustic analyses of voice measurements were used, with both serving as mini-assessment protocols. Two hundred and ten young females (110 smokers and 100 nonsmokers) attending the Technological Educational Institute of Epirus in the School of Health and Welfare were included. Statistically significant increases for physical and total scores of the Voice Handicap Index were found in the smokers group (P < 0.05). Significant changes were observed for the acoustic parameters between smoker and nonsmoker groups. The results of this study indicated observable signs of change in the voice acoustic characteristics of young adults with early smoking habits.

Plasmon enhanced water splitting mediated by hybrid bimetallic Au-Ag core-shell nanostructures.

In this work, we employed wet chemically synthesized bimetallic Au-Ag core-shell nanostructures (Au-AgNSs) to enhance the photocurrent density of mesoporous TiO2 for water splitting and we compared the results with monometallic Au nanoparticles (AuNPs). While Au-AgNSs incorporated photoanodes give rise to 14× enhancement in incident photon to charge carrier efficiency, AuNPs embedded photoanodes result in 6× enhancement. By varying nanoparticle concentration in the photoanodes, we observed ∼245× less Au-AgNSs are required relative to AuNPs to generate similar photocurrent enhancement for solar fuel conversion. Power-dependent measurements of Au-AgNSs and AuNPs showed a first order dependence to incident light intensity, relative to half-order dependence for TiO2 only photoanodes. This indicated that plasmonic nanostructures enhance charge carriers formed on the surface of the TiO2 which effectively participate in photochemical reactions. Our experiments and simulations suggest the enhanced near-field, far-field, and multipolar resonances of Au-AgNSs facilitating broadband absorption of solar radiation collectively gives rise to their superior performance in water splitting.

Unconscious neural processing differs with method used to render stimuli invisible.

Visual stimuli can be kept from awareness using various methods. The extent of processing that a given stimulus receives in the absence of awareness is typically used to make claims about the role of consciousness more generally. The neural processing elicited by a stimulus, however, may also depend on the method used to keep it from awareness, and not only on whether the stimulus reaches awareness. Here we report that the method used to render an image invisible has a dramatic effect on how category information about the unseen stimulus is encoded across the human brain. We collected fMRI data while subjects viewed images of faces and tools, that were rendered invisible using either continuous flash suppression (CFS) or chromatic flicker fusion (CFF). In a third condition, we presented the same images under normal fully visible viewing conditions. We found that category information about visible images could be extracted from patterns of fMRI responses throughout areas of neocortex known to be involved in face or tool processing. However, category information about stimuli kept from awareness using CFS could be recovered exclusively within occipital cortex, whereas information about stimuli kept from awareness using CFF was also decodable within temporal and frontal regions. We conclude that unconsciously presented objects are processed differently depending on how they are rendered subjectively invisible. Caution should therefore be used in making generalizations on the basis of any one method about the neural basis of consciousness or the extent of information processing without consciousness.