Distinct replay signatures for prospective decision-making and memory preservation.

Theories of neural replay propose that it supports a range of functions, most prominently planning and memory consolidation. Here, we test the hypothesis that distinct signatures of replay in the same task are related to model-based decision-making ("planning") and memory preservation. We designed a reward learning task wherein participants utilized structure knowledge for model-based evaluation, while at the same time had to maintain knowledge of two independent and randomly alternating task environments. Using magnetoencephalography and multivariate analysis, we first identified temporally compressed sequential reactivation, or replay, both prior to choice and following reward feedback. Before choice, prospective replay strength was enhanced for the current task-relevant environment when a model-based planning strategy was beneficial. Following reward receipt, and consistent with a memory preservation role, replay for the alternative distal task environment was enhanced as a function of decreasing recency of experience with that environment. Critically, these planning and memory preservation relationships were selective to pre-choice and post-feedback periods, respectively. Our results provide support for key theoretical proposals regarding the functional role of replay and demonstrate that the relative strength of planning and memory-related signals are modulated by ongoing computational and task demands.

Reward learning and working memory: Effects of massed versus spaced training and post-learning delay period.

Neuroscience research has illuminated the mechanisms supporting learning from reward feedback, demonstrating a critical role for the striatum and midbrain dopamine system. However, in humans, short-term working memory that is dependent on frontal and parietal cortices can also play an important role, particularly in commonly used paradigms in which learning is relatively condensed in time. Given the growing use of reward-based learning tasks in translational studies in computational psychiatry, it is important to understand the extent of the influence of working memory and also how core gradual learning mechanisms can be better isolated. In our experiments, we manipulated the spacing between repetitions along with a post-learning delay preceding a test phase. We found that learning was slower for stimuli repeated after a long delay (spaced-trained) compared to those repeated immediately (massed-trained), likely reflecting the remaining contribution of feedback learning mechanisms when working memory is not available. For massed learning, brief interruptions led to drops in subsequent performance, and individual differences in working memory capacity positively correlated with overall performance. Interestingly, when tested after a delay period but not immediately, relative preferences decayed in the massed condition and increased in the spaced condition. Our results provide additional support for a large role of working memory in reward-based learning in temporally condensed designs. We suggest that spacing training within or between sessions is a promising approach to better isolate and understand mechanisms supporting gradual reward-based learning, with particular importance for understanding potential learning dysfunctions in addiction and psychiatric disorders.

Reactivation of Single-Episode Pain Patterns in the Hippocampus and Decision Making.

Aversive and rewarding experiences can exert a strong influence on subsequent behavior. While decisions are often supported by the value of single past episodes, most research has focused on the role of well-learned value associations. Recent studies have begun to investigate the influence of reward-associated episodes, but it is unclear whether these results generalize to negative experiences, such as pain. To investigate whether and how the value of previous aversive experiences modulates behavior and brain activity, in our experiments female and male human participants experienced episodes of high or low pain in conjunction with incidental, trial-unique neutral pictures. In an incentive-compatible surprise test phase, we found that participants avoided pain-paired objects. In a separate fMRI experiment, at test, participants exhibited significant pain value memory. Neurally, when participants were re-exposed to pain-paired objects, we found no evidence for reactivation of pain-related patterns in pain-responsive regions, such as the anterior insula. Critically, however, we found significant reactivation of pain-related patterns of activity in the hippocampus, such that activity significantly discriminated high versus low pain episodes. Further, stronger reactivation in the anterior hippocampus was related to improved pain value memory performance. Our results demonstrate that single incidental aversive experiences can build memories that affect decision-making and that this influence may be supported by the hippocampus.SIGNIFICANCE STATEMENT Aversive and rewarding experiences can exert a strong influence on our subsequent behavior. While decisions are often supported by single past negative or positive episodes, most research has focused on the role of well-learned value associations. In experiments using aversive heat pain in conjunction with incidental objects, we found that participants' choices were biased by the level of pain associated with the objects. Further, when participants saw the objects again, pain-related neural patterns in the hippocampus were re-expressed, and this was related to pain value memory performance. These results suggest a mechanism by which even single negative experiences can guide our later decisions.

Episodic memory retrieval success is associated with rapid replay of episode content.

Retrieval of everyday experiences is fundamental for informing our future decisions. The fine-grained neurophysiological mechanisms that support such memory retrieval are largely unknown. We studied participants who first experienced, without repetition, unique multicomponent 40-80-s episodes. One day later, they engaged in cued retrieval of these episodes while undergoing magnetoencephalography. By decoding individual episode elements, we found that trial-by-trial successful retrieval was supported by the sequential replay of episode elements, with a temporal compression factor of >60. The direction of replay supporting retrieval, either backward or forward, depended on whether the task goal was to retrieve elements of an episode that followed or preceded, respectively, a retrieval cue. This sequential replay was weaker in very-high-performing participants, in whom instead we found evidence for simultaneous clustered reactivation. Our results demonstrate that memory-mediated decisions are supported by a rapid replay mechanism that can flexibly shift in direction in response to task goals.

Periodontitis and Cardiovascular Diseases. Consensus Report.

Background: In Europe cardiovascular disease (CVD) is responsible for 3.9 million deaths (45% of deaths), being ischaemic heart disease, stroke, hypertension (leading to heart failure) the major cause of these CVD related deaths. Periodontitis is also a chronic non-communicable disease (NCD) with a high prevalence, being severe periodontitis, affecting 11.2% of the world's population, the sixth most common human disease. Material and Methods: There is now a significant body of evidence to support independent associations between severe periodontitis and several NCDs, in particular CVD. In 2012 a joint workshop was held between the European Federation of Periodontology (EFP) and the American Academy of Periodontology to review the literature relating periodontitis and systemic diseases, including CVD. In the last five years important new scientific information has emerged providing important emerging evidence to support these associations. Results and Conclusions: The present review reports the proceedings of the workshop jointly organised by the EFP and the World Heart Federation (WHF), which has updated the existing epidemiological evidence for significant associations between periodontitis and CVD, the mechanistic links and the impact of periodontal therapy on cardiovascular and surrogate outcomes. This review has also focused on the potential risk and complications of periodontal therapy in patients on anti thrombotic therapy and has made recommendations for dentists, physicians and for patients visiting both the dental and medical practices.

Central systolic blood pressure increases with aortic stiffness.

OBJECTIVES: Central systolic blood pressure (CSBP) is the pressure in the root of aorta, which directly influences organs such as brain, heart and kidneys and is related to organ damage. Its value increases with the aortic stiffness. The aim of this study was to analyze the relationships of CSBP to aortic stiffness parameters. METHODS: Central blood pressure (BP) and related parameters were measured by Arteriograph, working based on oscillometric principle, using pulse wave analysis (PWA) approach. We examined 123 patients (69 females, 54 males) with a primary hypertension. RESULTS: Using a linear correlation analysis, we found that CSBP was correlated to aortic pulse wave velocity (PWV), aortic and brachial pulse pressure (PP), aortic augmentation index, return time of reflected pressure wave (RT) and aortic and brachial augmentation indexes. Multivariate analysis defines the aortic pulse pressure (PPao) as the most powerful parameter influencing CSBP. By an individual analysis of BP in each patient separately, we defined two different types of central hemodynamics; those with a higher CSBP than brachial SBP occur in stiffer aorta. CONCLUSION: The CSBP increases with aortic PP, the most powerful stiffness parameter of aorta. Higher CSBP than brachial SBP usually accompanies a stiffer aorta (Tab. 5, Ref. 19).

Learning of distant state predictions by the orbitofrontal cortex in humans.

Representations of our future environment are essential for planning and decision making. Previous research in humans has demonstrated that the hippocampus is a critical region for forming and retrieving associations, while the medial orbitofrontal cortex (OFC) is an important region for representing information about recent states. However, it is not clear how the brain acquires predictive representations during goal-directed learning. Here, we show using fMRI that while participants learned to find rewards in multiple different Y-maze environments, hippocampal activity was highest during initial exposure and then decayed across the remaining repetitions of each maze, consistent with a role in rapid encoding. Importantly, multivariate patterns in the OFC-VPFC came to represent predictive information about upcoming states approximately 30 s in the future. Our findings provide a mechanism by which the brain can build models of the world that span long-timescales to make predictions.

Retroactive and graded prioritization of memory by reward.

Many decisions are based on an internal model of the world. Yet, how such a model is constructed from experience and represented in memory remains unknown. We test the hypothesis that reward shapes memory for sequences of events by retroactively prioritizing memory for objects as a function of their distance from reward. Human participants encountered neutral objects while exploring a series of mazes for reward. Across six data sets, we find that reward systematically modulates memory for neutral objects, retroactively prioritizing memory for objects closest to the reward. This effect of reward on memory emerges only after a 24-hour delay and is stronger for mazes followed by a longer rest interval, suggesting a role for post-reward replay and overnight consolidation, as predicted by neurobiological data in animals. These findings demonstrate that reward retroactively prioritizes memory along a sequential gradient, consistent with the role of memory in supporting adaptive decision-making.

Reward Learning over Weeks Versus Minutes Increases the Neural Representation of Value in the Human Brain.

Over the past few decades, neuroscience research has illuminated the neural mechanisms supporting learning from reward feedback. Learning paradigms are increasingly being extended to study mood and psychiatric disorders as well as addiction. However, one potentially critical characteristic that this research ignores is the effect of time on learning: human feedback learning paradigms are usually conducted in a single rapidly paced session, whereas learning experiences in ecologically relevant circumstances and in animal research are almost always separated by longer periods of time. In our experiments, we examined reward learning in short condensed sessions distributed across weeks versus learning completed in a single "massed" session in male and female participants. As expected, we found that after equal amounts of training, accuracy was matched between the spaced and massed conditions. However, in a 3-week follow-up, we found that participants exhibited significantly greater memory for the value of spaced-trained stimuli. Supporting a role for short-term memory in massed learning, we found a significant positive correlation between initial learning and working memory capacity. Neurally, we found that patterns of activity in the medial temporal lobe and prefrontal cortex showed stronger discrimination of spaced- versus massed-trained reward values. Further, patterns in the striatum discriminated between spaced- and massed-trained stimuli overall. Our results indicate that single-session learning tasks engage partially distinct learning mechanisms from distributed training. Our studies begin to address a large gap in our knowledge of human learning from reinforcement, with potential implications for our understanding of mood disorders and addiction.SIGNIFICANCE STATEMENT Humans and animals learn to associate predictive value with stimuli and actions, and these values then guide future behavior. Such reinforcement-based learning often happens over long time periods, in contrast to most studies of reward-based learning in humans. In experiments that tested the effect of spacing on learning, we found that associations learned in a single massed session were correlated with short-term memory and significantly decayed over time, whereas associations learned in short massed sessions over weeks were well maintained. Additionally, patterns of activity in the medial temporal lobe and prefrontal cortex discriminated the values of stimuli learned over weeks but not minutes. These results highlight the importance of studying learning over time, with potential applications to drug addiction and psychiatry.

Automated classification of celiac disease during upper endoscopy: Status quo and quo vadis.

A large amount of digital image material is routinely captured during esophagogastroduodenoscopies but, for the most part, is not used for confirming the diagnosis process of celiac disease which is primarily based on histological examination of biopsies. Recently, considerable effort has been undertaken to make use of image material by developing semi- or fully-automated systems to improve the diagnostic workup. Recently, focus was especially laid on developing state-of-the-art deep learning architectures, exploiting the endoscopist's expert knowledge and on making systems fully automated and thereby completely observer independent. In this work, we summarize recent trends in the field of computer-aided celiac disease diagnosis based on upper endoscopy and discuss about recent progress, remaining challenges, limitations currently prohibiting a deployment in clinical practice and future efforts to tackle them.

Pain perception during debridement of hypersensitive teeth elicited by two ultrasonic scalers.

OBJECTIVES: The ultrasonic NO PAIN technology (Electro Medical Systems, Nyon, CH) promises minimal pain during debridement due to linear oscillating action combined with a sinusoidal power output and feedback control. The aim of the present study was to measure pain perception on a visual analogue scale (VAS) during supportive periodontal therapy including debridement of hypersensitive teeth. Two ultrasonic scalers were used, one with and one without NO PAIN technology. MATERIAL AND METHODS: In a randomized-controlled clinical study with split-mouth design, 100 hypersensitive teeth matched for air blast hypersensitivity were either treated with the ultrasonic device Piezon Master 700 or the Mini Piezon (both EMS, Nyon, CH). Pain perception during debridement was assessed by a VAS (range 0-10). RESULTS: The average VAS for the test device Piezon Master 700 with NO PAIN technology was 3.16 ± 2.10, and for the control device Mini Piezon without NO PAIN technology 3.40 ± 2.59 (p = 0.490). Placing an arbitrary threshold at the VAS score of 3 for significant pain experience, 60 % of the subjects experienced no significant pain with either instrument. CONCLUSION: No statistically significant difference in perceived pain between the instruments used was found. CLINICAL RELEVANCE: Both ultrasonic devices showed very small pain intensities during debridement of highly hypersensitive teeth and can therefore be recommended for supportive periodontal therapy.

Social Cognition as Reinforcement Learning: Feedback Modulates Emotion Inference.

Neuroscientific studies of social cognition typically employ paradigms in which perceivers draw single-shot inferences about the internal states of strangers. Real-world social inference features much different parameters: People often encounter and learn about particular social targets (e.g., friends) over time and receive feedback about whether their inferences are correct or incorrect. Here, we examined this process and, more broadly, the intersection between social cognition and reinforcement learning. Perceivers were scanned using fMRI while repeatedly encountering three social targets who produced conflicting visual and verbal emotional cues. Perceivers guessed how targets felt and received feedback about whether they had guessed correctly. Visual cues reliably predicted one target's emotion, verbal cues predicted a second target's emotion, and neither reliably predicted the third target's emotion. Perceivers successfully used this information to update their judgments over time. Furthermore, trial-by-trial learning signals-estimated using two reinforcement learning models-tracked activity in ventral striatum and ventromedial pFC, structures associated with reinforcement learning, and regions associated with updating social impressions, including TPJ. These data suggest that learning about others' emotions, like other forms of feedback learning, relies on domain-general reinforcement mechanisms as well as domain-specific social information processing.

Reactivation of Reward-Related Patterns from Single Past Episodes Supports Memory-Based Decision Making.

Rewarding experiences exert a strong influence on later decision making. While decades of neuroscience research have shown how reinforcement gradually shapes preferences, decisions are often influenced by single past experiences. Surprisingly, relatively little is known about the influence of single learning episodes. Although recent work has proposed a role for episodes in decision making, it is largely unknown whether and how episodic experiences contribute to value-based decision making and how the values of single episodes are represented in the brain. In multiple behavioral experiments and an fMRI experiment, we tested whether and how rewarding episodes could support later decision making. Participants experienced episodes of high reward or low reward in conjunction with incidental, trial-unique neutral pictures. In a surprise test phase, we found that participants could indeed remember the associated level of reward, as evidenced by accurate source memory for value and preferences to re-engage with rewarded objects. Further, in a separate experiment, we found that high-reward objects shown as primes before a gambling task increased financial risk taking. Neurally, re-exposure to objects in the test phase led to significant reactivation of reward-related patterns. Importantly, individual variability in the strength of reactivation predicted value memory performance. Our results provide a novel demonstration that affect-related neural patterns are reactivated during later experience. Reactivation of value information represents a mechanism by which memory can guide decision making.

Upregulation of osteoprotegerin expression correlates with bone invasion and predicts poor clinical outcome in oral cancer.

OBJECTIVES: We aimed to determine the prognostic significance of receptor activator of nuclear factor kappa-B ligand (RANKL), RANK and osteoprotegerin (OPG) in patients with oral squamous cell carcinoma (OSCC). MATERIALS AND METHODS: The protein expression of RANKL, RANK and OPG was assessed by immunohistochemistry on pretreatment biopsies of 93 patients with locally advanced OSCC who received preoperative chemoradiotherapy (CRT). The primary endpoint was cancer-specific survival. Secondary endpoints were correlation of biomarkers with bone invasion and pathological tumor response. Kaplan-Meier curves and Cox regression models were used for survival analyses. RESULTS: A significantly higher OPG expression was demonstrated in patients with malignant bone invasion and non-responders to CRT as compared to patients without bone invasion and responders (p=0.032 and p=0.033, respectively). Multivariate analysis revealed that higher OPG expression was independently associated with shorter cancer-specific survival (p=0.04). The expression status of RANKL and RANK was not significantly related to clinicopathological characteristics and had no impact on survival of OSCC patients. CONCLUSION: Upregulation of OPG expression is associated with bone invasion, poor pathological tumor regression to neoadjuvant CRT, and worse long-term cancer-specific survival in patients with locally advanced OSCC. Our results indicate that OPG may be a novel prognostic biomarker in oral cancer.

Episodic memory encoding interferes with reward learning and decreases striatal prediction errors.

Learning is essential for adaptive decision making. The striatum and its dopaminergic inputs are known to support incremental reward-based learning, while the hippocampus is known to support encoding of single events (episodic memory). Although traditionally studied separately, in even simple experiences, these two types of learning are likely to co-occur and may interact. Here we sought to understand the nature of this interaction by examining how incremental reward learning is related to concurrent episodic memory encoding. During the experiment, human participants made choices between two options (colored squares), each associated with a drifting probability of reward, with the goal of earning as much money as possible. Incidental, trial-unique object pictures, unrelated to the choice, were overlaid on each option. The next day, participants were given a surprise memory test for these pictures. We found that better episodic memory was related to a decreased influence of recent reward experience on choice, both within and across participants. fMRI analyses further revealed that during learning the canonical striatal reward prediction error signal was significantly weaker when episodic memory was stronger. This decrease in reward prediction error signals in the striatum was associated with enhanced functional connectivity between the hippocampus and striatum at the time of choice. Our results suggest a mechanism by which memory encoding may compete for striatal processing and provide insight into how interactions between different forms of learning guide reward-based decision making.

Representation of aversive prediction errors in the human periaqueductal gray.

Pain is a primary driver of learning and motivated action. It is also a target of learning, as nociceptive brain responses are shaped by learning processes. We combined an instrumental pain avoidance task with an axiomatic approach to assessing fMRI signals related to prediction errors (PEs), which drive reinforcement-based learning. We found that pain PEs were encoded in the periaqueductal gray (PAG), a structure important for pain control and learning in animal models. Axiomatic tests combined with dynamic causal modeling suggested that ventromedial prefrontal cortex, supported by putamen, provides an expected value-related input to the PAG, which then conveys PE signals to prefrontal regions important for behavioral regulation, including orbitofrontal, anterior mid-cingulate and dorsomedial prefrontal cortices. Thus, pain-related learning involves distinct neural circuitry, with implications for behavior and pain dynamics.

Transfer of learning relates to intrinsic connectivity between hippocampus, ventromedial prefrontal cortex, and large-scale networks.

An important aspect of adaptive learning is the ability to flexibly use past experiences to guide new decisions. When facing a new decision, some people automatically leverage previously learned associations, while others do not. This variability in transfer of learning across individuals has been demonstrated repeatedly and has important implications for understanding adaptive behavior, yet the source of these individual differences remains poorly understood. In particular, it is unknown why such variability in transfer emerges even among homogeneous groups of young healthy participants who do not vary on other learning-related measures. Here we hypothesized that individual differences in the transfer of learning could be related to relatively stable differences in intrinsic brain connectivity, which could constrain how individuals learn. To test this, we obtained a behavioral measure of memory-based transfer outside of the scanner and on a separate day acquired resting-state functional MRI images in 42 participants. We then analyzed connectivity across independent component analysis-derived brain networks during rest, and tested whether intrinsic connectivity in learning-related networks was associated with transfer. We found that individual differences in transfer were related to intrinsic connectivity between the hippocampus and the ventromedial prefrontal cortex, and between these regions and large-scale functional brain networks. Together, the findings demonstrate a novel role for intrinsic brain dynamics in flexible learning-guided behavior, both within a set of functionally specific regions known to be important for learning, as well as between these regions and the default and frontoparietal networks, which are thought to serve more general cognitive functions.

Preference by association: how memory mechanisms in the hippocampus bias decisions.

Every day people make new choices between alternatives that they have never directly experienced. Yet, such decisions are often made rapidly and confidently. Here, we show that the hippocampus, traditionally known for its role in building long-term declarative memories, enables the spread of value across memories, thereby guiding decisions between new choice options. Using functional brain imaging in humans, we discovered that giving people monetary rewards led to activation of a preestablished network of memories, spreading the positive value of reward to nonrewarded items stored in memory. Later, people were biased to choose these nonrewarded items. This decision bias was predicted by activity in the hippocampus, reactivation of associated memories, and connectivity between memory and reward regions in the brain. These findings explain how choices among new alternatives emerge automatically from the associative mechanisms by which the brain builds memories. Further, our findings demonstrate a previously unknown role for the hippocampus in value-based decisions.

Generalization of value in reinforcement learning by humans.

Research in decision-making has focused on the role of dopamine and its striatal targets in guiding choices via learned stimulus-reward or stimulus-response associations, behavior that is well described by reinforcement learning theories. However, basic reinforcement learning is relatively limited in scope and does not explain how learning about stimulus regularities or relations may guide decision-making. A candidate mechanism for this type of learning comes from the domain of memory, which has highlighted a role for the hippocampus in learning of stimulus-stimulus relations, typically dissociated from the role of the striatum in stimulus-response learning. Here, we used functional magnetic resonance imaging and computational model-based analyses to examine the joint contributions of these mechanisms to reinforcement learning. Humans performed a reinforcement learning task with added relational structure, modeled after tasks used to isolate hippocampal contributions to memory. On each trial participants chose one of four options, but the reward probabilities for pairs of options were correlated across trials. This (uninstructed) relationship between pairs of options potentially enabled an observer to learn about option values based on experience with the other options and to generalize across them. We observed blood oxygen level-dependent (BOLD) activity related to learning in the striatum and also in the hippocampus. By comparing a basic reinforcement learning model to one augmented to allow feedback to generalize between correlated options, we tested whether choice behavior and BOLD activity were influenced by the opportunity to generalize across correlated options. Although such generalization goes beyond standard computational accounts of reinforcement learning and striatal BOLD, both choices and striatal BOLD activity were better explained by the augmented model. Consistent with the hypothesized role for the hippocampus in this generalization, functional connectivity between the ventral striatum and hippocampus was modulated, across participants, by the ability of the augmented model to capture participants' choice. Our results thus point toward an interactive model in which striatal reinforcement learning systems may employ relational representations typically associated with the hippocampus.