Having multiple selves helps learning agents explore and adapt in complex changing worlds.

Satisfying a variety of conflicting needs in a changing environment is a fundamental challenge for any adaptive agent. Here, we show that designing an agent in a modular fashion as a collection of subagents, each dedicated to a separate need, powerfully enhanced the agent's capacity to satisfy its overall needs. We used the formalism of deep reinforcement learning to investigate a biologically relevant multiobjective task: continually maintaining homeostasis of a set of physiologic variables. We then conducted simulations in a variety of environments and compared how modular agents performed relative to standard monolithic agents (i.e., agents that aimed to satisfy all needs in an integrated manner using a single aggregate measure of success). Simulations revealed that modular agents a) exhibited a form of exploration that was intrinsic and emergent rather than extrinsically imposed; b) were robust to changes in nonstationary environments, and c) scaled gracefully in their ability to maintain homeostasis as the number of conflicting objectives increased. Supporting analysis suggested that the robustness to changing environments and increasing numbers of needs were due to intrinsic exploration and efficiency of representation afforded by the modular architecture. These results suggest that the normative principles by which agents have adapted to complex changing environments may also explain why humans have long been described as consisting of "multiple selves."

The pursuit of happiness: A reinforcement learning perspective on habituation and comparisons.

In evaluating our choices, we often suffer from two tragic relativities. First, when our lives change for the better, we rapidly habituate to the higher standard of living. Second, we cannot escape comparing ourselves to various relative standards. Habituation and comparisons can be very disruptive to decision-making and happiness, and till date, it remains a puzzle why they have come to be a part of cognition in the first place. Here, we present computational evidence that suggests that these features might play an important role in promoting adaptive behavior. Using the framework of reinforcement learning, we explore the benefit of employing a reward function that, in addition to the reward provided by the underlying task, also depends on prior expectations and relative comparisons. We find that while agents equipped with this reward function are less happy, they learn faster and significantly outperform standard reward-based agents in a wide range of environments. Specifically, we find that relative comparisons speed up learning by providing an exploration incentive to the agents, and prior expectations serve as a useful aid to comparisons, especially in sparsely-rewarded and non-stationary environments. Our simulations also reveal potential drawbacks of this reward function and show that agents perform sub-optimally when comparisons are left unchecked and when there are too many similar options. Together, our results help explain why we are prone to becoming trapped in a cycle of never-ending wants and desires, and may shed light on psychopathologies such as depression, materialism, and overconsumption.

If it's important, then I'm curious: Increasing perceived usefulness stimulates curiosity.

Curiosity is considered essential for learning and sustained engagement, yet stimulating curiosity in educational contexts remains a challenge. Can people's curiosity about a scientific topic be stimulated by providing evidence that knowledge about the topic has potential value to society? Here, we show that increasing perceptions of 'social usefulness' regarding a scientific topic also increases curiosity and subsequent information search. Our results also show that simply presenting interesting facts is not enough to influence curiosity, and that people are more likely to be curious about a scientific topic if they perceive it to be useful personally and socially. Given the link between curiosity and learning, these results have important implications for science communication and education more broadly.

Curiosity Is Contagious: A Social Influence Intervention to Induce Curiosity.

Our actions and decisions are regularly influenced by the social environment around us. Can social cues be leveraged to induce curiosity and affect subsequent behavior? Across two experiments, we show that curiosity is contagious: The social environment can influence people's curiosity about the answers to scientific questions. Participants were presented with everyday questions about science from a popular on-line forum, and these were shown with a high or low number of up-votes as a social cue to popularity. Participants indicated their curiosity about the answers, and they were given an opportunity to reveal a subset of those answers. Participants reported greater curiosity about the answers to questions when the questions were presented with a high (vs. low) number of up-votes, and they were also more likely to choose to reveal the answers to questions with a high (vs. low) number of up-votes. These effects were partially mediated by surprise and by the inferred usefulness of knowledge, with a more dramatic effect of low up-votes in reducing curiosity than of high up-votes in boosting curiosity. Taken together, these results highlight the important role social information plays in shaping our curiosity.

Reconciling novelty and complexity through a rational analysis of curiosity.

Curiosity is considered to be the essence of science and an integral component of cognition. What prompts curiosity in a learner? Previous theoretical accounts of curiosity remain divided-novelty-based theories propose that new and highly uncertain stimuli pique curiosity, whereas complexity-based theories propose that stimuli with an intermediate degree of uncertainty stimulate curiosity. In this article, we present a rational analysis of curiosity by considering the computational problem underlying curiosity, which allows us to model these distinct accounts of curiosity in a common framework. Our approach posits that a rational agent should explore stimuli that maximally increase the usefulness of its knowledge and that curiosity is the mechanism by which humans approximate this rational behavior. Critically, our analysis show that the causal structure of the environment can determine whether curiosity is driven by either highly uncertain or moderately uncertain stimuli. This suggests that previous theories need not be in contention but are special cases of a more general account of curiosity. Experimental results confirm our predictions and demonstrate that our theory explains a wide range of findings about human curiosity, including its subjectivity and malleability. (PsycInfo Database Record (c) 2020 APA, all rights reserved).