Computational evolution of social norms in well-mixed and group-structured populations.

Models of indirect reciprocity study how social norms promote cooperation. In these models, cooperative individuals build up a positive reputation, which in turn helps them in their future interactions. The exact reputational benefits of cooperation depend on the norm in place, which may change over time. Previous research focused on the stability of social norms. Much less is known about how social norms initially evolve when competing with many others. A comprehensive evolutionary analysis, however, has been difficult. Even among the comparably simple space of so-called third-order norms, there are thousands of possibilities, each one inducing its own reputation dynamics. To address this challenge, we use large-scale computer simulations. We study the reputation dynamics of each third-order norm and all evolutionary transitions between them. In contrast to established work with only a handful of norms, we find that cooperation is hard to maintain in well-mixed populations. However, within group-structured populations, cooperation can emerge. The most successful norm in our simulations is particularly simple. It regards cooperation as universally positive, and defection as usually negative-unless defection takes the form of justified punishment. This research sheds light on the complex interplay of social norms, their induced reputation dynamics, and population structure.

Evolution of reciprocity with limited payoff memory.

Direct reciprocity is a mechanism for the evolution of cooperation in repeated social interactions. According to the literature, individuals naturally learn to adopt conditionally cooperative strategies if they have multiple encounters with their partner. Corresponding models have greatly facilitated our understanding of cooperation, yet they often make strong assumptions on how individuals remember and process payoff information. For example, when strategies are updated through social learning, it is commonly assumed that individuals compare their average payoffs. This would require them to compute (or remember) their payoffs against everyone else in the population. To understand how more realistic constraints influence direct reciprocity, we consider the evolution of conditional behaviours when individuals learn based on more recent experiences. Even in the most extreme case that they only take into account their very last interaction, we find that cooperation can still evolve. However, such individuals adopt less generous strategies, and they cooperate less often than in the classical setup with average payoffs. Interestingly, once individuals remember the payoffs of two or three recent interactions, cooperation rates quickly approach the classical limit. These findings contribute to a literature that explores which kind of cognitive capabilities are required for reciprocal cooperation. While our results suggest that some rudimentary form of payoff memory is necessary, it suffices to remember a few interactions.

Efficiency and resilience of cooperation in asymmetric social dilemmas.

Direct reciprocity is a powerful mechanism for cooperation in social dilemmas. The very logic of reciprocity, however, seems to require that individuals are symmetric, and that everyone has the same means to influence each others' payoffs. Yet in many applications, individuals are asymmetric. Herein, we study the effect of asymmetry in linear public good games. Individuals may differ in their endowments (their ability to contribute to a public good) and in their productivities (how effective their contributions are). Given the individuals' productivities, we ask which allocation of endowments is optimal for cooperation. To this end, we consider two notions of optimality. The first notion focuses on the resilience of cooperation. The respective endowment distribution ensures that full cooperation is feasible even under the most adverse conditions. The second notion focuses on efficiency. The corresponding endowment distribution maximizes group welfare. Using analytical methods, we fully characterize these two endowment distributions. This analysis reveals that both optimality notions favor some endowment inequality: More productive players ought to get higher endowments. Yet the two notions disagree on how unequal endowments are supposed to be. A focus on resilience results in less inequality. With additional simulations, we show that the optimal endowment allocation needs to account for both the resilience and the efficiency of cooperation.

Vacancies in growing habitats promote the evolution of cooperation.

We study evolutionary game dynamics in a growing habitat with vacancies. Fitness is determined by the global effect of the environment and a local prisoner's dilemma among neighbors. We study population growth on a one-dimensional lattice and analyze how the environment affects evolutionary competition. As the environment becomes harsh, an absorbing phase transition from growing populations to extinction occurs. The transition point depends on which strategies are present in the population. In particular, we find a 'cooperative window' in parameter space, where only cooperators can survive. A mutant defector in a cooperative community might briefly proliferate, but over time naturally occurring vacancies separate cooperators from defectors, thereby driving defectors to extinction. Our model reveals that vacancies provide a strong boost for cooperation by spatial selection.

Indirect reciprocity with stochastic and dual reputation updates.

Cooperation is a crucial aspect of social life, yet understanding the nature of cooperation and how it can be promoted is an ongoing challenge. One mechanism for cooperation is indirect reciprocity. According to this mechanism, individuals cooperate to maintain a good reputation. This idea is embodied in a set of social norms called the "leading eight". When all information is publicly available, these norms have two major properties. Populations that employ these norms are fully cooperative, and they are stable against invasion by alternative norms. In this paper, we extend the framework of the leading eight in two directions. First, we include norms with 'dual' reputation updates. These norms do not only assign new reputations to an acting donor; they also allow to update the reputation of the passive recipient. Second, we allow social norms to be stochastic. Such norms allow individuals to evaluate others with certain probabilities. Using this framework, we characterize all evolutionarily stable norms that lead to full cooperation in the public information regime. When only the donor's reputation is updated, and all updates are deterministic, we recover the conventional model. In that case, we find two classes of stable norms: the leading eight and the 'secondary sixteen'. Stochasticity can further help to stabilize cooperation when the benefit of cooperation is comparably small. Moreover, updating the recipients' reputations can help populations to recover more quickly from errors. Overall, our study highlights a remarkable trade-off between the evolutionary stability of a norm and its robustness with respect to errors. Norms that correct errors quickly require higher benefits of cooperation to be stable.

The effect of environmental information on evolution of cooperation in stochastic games.

Many human interactions feature the characteristics of social dilemmas where individual actions have consequences for the group and the environment. The feedback between behavior and environment can be studied with the framework of stochastic games. In stochastic games, the state of the environment can change, depending on the choices made by group members. Past work suggests that such feedback can reinforce cooperative behaviors. In particular, cooperation can evolve in stochastic games even if it is infeasible in each separate repeated game. In stochastic games, participants have an interest in conditioning their strategies on the state of the environment. Yet in many applications, precise information about the state could be scarce. Here, we study how the availability of information (or lack thereof) shapes evolution of cooperation. Already for simple examples of two state games we find surprising effects. In some cases, cooperation is only possible if there is precise information about the state of the environment. In other cases, cooperation is most abundant when there is no information about the state of the environment. We systematically analyze all stochastic games of a given complexity class, to determine when receiving information about the environment is better, neutral, or worse for evolution of cooperation.

Adaptive dynamics of memory-one strategies in the repeated donation game.

Human interactions can take the form of social dilemmas: collectively, people fare best if all cooperate but each individual is tempted to free ride. Social dilemmas can be resolved when individuals interact repeatedly. Repetition allows them to adopt reciprocal strategies which incentivize cooperation. The most basic model for direct reciprocity is the repeated donation game, a variant of the prisoner's dilemma. Two players interact over many rounds; in each round they decide whether to cooperate or to defect. Strategies take into account the history of the play. Memory-one strategies depend only on the previous round. Even though they are among the most elementary strategies of direct reciprocity, their evolutionary dynamics has been difficult to study analytically. As a result, much previous work has relied on simulations. Here, we derive and analyze their adaptive dynamics. We show that the four-dimensional space of memory-one strategies has an invariant three-dimensional subspace, generated by the memory-one counting strategies. Counting strategies record how many players cooperated in the previous round, without considering who cooperated. We give a partial characterization of adaptive dynamics for memory-one strategies and a full characterization for memory-one counting strategies.

Mutation enhances cooperation in direct reciprocity.

Direct reciprocity is a powerful mechanism for the evolution of cooperation based on repeated interactions between the same individuals. But high levels of cooperation evolve only if the benefit-to-cost ratio exceeds a certain threshold that depends on memory length. For the best-explored case of one-round memory, that threshold is two. Here, we report that intermediate mutation rates lead to high levels of cooperation, even if the benefit-to-cost ratio is only marginally above one, and even if individuals only use a minimum of past information. This surprising observation is caused by two effects. First, mutation generates diversity which undermines the evolutionary stability of defectors. Second, mutation leads to diverse communities of cooperators that are more resilient than homogeneous ones. This finding is relevant because many real-world opportunities for cooperation have small benefit-to-cost ratios, which are between one and two, and we describe how direct reciprocity can attain cooperation in such settings. Our result can be interpreted as showing that diversity, rather than uniformity, promotes evolution of cooperation.

Quantitative assessment can stabilize indirect reciprocity under imperfect information.

The field of indirect reciprocity investigates how social norms can foster cooperation when individuals continuously monitor and assess each other's social interactions. By adhering to certain social norms, cooperating individuals can improve their reputation and, in turn, receive benefits from others. Eight social norms, known as the "leading eight," have been shown to effectively promote the evolution of cooperation as long as information is public and reliable. These norms categorize group members as either 'good' or 'bad'. In this study, we examine a scenario where individuals instead assign nuanced reputation scores to each other, and only cooperate with those whose reputation exceeds a certain threshold. We find both analytically and through simulations that such quantitative assessments are error-correcting, thus facilitating cooperation in situations where information is private and unreliable. Moreover, our results identify four specific norms that are robust to such conditions, and may be relevant for helping to sustain cooperation in natural populations.

Cooperation and coordination in heterogeneous populations.

One landmark application of evolutionary game theory is the study of social dilemmas. This literature explores why people cooperate even when there are strong incentives to defect. Much of this literature, however, assumes that interactions are symmetric. Individuals are assumed to have the same strategic options and the same potential pay-offs. Yet many interesting questions arise once individuals are allowed to differ. Here, we study asymmetry in simple coordination games. In our set-up, human participants need to decide how much of their endowment to contribute to a public good. If a group's collective contribution reaches a pre-defined threshold, all group members receive a reward. To account for possible asymmetries, individuals either differ in their endowments or their productivities. According to a theoretical equilibrium analysis, such games tend to have many possible solutions. In equilibrium, group members may contribute the same amount, different amounts or nothing at all. According to our behavioural experiment, however, humans favour the equilibrium in which everyone contributes the same proportion of their endowment. We use these experimental results to highlight the non-trivial effects of inequality on cooperation, and we discuss to which extent models of evolutionary game theory can account for these effects. This article is part of the theme issue 'Half a century of evolutionary games: a synthesis of theory, application and future directions'.

Evolution of direct reciprocity in group-structured populations.

People tend to have their social interactions with members of their own community. Such group-structured interactions can have a profound impact on the behaviors that evolve. Group structure affects the way people cooperate, and how they reciprocate each other's cooperative actions. Past work has shown that population structure and reciprocity can both promote the evolution of cooperation. Yet the impact of these mechanisms has been typically studied in isolation. In this work, we study how the two mechanisms interact. Using a game-theoretic model, we explore how people engage in reciprocal cooperation in group-structured populations, compared to well-mixed populations of equal size. In this model, the population is subdivided into groups. Individuals engage in pairwise interactions within groups while they also have chances to imitate strategies outside the groups. To derive analytical results, we focus on two scenarios. In the first scenario, we assume a complete separation of time scales. Mutations are rare compared to between-group comparisons, which themselves are rare compared to within-group comparisons. In the second scenario, there is a partial separation of time scales, where mutations and between-group comparisons occur at a comparable rate. In both scenarios, we find that the effect of population structure depends on the benefit of cooperation. When this benefit is small, group-structured populations are more cooperative. But when the benefit is large, well-mixed populations result in more cooperation. Overall, our results reveal how group structure can sometimes enhance and sometimes suppress the evolution of cooperation.

Reputation effects drive the joint evolution of cooperation and social rewarding.

People routinely cooperate with each other, even when cooperation is costly. To further encourage such pro-social behaviors, recipients often respond by providing additional incentives, for example by offering rewards. Although such incentives facilitate cooperation, the question remains how these incentivizing behaviors themselves evolve, and whether they would always be used responsibly. Herein, we consider a simple model to systematically study the co-evolution of cooperation and different rewarding policies. In our model, both social and antisocial behaviors can be rewarded, but individuals gain a reputation for how they reward others. By characterizing the game's equilibria and by simulating evolutionary learning processes, we find that reputation effects systematically favor cooperation and social rewarding. While our baseline model applies to pairwise interactions in well-mixed populations, we obtain similar conclusions under assortment, or when individuals interact in larger groups. According to our model, rewards are most effective when they sway others to cooperate. This view is consistent with empirical observations suggesting that people reward others to ultimately benefit themselves.

Direct reciprocity between individuals that use different strategy spaces.

In repeated interactions, players can use strategies that respond to the outcome of previous rounds. Much of the existing literature on direct reciprocity assumes that all competing individuals use the same strategy space. Here, we study both learning and evolutionary dynamics of players that differ in the strategy space they explore. We focus on the infinitely repeated donation game and compare three natural strategy spaces: memory-1 strategies, which consider the last moves of both players, reactive strategies, which respond to the last move of the co-player, and unconditional strategies. These three strategy spaces differ in the memory capacity that is needed. We compute the long term average payoff that is achieved in a pairwise learning process. We find that smaller strategy spaces can dominate larger ones. For weak selection, unconditional players dominate both reactive and memory-1 players. For intermediate selection, reactive players dominate memory-1 players. Only for strong selection and low cost-to-benefit ratio, memory-1 players dominate the others. We observe that the supergame between strategy spaces can be a social dilemma: maximum payoff is achieved if both players explore a larger strategy space, but smaller strategy spaces dominate.

Cooperation in alternating interactions with memory constraints.

In repeated social interactions, individuals often employ reciprocal strategies to maintain cooperation. To explore the emergence of reciprocity, many theoretical models assume synchronized decision making. In each round, individuals decide simultaneously whether to cooperate or not. Yet many manifestations of reciprocity in nature are asynchronous. Individuals provide help at one time and receive help at another. Here, we explore such alternating games in which players take turns. We mathematically characterize all Nash equilibria among memory-one strategies. Moreover, we use evolutionary simulations to explore various model extensions, exploring the effect of discounted games, irregular alternation patterns, and higher memory. In all cases, we observe that mutual cooperation still evolves for a wide range of parameter values. However, compared to simultaneous games, alternating games require different strategies to maintain cooperation in noisy environments. Moreover, none of the respective strategies are evolutionarily stable.

Evolutionary instability of selfish learning in repeated games.

Across many domains of interaction, both natural and artificial, individuals use past experience to shape future behaviors. The results of such learning processes depend on what individuals wish to maximize. A natural objective is one's own success. However, when two such "selfish" learners interact with each other, the outcome can be detrimental to both, especially when there are conflicts of interest. Here, we explore how a learner can align incentives with a selfish opponent. Moreover, we consider the dynamics that arise when learning rules themselves are subject to evolutionary pressure. By combining extensive simulations and analytical techniques, we demonstrate that selfish learning is unstable in most classical two-player repeated games. If evolution operates on the level of long-run payoffs, selection instead favors learning rules that incorporate social (other-regarding) preferences. To further corroborate these results, we analyze data from a repeated prisoner's dilemma experiment. We find that selfish learning is insufficient to explain human behavior when there is a trade-off between payoff maximization and fairness.

(Mis)perceiving cooperativeness.

Cooperation is crucial for the success of social interactions. Given its importance, humans should readily be able to use available cues to predict how likely others are to cooperate. Here, we review the empirical literature on how accurate such predictions are. To this end, we distinguish between three classes of cues: behavioral (including past decisions), personal (including gender, attractiveness, and group membership) and situational (including the benefits to cooperation and the ability to communicate with each other). We discuss (i) how each cue correlates with future cooperative decisions and (ii) whether people correctly anticipate each cue's predictive value. We find that people are fairly accurate in interpreting behavioral and situational cues. However, they often misperceive the value of personal cues.

Evolution of cooperation through cumulative reciprocity.

Reciprocity is a simple principle for cooperation that explains many of the patterns of how humans seek and receive help from each other. To capture reciprocity, traditional models often assume that individuals use simple strategies with restricted memory. These memory-1 strategies are mathematically convenient, but they miss important aspects of human reciprocity, where defections can have lasting effects. Here we instead propose a strategy of cumulative reciprocity. Cumulative reciprocators count the imbalance of cooperation across their previous interactions with their opponent. They cooperate as long as this imbalance is sufficiently small. Using analytical and computational methods, we show that this strategy can sustain cooperation in the presence of errors, that it enforces fair outcomes and that it evolves in hostile environments. Using an economic experiment, we confirm that cumulative reciprocity is more predictive of human behaviour than several classical strategies. The basic principle of cumulative reciprocity is versatile and can be extended to a range of social dilemmas.

The evolution of indirect reciprocity under action and assessment generosity.

Indirect reciprocity is a mechanism for the evolution of cooperation based on social norms. This mechanism requires that individuals in a population observe and judge each other's behaviors. Individuals with a good reputation are more likely to receive help from others. Previous work suggests that indirect reciprocity is only effective when all relevant information is reliable and publicly available. Otherwise, individuals may disagree on how to assess others, even if they all apply the same social norm. Such disagreements can lead to a breakdown of cooperation. Here we explore whether the predominantly studied 'leading eight' social norms of indirect reciprocity can be made more robust by equipping them with an element of generosity. To this end, we distinguish between two kinds of generosity. According to assessment generosity, individuals occasionally assign a good reputation to group members who would usually be regarded as bad. According to action generosity, individuals occasionally cooperate with group members with whom they would usually defect. Using individual-based simulations, we show that the two kinds of generosity have a very different effect on the resulting reputation dynamics. Assessment generosity tends to add to the overall noise and allows defectors to invade. In contrast, a limited amount of action generosity can be beneficial in a few cases. However, even when action generosity is beneficial, the respective simulations do not result in full cooperation. Our results suggest that while generosity can favor cooperation when individuals use the most simple strategies of reciprocity, it is disadvantageous when individuals use more complex social norms.

A unified framework of direct and indirect reciprocity.

Direct and indirect reciprocity are key mechanisms for the evolution of cooperation. Direct reciprocity means that individuals use their own experience to decide whether to cooperate with another person. Indirect reciprocity means that they also consider the experiences of others. Although these two mechanisms are intertwined, they are typically studied in isolation. Here, we introduce a mathematical framework that allows us to explore both kinds of reciprocity simultaneously. We show that the well-known 'generous tit-for-tat' strategy of direct reciprocity has a natural analogue in indirect reciprocity, which we call 'generous scoring'. Using an equilibrium analysis, we characterize under which conditions either of the two strategies can maintain cooperation. With simulations, we additionally explore which kind of reciprocity evolves when members of a population engage in social learning to adapt to their environment. Our results draw unexpected connections between direct and indirect reciprocity while highlighting important differences regarding their evolvability.

Evolving cooperation in multichannel games.

Humans routinely engage in many distinct interactions in parallel. Team members collaborate on several concurrent projects, and even whole nations interact with each other across a variety of issues, including trade, climate change and security. Yet the existing theory of direct reciprocity studies isolated repeated games. Such models cannot account for strategic attempts to use the vested interests in one game as a leverage to enforce cooperation in another. Here we introduce a general framework of multichannel games. Individuals interact with each other over multiple channels; each channel is a repeated game. Strategic choices in one channel can affect decisions in another. With analytical equilibrium calculations for the donation game and evolutionary simulations for several other games we show that such linkage facilitates cooperation. Our results suggest that previous studies tend to underestimate the human potential for reciprocity. When several interactions occur in parallel, people often learn to coordinate their behavior across games to maximize cooperation in each of them.