Evolution of Groupwise Cooperation: Generosity, Paradoxical Behavior, and Non-Linear Payoff Functions

Abstract

Evolution of cooperation by reciprocity has been studied using two-player and n-player repeated prisoner’s dilemma games. An interesting feature specific to the n-player case is that players can vary in generosity, or how many defections they tolerate in a given round of a repeated game. Reciprocators are quicker to detect defectors to withdraw further cooperation when less generous, and better at maintaining a long-term cooperation in the presence of rare defectors when more generous. A previous analysis on a stochastic evolutionary model of the n-player repeated prisoner’s dilemma has shown that the fixation probability of a single reciprocator in a population of defectors can be maximized for a moderate level of generosity. However, the analysis is limited in that it considers only tit-for-tat-type reciprocators within the conventional linear payoff assumption. Here we extend the previous study by removing these limitations and show that, if the games are repeated sufficiently many times, considering non-tit-for-tat type strategies does not alter the previous results, while the introduction of non-linear payoffs sometimes does. In particular, under certain conditions, the fixation probability is maximized for a “paradoxical” strategy, which cooperates in the presence of fewer cooperating opponents than in other situations in which it defects.

1. Introduction

Reciprocity is a key factor in the study of evolution of cooperation [1,2]. Evolution of pairwise and groupwise cooperation by reciprocity has been studied by using the two-player and $n$-player repeated prisoner’s dilemma (PD) games [2,3,4,5,6,7,8,9,10,11,12]. While a population of reciprocators can be stable against invasion by defectors in repeated PDs, so long as games are repeated sufficiently many times, various additional mechanisms have been proposed to explain the initial emergence of reciprocators in a population of defectors, such as invasion by a mass of reciprocators [13], formation of spatial clusters [14,15], stochastic aggregate payoffs [16], and random drift [17]. Of particular relevance to the present study is Nowak et al. (2004). They developed a model of stochastic evolutionary dynamics for the two-player repeated PD and derived the fixation probability with which a single mutant of reciprocator that appears in a population of defectors will eventually replace the whole population. Nowak et al.’s (2004) framework has been extended to include the more general $n$-player repeated PD [18,19], which has enabled to investigate the evolution of group-wise cooperation [20,21,22] considering the effect of random drift. In this article, we report our further investigation on the stochastic evolutionary model of the $n$-player repeated PD.

An interesting feature of group-wise cooperation, or more specifically, the $n$-player repeated PD, is that we can conceive of multiple types of reciprocators that differ in their levels of generosity. Take tit-for-tat (TFT) for example. TFT is a reciprocal strategy in the two-player repeated PD, which cooperates in the first round of a repeated game and, from the second round onward, cooperates if and only if its opponent has cooperated in the previous round. In the $n$-player repeated PD, $n$ different reciprocal strategies analogous to TFT, or TFT_a for $a\in \left\{0, 1, \cdots , n-1\right\}$, are possible, where $a$ represents the strategy’s level of generosity [23]. Namely, TFT_a, cooperates in the first round and after that cooperates if and only if $a$ or more of its opponents have cooperated in the previous round. Thus, TFT_a with a smaller value of $a$ is more generous toward the group members’ defection.

Suppose that a single mutant of the strategy that always defects (ALLD) appears in a population of TFT_a. Obviously, less generous TFT_a (i.e., having larger $a$) is quicker to find ALLDs in the group and withdraw further cooperation and, thus, more likely to be stable against their invasion. On the other hand, for TFT_a to be advantageous over ALLD, it is always necessary that each TFT_a individual tends to have $a$ or more other TFT_a individuals in the same group; otherwise, TFT_a cooperates only in the first round of each game to be exploited by ALLDs. This means that more generous TFT_a (i.e., having smaller $a$) may compete better against ALLD, particularly when TFT_a is rare in the population; in other words, generosity may serve to facilitate the initial emergence of TFT_a.

This intuition has been confirmed by Kurokawa et al. (2010), who compared the fixation probabilities of TFT_as with different values of $a$ when appears as a single mutant in a population of ALLD [19]. They found that more generous TFT_a can sometimes have a greater fixation probability than less generous ones. They also specified the optimal level of generosity that maximizes the fixation probability, showing that there is a certain threshold level of generosity above which the largest fixation probability can never be attained. Specifically, it was shown that any such TFT_a that tolerates defections by a half or more of its opponents can never have the largest fixation probability.

Although Kurokawa et al. (2010) focused on the strategies classified as TFT_a, they are not the only ones that can potentially establish mutual cooperation in the $n$-player repeated PD. In fact, tolerating the presence of some defectors in a group may have a similar enhancing effect on the evolution of reciprocal strategies in general. However, this possibility has not been explored thus far. Here we extend Kurokawa et al.’s (2010) analysis to consider the set of all “reactive strategies” that (i) cooperates with probability $d_{\phi }\in \left\{0, 1\right\}$ in the first round and (ii) cooperates in the $k$th round ($k\ge 2$) with probability $d_j\in \left\{0, 1\right\}$, which depends on the number of opponents, $j$, who have cooperated in the $k-1$th round. There are $2^{n+1}$ such strategies in total, including $n-1$ TFT_as and $2^n$ strategies that defect in the first move [24,25,26,27,28]. The rest of the strategies in the set are “paradoxical” in the sense that each of them has a certain situation in which it cooperates even though there are fewer cooperators in the group than in other situations in which it withdraws cooperation. It is of interest whether the strategies defecting in the first move or exhibiting paradoxical behavior can have the largest fixation probability among the set of all reactive strategies when invading ALLD.

Another limitation of Kurokawa et al.’s (2010) work is that they only considered the conventional linear payoff functions (i.e., linear public goods games. However, there are at least two reasons why non-linear payoff functions should be also considered. First, in the real world, payoffs are not always linear [29,30,31,32,33,34,35,36,37,38,39] and it has been demonstrated that introduction of non-linearity can affect the outcome of evolutionary models (e.g., [40,41,42,43,44,45,46,47]. Second, if payoffs increase non-linearly with the number of cooperating members in the group, it can be said that the efficiency of an act of cooperation varies depending on the number of cooperators. For example, in case payoffs increase more rapidly when cooperators are fewer, an individual’s act of cooperation is more efficient in the presence of fewer cooperators. In such a case, more generous strategies may be selectively favored because generosity promotes cooperation when it is efficient. To examine these possibilities, we also extend Kurokawa et al. (2010) in terms of payoff functions.

In addition, Kurokawa et al. (2010) derived their results under the assumption that selection is sufficiently weak. However, the intensity of selection can affect the outcomes of evolutionary models (e.g., [17,48,49,50]. Hence, we investigate to what extent our results based on the weak selection assumption may be affected by the selection intensity.

In this paper, we extend Kurokawa et al.’s (2010) analysis by removing these limitations. In particular, taking non-linear payoff functions and paradoxical (i.e., non-tit-for-tat type) strategies into consideration, we ask how non-linear payoffs may facilitate the evolution of excessive generosity and whether any paradoxical strategies can ever attain the highest fixation probability among all reactive strategies considered. Additionally, we examine how our results may depend on the intensity of natural selection. In what follows, we first introduce the $n$-player repeated PD and our model of stochastic evolutionary dynamics (Model). Then we derive, based on the weak selection approximation, the best reactive strategies, or the strategies maximizing the fixation probability, and provide detailed analyses for linear and non-linear payoff cases. We also examine to what extent our results may be affected if selection is not sufficiently weak (Results). Finally, we summarize the results and discuss some caveats and possibilities of further investigations (Discussion).

2. Model

We investigate stochastic evolutionary dynamics of a population of individuals whose fitness is determined by $n$-player repeated PD games. We consider a set of “reactive strategies,” as defined below, and compare the fixation probabilities of different reactive strategies when introduced as a mutant into a population of the strategy that always defects, or ALLD. Our goal is to specify the “best reactive strategies,” which maximize the fixation probability.

2.1. The $n$-Player Repeated Prisoner’s Dilemma

In the n-player repeated PD, a group of $n$ individuals play a game consisting of one or more rounds, in each of which each individual chooses to either cooperate or defect. After one round is finished, another round will be played with probability $\delta$; otherwise, the group will be dismissed ($0<\delta <1$. Thus, the expected number of rounds is $1/\left(1-\delta \right)$. Suppose that there are $k$ cooperating and $n-k$ defecting individuals in a given round. The payoffs to cooperating (C) and defecting (D) individuals in that round are denoted by $V\left(C\right|k)$ and $V(D|k)$, respectively ($0\le k\le n$. Note that $V\left(C|0\right)$ and $V\left(D|n\right)$ are defined arbitrarily and used only for the sake of notational convenience. Following Boyd & Richerson (1988), n-player PD requires the following four conditions to be satisfied:

$$V\left(C|k+1\right)<V\left(D|k\right),$$

(1)

$$V\left(C|k\right)<V\left(C|k+1\right),$$

(2)

$$V\left(D|k\right)<V\left(D|k+1\right),$$

(3)

$$\left(n-k-1\right)V\left(D|k+1\right)+\left(k+1\right)V\left(C|k+1\right)>\left(n-k\right)V\left(D|k\right)+kV\left(C|k\right),$$

(4)

where $0\le k\le n-1$. Conditions (1) states that an individual always gains more from defecting than cooperating. Conditions (2) and (3) mean that an individual always gains more when there are more cooperators in the group. Conditions (4) indicates that the total payoff of the group is larger when there are more cooperators in the group.

We define a reactive strategy in the $n$-player repeated PD as follows: an individual with a reactive strategy cooperates with a certain probability in the first round, and from the second round on, cooperates with a certain probability determined by the number of opponents (i.e., group members other than the self) who have cooperated in the preceding round. Any reactive strategy is described by a vector $d=\left(d_{\phi }, d_0, d_1, \cdots , d_{n-1}\right)$, where $d_{\phi }$ denotes the probability with which an individual obeying this strategy cooperates in the first move and $d_j$ denotes the probability with which he cooperates in a given round, provided that $j$ of the $n-1$ opponents have cooperated in the preceding round. We focus on the set, $\mathsf{\Omega }$, of all reactive strategies in which $d_{\phi }$ is either 0 or 1 and $d_j$ is either 0 or 1, namely, $\mathsf{\Omega }=\left\{\left(d_{\phi },d_0, d_1, \cdots , d_{n-1}\right)|d_{\phi }\in \left\{0, 1\right\}, d_j\in \left\{0, 1\right\}, j\in \left\{0, 1,\dots ,n-1\right\}\right\}$. Obviously, there are $2^{n+1}$ strategies in $\mathsf{\Omega }$.

Table 1. The payoff matrix of the general n-player game.

	$\mathbf{Number}\mathbf{of}\mathbf{a}\mathbf{Individuals}\mathbf{among}\mathbf{the}\mathit{n}-1\mathbf{Opponents}$
Strategy of the focal individual	$n-1$	$n-2$	$n-3$	$\cdots$	$1$	$0$
A	$a_1$	$a_2$	$a_3$	$\cdots$	$a_{n-1}$	$a_n$
B	$b_1$	$b_2$	$b_3$	$\cdots$	$b_{n-1}$	$b_n$

shows the payoff matrix of the general two-strategy $n$-player game, where $a_l$ denotes the payoff to an individual with strategy A whose opponents are $n-l$ individuals with strategy A and $l-1$ individuals with strategy B; likewise, $b_l$ denotes the payoff to an individual with strategy B confronting with the same composition of opponents. Let A and B in Table 1 represent a strategy $d$ in $\mathsf{\Omega }$ and ALLD, respectively. To describe the game involving $d$ and ALLD, we need to write down all $a_l$ and $b_l$ in Table 1 in terms of $\delta$, $V\left(C|k\right)$, $V\left(D|k\right)$, and $d$. Consider a group consisting of $k$ individuals adopting $d$ and $n-k$ individuals adopting ALLD. We denote by $h_k$ the expected number of rounds in which a $d$ individual cooperates before the group is dismissed. Since there are on average $1/\left(1-\delta \right)$ rounds, we have $0\le h_k\le 1/\left(1-\delta \right).$ Further, since we do not consider any errors in behavior, $d$ individuals perfectly coordinate their behavior, that is, either all or none of them cooperate in a given round. Hence, we obtain:

$$a_{n-k+1}=h_{k}V\left(C|k\right)+\left(\frac{1}{1-\delta }-h_k\right)V\left(D|0\right),$$

(5)

$$b_{n-k}=h_{k}V\left(D|k\right)+\left(\frac{1}{1-\delta }-h_k\right)V\left(D|0\right).$$

(6)

For any group composition, $h_k$ is determined by $d_{\phi }$, $d_{k-1}$, and $d_0$ in the following manner. First, suppose that $d_{\phi }=d_{k-1}=1$. In this case, $d$ never defects and, hence, $h_k=1/\left(1-\delta \right)$. Second, if $d_{\phi }=1$ and $d_{k-1}=d_0=0$, $d$ cooperates only in the first round and, thus, $h_k=1$. Third, if $d_{\phi }=d_0=1$ and $d_{k-1}=0$, $d$ alternates cooperation and defection indefinitely, in which case $h_k=1/\left(1-{\delta }^2\right)$. Fourth, if $d_{\phi }=d_0=0$, $d$ defects in every round and hence $h_k=0$. Fifth, if $d_{\phi }=0$ and $d_0=d_{k-1}=1$, $d$ defects only in the first round and thus $h_k=\delta /\left(1-\delta \right).$ Sixth, if $d_{\phi }=d_{k-1}=0$ and $d_0=1$, $d$ alternates defection and cooperation indefinitely, in which case $h_k=\delta /\left(1-{\delta }^2\right)$. In sum, $h_k$ is given by:

$$h_k=d_{\phi }\left[1+\frac{\delta d_{k-1}}{1-\delta }+\frac{{\delta }^2{d}_0\left(1-d_{k-1}\right)}{1-{\delta }^2}\right]+\left(1-d_{\phi }\right)d_0\left[\frac{\delta }{1-{\delta }^2}+d_{k-1}\frac{{\delta }^2}{1-{\delta }^2}\right].$$

(7)

Therefore, an $n$-player repeated PD involving $d$ and ALLD is specified using Equations (5)–(7).

2.2. Evolutionary Dynamics

We use a model of stochastic evolutionary dynamics developed for the general two-strategy $n$-player game [18]. A population consisting of $i$ individuals adopting strategy A and $N-i$ individuals adopting strategy B is considered, where the population size, $N$, is constant ($0\le i\le N)$. Groups of $n$ individuals are formed by choosing individuals at random from the population and a game is played within each of these groups ($2\le n\le N)$. The payoffs of the game are as shown in Table 1. Let $F_i$ and $G_i$ be the expected payoffs to an A individual and a B individual, respectively, when the number of A individuals in the population is $i$. The fitness of A and B individuals are given by $f_i=1-w+w{F}_i$ and $g_i=1-w+w{G}_i$, respectively, where $w$ is the selection intensity ($0<w<1)$. The population dynamics are formulated as a Moran process with frequency-dependent fitness: at each time step, an individual is chosen for reproduction with the probability proportional to its fitness, and then one identical offspring is produced to replace another individual randomly chosen for death with probability 1/N [17,51]. Denote by ${\rho }_{A,B}$ the probability that a single individual with strategy A in a population of N − 1 individuals with strategy B will finally take over the whole population (i.e., the fixation probability). Throughout the paper, we assume that the interaction group is smaller than the population ($N>n)$. Here, we have:

$${\rho }_{A,B}=1/\left(1+{\displaystyle \sum }_{l=1}^{N-1}{\displaystyle \prod }_{i=1}^l\frac{1-w+w{\sum }_{k=1}^n\frac{\left(\begin{array}{c}i\\ n-k\end{array}\right)\left(\begin{array}{c}N-i-1\\ k-1\end{array}\right)}{\left(\begin{array}{c}N-1\\ n-1\end{array}\right)}{b}_k}{1-w+w{\sum }_{k=1}^n\frac{\left(\begin{array}{c}i-1\\ n-k\end{array}\right)\left(\begin{array}{c}N-i\\ k-1\end{array}\right)}{\left(\begin{array}{c}N-1\\ n-1\end{array}\right)}{a}_k}\right),$$

(8)

where $\left(\begin{array}{c}s\\ t\end{array}\right)$ represents a binomial coefficient when $s\ge t$ and is defined zero when $s<t$.

Assuming that selection is weak ($w\ll 1$), [19,52,53] have shown that ${\rho }_{A,B}$ is given approximately by:

$${\rho }_{A,B}\approx \frac{1}{N}\frac{1}{1-\alpha w/\left[n\left(n+1\right)\right]},$$

(9)

where:

$$\alpha =\left(N-n\right){\displaystyle \sum }_{k=1}^{n}k\left(a_k-b_k\right)+\left(n+1\right){\displaystyle \sum }_{k=1}^{n-1}k\left(a_{k+1}-b_k\right)$$

(10)

(see also [18,54,55]). Since ${\rho }_{A, B}$ increases with $\alpha$, the strategy with the largest fixation probability maximizes the right-hand side of Equation (10).

3. Results

3.1. General Case

We define the best reactive strategy as the strategy (or strategies) that maximizes the fixation probability when introduced as a single mutant in a population of ALLD. The best reactive strategy maximizes the right-hand side of Equation (10) in the context of the $n$-player repeated PD specified by Equations (5)–(7). Equation (10) is equivalent to:

$$\alpha =\left(N-n\right)\left({\sum }_{k=1}^n{a}_{n-k+1}-n{b}_n\right)+\left(N+1\right){\sum }_{k=1}^n\left(n-k\right)\left(a_{n-k+1}-b_{n-k}\right).$$

(11)

Putting Equations (5) and (6) into Equation (11) and explicitly denoting the dependence on strategy $d\in \mathsf{\Omega }$, we obtain:

$$\alpha \left(d\right)={\sum }_{k=1}^n{h}_k\left(d\right)f\left(k\right),$$

(12)

where:

$$f\left(k\right)=\left(N-n\right)\left(V\left(C|k\right)-V\left(D|0\right)\right)+\left(N+1\right)\left(n-k\right)\left(V\left(C|k\right)-V\left(D|k\right)\right),$$

(13)

is independent of strategy $d$. Note that $h_k\left(d\right)$ is given by Equation (7) and is dependent on $d$ in the following manner:

$$h_k\left(d\right)=h\left(d_{\phi }, d_0,d_{k-1},k\right).$$

(14)

The best reactive strategy $d^{\ast }$ is given by the following equation:

$$d^{\ast }={\mathrm{argmax}}_{d\in \mathsf{\Omega }}\left({\displaystyle \sum }_{k=1}^n{h}_k\left(d\right)f\left(k\right)\right).$$

(15)

As shown in Appendix A, the solution of the maximization problem in Equation (15) is given as follows. On the one hand, when:

$$\frac{1}{1-\delta }{\displaystyle \sum }_{k=1}^n\max \left\{f\left(k\right), 0\right\}+{\displaystyle \sum }_{k=1}^n\min \left\{f\left(k\right), 0\right\}>0,$$

(16)

the solution of Equation (15) is given by:

$$\{\begin{array}{cc}{d}_{\phi }^{\ast }=1,& \\ d_{k-1}^{\ast }=1& \mathrm{if} f\left(k\right)>0,\\ d_{k-1}^{\ast }=0& \mathrm{if} f\left(k\right)<0,\end{array}$$

(17)

for $1\le k\le n$. Note that $d_{k-1}^{\ast }$ can be either $0$ or $1$ in case $f\left(k\right)=0$. On the other hand, when the inequality in Inequality (16) is reversed, the solution of Equation (15) is given by:

$$d_{\phi }^{\ast }=d_0^{\ast }=0,$$

(18)

where $d_{k-1}^{\ast }$ can be either $0$ or $1$ for $2\le k\le n$. In sum, Conditions (16)–(18) specify the best reactive strategy, $d^{\ast }$.

Our main result Conditions (16)–(18) states that so long as the game is repeated sufficiently many times (i.e., $\delta$ is sufficiently large), each bit, $d_{k-1}$, of the best reactive strategy is solely determined by the sign of $f\left(k\right)$, which is determined by the payoff structure of the game (and the group size and the population size. In Equation (13), the term $V\left(C|k\right)-V\left(D|0\right)$ is the payoff to a cooperating player in a group where k players cooperate minus the payoff to a defecting player in a group where no one cooperates. Thus, this term represents the benefit of cooperation. The term $V\left(C|k\right)-V\left(D|k\right)$ represents the payoff difference between a cooperating player and a defecting player in a group where k individuals cooperate. Thus, this negative term represents the cost of cooperation. Weighted sum of them determines whether the best reactive strategy should cooperate if $k-1$ players have cooperated in the previous round.

In the following analysis, we consider cases in which Inequality (16) holds unless otherwise stated.

3.2. Linear Payoff

Let us begin with a special case where the following conventional linear payoff assumption holds:

$$V\left(D|k\right)=\frac{bk}{n},$$

(19)

$$V\left(C|k\right)=V\left(D|k\right)-c.$$

(20)

From Inequalities (1)–(4), it has to be that $0<c<b<nc$. Putting Equations (19) and (20) into Equation (13), we obtain:

$$f\left(k\right)=\left(N-n\right)\left(\frac{bk}{n}-c\right)-\left(N+1\right)\left(n-k\right)c.$$

(21)

Since $f\left(k\right)$ increases linearly with $k$ in this case, there exists a critical value above which $f\left(k\right)$ is positive (Figure 1a. Hence, from Equation (17), $\alpha$ is maximized when the following condition is met:

$$\{\begin{array}{c}{d}_k=1 \mathrm{for} k>k^{\ast },\\ d_k=0 \mathrm{for} k<k^{\ast },\end{array}$$

(22)

where:

$$k^{\ast }=\frac{\left(n+1\right)N}{\left(N-n\right)b/\left(nc\right)+N+1}-1.$$

(23)

From Equation (23), $k^{\ast }$ decreases with increasing $N$, and for an infinitely large population, $k^{\ast }$ is given by:

$$k^{\ast }=\frac{n+1}{b/\left(nc\right)+1}-1.$$

(24)

Since $b/\left(nc\right)<1$ by assumption, $k^{\ast }>\left(n-1\right)/2$ holds. Therefore, any strategy $d$ that cooperates when a half or less of the opponents have cooperated in the previous round cannot be the best reactive strategy.

A similar, but slightly different, result has been obtained by Kurokawa et al. (2010) [19]. They examined the fixation probability of a strategy called TFT_a, being introduced as a mutant into a population of ALLD, under the linear payoff assumption as given by Equations (19) and (20). An individual adopting TFT_a cooperates in the first round, and from the second round on, cooperates if and only if $a$ or more of the opponents have cooperated in the previous round, where $a\in \left\{0, 1, \cdots n-1\right\}$ measures the level of generosity [23]. Thus, there are $n$ strategies that are classified as TFT_a with different levels of generosity. From these, Kurokawa et al. (2010) specified $a_{opt}$, the optimal level of generosity, which maximizes the fixation probability. In the terminology of the present study, the $n$ strategies constitute the subset of $\mathsf{\Omega }$ for which $d_{\phi }=1$ and $d_0\le d_1\le \cdots \le d_{n-1}=1$. In addition to these, we consider two kinds of non-TFT_a reactive strategies. One is those that defect in the first round (i.e., $d_{\phi }=0$. Our analysis shows that some of this kind can maximize the fixation probability if the expected number of rounds is sufficiently small, partially modifying Kurokawa et al.’s (2010) finding. Another kind is what we call “paradoxical” strategies (see Introduction), in which $d_k>d_{k+1}$ holds for at least some $k$; for example, $d=\left(1, 0, 1, 1, 0, \cdots , 0, 1\right)$. It turns out that Equations (22)–(24) is equivalent to the optimal level of generosity obtained by Kurokawa et al. (2010). Thus, their result is not changed by taking paradoxical strategies into consideration.

3.3. Non-Linear Payoff

The n-player PD can be further subcategorized according to characteristics of the fitness functions [39,46]. To illustrate, we define the cost and benefit associated with cooperation as:

$$c\left(k\right)=V\left(D|k\right)-V\left(C|k\right),$$

(25)

$$b\left(k\right)=V\left(D|k\right),$$

(26)

respectively. From Inequalities (1) and (2), $c\left(k\right)>0$ is required for all $k$. We can measure how the cost and benefit depend on the number of cooperators by considering:

$$\Delta c\left(k\right)=c\left(k+1\right)-c\left(k\right),$$

(27)

$$\Delta b\left(k\right)=b\left(k+1\right)-b\left(k\right).$$

(28)

For example, when $\Delta c\left(k\right)=0$ holds for all $k$, the cost is constant. Let us also consider:

$${\Delta }^{2}b\left(k\right)=\Delta b\left(k+1\right)-\Delta b\left(k\right),$$

(29)

for $0\le k\le n-2$. It should be intuitively clear that the n-player PD requires $\Delta b\left(k\right)>0$, which is equivalent to (3), while ${\Delta }^{2}b\left(k\right)$ can be either positive or negative (or zero. If, for instance, ${\Delta }^{2}b\left(k\right)\ge 0$ holds for all $k$, the benefit increases in an accelerative manner with the number of cooperators (e.g., the weakest-link game; [56]), whereas if ${\Delta }^{2}b\left(k\right)\le 0$ always holds, the increase is decelerated (e.g., the volunteer’s dilemma; [57]. The class of n-player PDs for which ${\Delta }^{2}b\left(k\right)=0$ always holds represent the public goods games in which the amount of public goods increases linearly with the number of cooperators [47].

The linear payoff assumptions in Equations (19) and (20) mean that the following conditions are satisfied for all $k$:

$${\Delta }^{2}b\left(k\right)=0,$$

(30)

$$\Delta c\left(k\right)=0.$$

(31)

Below we examine how evolution of generosity and paradoxical behavior may be affected by relaxing Equations (30) and (31).

3.3.1. Evolution of Generosity with Non-Linear Payoff

Constant Cost with Non-Linear Benefit

Consider the case when there is no restriction on the benefit function (except Inequalities (1)–(4)), that is, Equation (30) is not necessarily true. For the moment, we assume Equation (31). From Equations (13) and (20), we have:

$$f\left(k\right)=\left(N-n\right)\left(V\left(D|k\right)-V\left(D|0\right)\right)-\left[\left(n+1\right)N-k\left(N+1\right)\right]c.$$

(32)

Using Conditions (1) and (20), we obtain:

$$V\left(D|k\right)-V\left(D|0\right)={\displaystyle \sum }_{l=1}^k\left[V\left(D|l\right)-V\left(C|l\right)+V\left(C|l\right)-V\left(D|l-1\right)\right]<kc,$$

(33)

which gives:

$$f\left(k\right)<\left[\left(2N-n+1\right)k-\left(n+1\right)N\right]c.$$

(34)

Thus, we obtain a necessary condition for $f\left(k\right)>0$, namely:

$$k>\frac{\left(n+1\right)N}{2N-n+1}.$$

(35)

The right-hand side of Inequality (35) decreases and approaches $\left(n+1\right)/2$ with increasing $N$. Hence, from Equation (17), $d_k=1$ holds in the best reactive strategy only if $k>\left(n-1\right)/2$. This means that any strategy $d$ who cooperates when a half or less of the opponents have cooperated in the previous round cannot be the best reactive strategy.

Variable Cost with Non-Decelerating Benefit

Let us now remove the restriction on the cost function, that is, we consider the case when Equation (31) does not necessarily hold. Instead of Equation (30), we assume:

$${\Delta }^{2}b\left(k\right)\ge 0,$$

(36)

which means that the benefit from cooperation increases with the number of cooperators in either linear or accelerative fashion. From Equations (13), (26) and (28), we have:

$$f\left(k\right)=\left(N-n\right)\left({\displaystyle \sum }_{l=0}^{k-1}\Delta b\left(l\right)\right)-\left[\left(N+1\right)\left(n-k\right)+\left(N-n\right)\right]c\left(k\right).$$

(37)

Using Inequality (36), we obtain:

$${\displaystyle \sum }_{l=0}^{k-1}\Delta b\left(l\right)\le k\Delta b\left(k-1\right).$$

(38)

From Conditions (37) and (38), we have:

$$f\left(k\right)\le \left(N-n\right)k\Delta b\left(k-1\right)-\left[\left(N+1\right)\left(n-k\right)+\left(N-n\right)\right]c\left(k\right).$$

(39)

Meanwhile, using Conditions (1), (25), (26) and (28), we obtain:

$$\Delta b\left(k-1\right)=V\left(D|k\right)-V\left(D|k-1\right)<V\left(D|k\right)-V\left(C|k\right)=c\left(k\right).$$

(40)

Therefore, combining Equation (39) and (40), we obtain:

$$f\left(k\right)<\left[\left(2N-n+1\right)k-\left(n+1\right)N\right]c\left(k\right).$$

(41)

Hence, as in the case of constant cost, Inequality (35) is necessary for $f\left(k\right)>0$, which again means that the best reactive strategy must defect when a half or more of the opponents have defected in the previous round.

A Numerical Example

The results for the above two special cases are consistent with our finding in the analysis under the linear payoff assumption. However, the result may be altered qualitatively in the absence of any restriction on the benefit or cost functions. As an illustration, consider the following payoff functions:

$$V\left(D|k\right)=1-{\left(\frac{n-k+1}{n+1}\right)}^2,$$

(42)

$$V\left(C|k\right)=1-{\left(\frac{n-k+2}{n+1}\right)}^2-\frac{1}{2{\left(n+1\right)}^2},$$

(43)

which satisfies Inequalities (1)–(4), but is not consistent with Equations (30) or (31). To be specific, let us set $N=400$, $n=30$, and $\delta =0.8$. Putting them into Equation (13), we obtain:

$$f\left(k\right)=-\frac{25400}{31}+\frac{146407k}{1922}-\frac{1172k^2}{961}.$$

(44)

In this case, Inequality (16) is met, and as shown in Figure 1b, $f\left(k\right)>0$ holds if and only if $k\ge 14$. Hence, from Equation (17), the best reactive strategy ($d^{\ast }$) satisfies the following:

$$\{\begin{array}{cc}{d}_{\phi }^{\ast }=1,& \\ d_k^{\ast }=1& \mathrm{for} k\ge 13,\\ d_k^{\ast }=0& \mathrm{for} k\le 12.\end{array}$$

(45)

Remember that $d_k$ is defined as the probability with which he cooperates in a given round, provided that $k$ of the $n-1$ “opponents” have cooperated in the preceding round. This demonstrates that there exist situations in which such a strategy that cooperates when a half or more of the opponents have defected in the previous round can be the best reactive strategy.

For comparison, the second and third best strategies following Equation (45) are given by:

$$\{\begin{array}{cc}{d}_{\phi }^{\ast }=1,& \\ d_k^{\ast }=1& \mathrm{for} k\ge 14,\\ d_k^{\ast }=0& \mathrm{for} k\le 13,\end{array}$$

(46)

$$\{\begin{array}{cc}{d}_{\phi }^{\ast }=1,& \\ d_k^{\ast }=1& \mathrm{for} k\ge 12,\\ d_k^{\ast }=0& \mathrm{for} k\le 11,\end{array}$$

(47)

respectively. For $w=0.001$, the approximated fixation probabilities for Equations (45)–(47), calculated from Equation (9), are 0.00254215, 0.00254207, and 0.00254176, respectively.

3.3.2. Evolution of Paradoxical Behavior with Non-Linear Payoff

Non-Increasing Cost and Non-Linear Benefit

Consider the case when there is no restriction on the benefit function (except Inequalities (1)–(4)) and, thus, Equation (30) is not necessarily true. As for the cost function, we partially relax Equation (31) by assuming instead:

$$\Delta c\left(k\right)\le 0,$$

(48)

which means that the cost of cooperation either is constant or decreases with increasing number of cooperators. Equation (13) changes into:

$$f\left(k\right)=\left(N-n\right)\left(V\left(C|k\right)-V\left(D|0\right)\right)-\left(N+1\right)\left(n-k\right)c\left(k\right).$$

(49)

Hence, we have:

$$f\left(k+1\right)=\left(N-n\right)\left(V\left(C|k+1\right)-V\left(D|0\right)\right)-\left(N+1\right)\left(n-k-1\right)c\left(k+1\right).$$

(50)

From Conditions (48)–(50), we have:

$$f\left(k+1\right)>f\left(k\right).$$

(51)

This means that if $f\left(k\right)>0$ then $f\left(m\right)>0$ for any $m>k$. Hence, from Equation (17), a paradoxical strategy cannot be the best reactive strategy.

A Numerical Example

Our analysis thus far has shown that a paradoxical strategy cannot be the best reactive strategy so long as the cost of cooperation is either unaltered or decreased by increasing number of cooperators. However, this may not be the case if the cost of cooperation sometimes increases with the number of cooperators. Let us provide a numerical example in which the best reactive strategy exhibits a paradoxical behavior. We set $N=400$, $n=30$, $\delta =0.8$, and $b=1$ and assume that Equation (30) is met. Instead of Equation (31), here we assume:

$$c\left(k\right)=\{\begin{array}{c}0.07 \mathrm{for} k\ge 20,\\ 0.04 \mathrm{for} k\le 19,\end{array}$$

(52)

for which Inequalities (1)–(4) are satisfied. This represents a case when the cost of cooperation is larger in the presence of more cooperators in the group. Intuitively, in such a situation, a paradoxical strategy that cooperates only when there are relatively few cooperators may be more advantageous than non-paradoxical strategies that, which cooperate whenever the number of cooperators exceeds a threshold. Indeed, since in this case Inequality (16) is met, and $f\left(k\right)$ is non-monotonic in this case (Figure 1c), the best reactive strategy ($d^{\ast }$) is given from Equation (17) by:

$$\{\begin{array}{cc}{d}_{\phi }^{\ast }=1,& \\ d_k^{\ast }=1& \mathrm{for} k=17, 18, k\ge 21,\\ d_k^{\ast }=0& \mathrm{for}k\le 16, k=19, 20.\end{array}$$

(53)

That is, the best reactive strategy cooperates when there are 17 or more opponents who have cooperated in the previous round, except when the number of cooperators is 19 or 20, in which case it defects. The relative advantages of cooperation and defection are determined by a balance between the following two factors: On the one hand, obviously, cooperation is more advantageous when the associated cost is smaller. On the other hand, given that the cost is constant, the relative advantage of cooperation increases with the number of cooperators in the group. In the current example, there is a rise of the cost at $k=20$ (see Equation (52)), which can be compensated for only when the number of cooperating opponents increases to 21 or more.

The second and third best strategies following Equation (53) are given respectively by:

$$\{\begin{array}{cc}{d}_{\phi }^{\ast }=1,& \\ d_k^{\ast }=1& \mathrm{for} k=16, 17, 18, k\ge 21,\\ d_k^{\ast }=0& \mathrm{for}k\le 15, k=19, 20,\end{array}$$

(54)

$$\{\begin{array}{cc}{d}_{\phi }^{\ast }=1,& \\ d_k^{\ast }=1& \mathrm{for} k=18, k\ge 21,\\ d_k^{\ast }=0& \mathrm{for}k\le 17, k=19, 20.\end{array}$$

(55)

For $w=0.001$, the fixation probabilities of strategies Equations (53)–(55) with the small $w$ approximation are 0.00251170, 0.00251155, and 0.00251154, respectively.

3.4. The Best Reactive Strategy under Moderate Selection Intensity

Thus far we have assumed that selection is sufficiently weak ($w\ll 1$) so that the fixation probabilities can be approximated by Equation (9). Here we numerically obtain the exact fixation probabilities using (8) without the weak selection assumption to examine how this may affect the identity of the best reactive strategy.

For simplicity, we make the linear payoff assumption, Equations (19) and (20). Figure 2 illustrates how the best reactive strategy ($d^{\ast }$), obtained by using Equation (8), changes depending on the values of $\delta$ and $w$ for parameter values $b=9$, $c=1$, $N=50$, and $n=10$. For this parameter setting, a comparison of fixation probabilities calculated from Equation (9) would indicate TFT₆ as the best reactive strategy when $\delta >{\delta }^{\ast }$ holds, where ${\delta }^{\ast }\approx 0.49$, and $d_{\phi }=d_0=0$ when $\delta <{\delta }^{\ast }$. This gives a good approximation of the exact numerical solution for $w=0.001$ as shown in Figure 2. However, when $w$ is larger, the approximation is no longer valid. Figure 2 shows that there exist situations in which such a strategy that cooperates when a half or more of the opponents have defected in the previous round can be the best reactive strategy, which cannot be the case in the limit of week selection. A reactive strategy’s fixation probability is determined by its payoffs relative to the unconditional defector at various frequencies ($i=1,2,\dots ,N-1$ in Equation (8). Among the $N-1$ values, a relative payoff at a lower frequency has a greater impact on the fixation probability. Hence, if, as numerically suggested by Kurokawa et al. (2010), the relative payoff at a low frequency tends to be larger for more generous reactive strategies, this is likely to favor the evolution of generosity. This effect might be more pronounced when selection, rather than random drift, plays a greater role. These considerations all in all accord with our numerical example suggesting that more generous reactive strategies are selectively favored when selection is more intense. It was also numerically shown that $\delta$ can affect the identity of the best reactive strategy, while in the limit of weak selection $\delta$ does not affect what the best reactive strategy is as far as it satisfies Inequality (16). In addition, within this particular range of parameter values, we did not find any case in which a paradoxical strategy is the best reactive strategy.

4. Discussion

We have investigated stochastic evolutionary dynamics of a population in which an individual’s fitness is determined by the n-player repeated PD. We have compared the fixation probabilities of different reactive strategies when they appear as a rare mutant in a population of unconditional defectors. Reactive strategies in our analysis are described by a vector $d=\left(d_{\phi }, d_0, d_1, \cdots , d_{n-1}\right)$, where $d_{\phi }$ represents the probability with which the strategy cooperates in the first round and $d_j$ represents the probability with which the strategy cooperates in a given round of a repeated game when $j$ of the $n-1$ opponents have cooperated in the previous round. We have considered a set, $\mathsf{\Omega }$, of all reactive strategies for which $d_{\phi }$ is either 0 or 1 and $d_j$ is either 0 or 1 to specify the best reactive strategy attaining the maximum fixation probability. In a repeated PD, after one round is played, another round will be played with probability $\delta$. Under the assumption of weak selection, we have specified a threshold of $\delta$ below which all strategies satisfying $d_{\phi }=d_0=0$ are the best reactive strategies. We have also found that when $\delta$ exceeds the threshold, $d_{k-1}$ of the best reactive strategy is solely determined by the sign of $f\left(k\right)$, as given by Equation (13), which is interpreted as a weighted sum of the cost and benefit of cooperation when $k-1$ opponents have cooperated in the proceeding round.

The present study extends our previous analysis (Kurokawa et al. (2010), [19]) by considering a broader set of reactive strategies and non-linear payoff functions. We also investigate how robust the indicated identity of the best reactive strategy is to deviations from the weak selection assumption. Kurokawa et al. (2010) specified the best reactive strategy from $n$ reactive strategies classified as TFT_a, which constitute a subset of $\mathsf{\Omega }$, under the conventional linear payoff assumption. One of their findings was that any TFT_a that tolerates defection by more than a half of the group members can never be the best reactive strategy. The present study has shown, on the one hand, that under the linear payoff assumption, the previous finding holds true even for all reactive strategies in $\mathsf{\Omega }$. For non-linear payoff functions, on the other hand, we have shown that the previous finding does not always hold, that is, when the benefit increases in a decelerative manner with the number of cooperators, the best reactive strategy tolerates defection by more than a half of the opponents. We have also found that a strategy tolerating defection by more than a half of the opponents can be the best reactive strategy when selection is not weak.

We have also demonstrated that a paradoxical strategy, in which $d_k>d_{k+1}$ holds for at least some $k$, can be the best reactive strategy when the cost of cooperation increases with the number of cooperators in a group. As far as we know, it has not been pointed out that a conditional cooperator can sometimes do better by behaving paradoxically in competition with defectors. A potentially relevant observation is that human participants in laboratory experiments of the repeated PD sometimes behave as if they were following a paradoxical strategy (e.g., [58,59,60,61]), though these studies assume linear payoff functions. The present study suggests that it may be of interest to investigate the effect of the shape of the payoff functions on the occurrence of paradoxical behaviors in the laboratory setting.

So far, we have examined the fixation probability of strategy $d$ when appeared as a single mutant in a population of ALLD. In Appendix B we also compare the fixation probability of ALLD when introduced as a single mutant in a population of reactive strategy $d$, across all $d$ in $\mathsf{\Omega }$. A reactive strategy associated with a lower fixation probability of ALLD is regarded as more robust against invasion by ALLD. The reactive strategy that minimizes the fixation probability of ALLD turns out to be TFT_n-1, that is, $d=\left(1,0, 0, \cdots , 0, 1\right)$, when $\delta$ is larger than the threshold specified by Inequality (16), and the strategies satisfying $d_{\phi }=d_0=0$ when otherwise. Further, we investigate the ratio of the fixation probability of $d$ in a population of ALLD to the fixation probability of ALLD in a population of $d$, across all $d$ in $\mathsf{\Omega }$. As shown in Appendix C, the reactive strategy maximizing the ratio of the fixation probabilities is TFT_n-1 when $\delta$ is larger than the threshold, and $d_{\phi }=d_0=0$ when otherwise. These results are consistent with our earlier analysis [19], even though the present study considers paradoxical strategies and non-linear payoff functions, which were not considered in the previous study.

Natural selection is regarded as favoring a mutant reactive strategy replacing a population of ALLD if the fixation probability of the reactive strategy exceeds $1/N$, which is the fixation probability for neutral evolution. The reactive strategies that are the best when $\delta$ is small so that Inequality (16) is not satisfied (i.e., $d_{\phi }=d_0=0$) always have the fixation probability $1/N$. Thus, when Inequality (16) is not met, the best reactive strategies and ALLD are selectively neutral. On the other hand, when $\delta$ is large so that Inequality (16) is satisfied, there are other reactive strategies, including the best one, with the fixation probability larger than $1/N$. Hence, in this case, the best reactive strategy is always selectively favored over ALLD. See Appendix D for a further analysis on the case assuming linear payoff functions.

Thus far, our focus has been on the best reactive strategy, which maximizes the fixation probability. However, it is also of interest to examine characteristics of other reactive strategies whose fixation probabilities are relatively high and difference in fixation probabilities among those best strategies. For example,

Table 2. A list of the ten best reactive strategies for $n=10$ under the linear payoff assumption of Equations (19) and (20). Vector $d$ and the corresponding value of $\left(N{\rho }_{d,\mathrm{ALLD}}-1\right)/w$ are given. Parameter values are $N=100$, $b=8$, $c=1$, and $\delta =0.8$.

Ranking	$\mathbf{d}$	$\left(\mathit{N}{\mathbf{\rho }}_{\mathbf{d}, \mathbf{A}\mathbf{L}\mathbf{L}\mathbf{D}}-1\right)/\mathit{w}$
1	(1,0,0,0,0,0,0,1,1,1,1)	40.39
2	(1,0,0,0,0,0,1,1,1,1,1)	38.14
3	(1,0,0,0,0,0,0,0,1,1,1)	36.35
4	(1,0,0,0,0,0,1,0,1,1,1)	34.1
5	(1,0,0,0,0,1,0,1,1,1,1)	31.85
6	(1,0,0,0,0,0,0,1,0,1,1)	30.06
7	(1,0,0,0,0,1,1,1,1,1,1)	29.59
8	(1,0,0,0,0,1,0,0,1,1,1)	27.81
8	(1,0,0,0,0,0,1,1,0,1,1)	27.81
10	(1,0,0,0,0,0,0,0,0,1,1)	26.03

gives a list of ten best strategies in the case of $n=10$ assuming Equations (19) and (20). Figure 3 illustrates the fixation probabilities of the 32 possible strategies in the case of $n=4$ for various values of $\delta$, under the assumption of Equations (19) and (20). The value of $\delta$ makes difference in the order of the strategies. Note that as shown in Appendix A, the value of $\delta$ does not have an effect on the order of the strategies belonging to subset of ${\mathsf{\Omega }}^{d_{\phi },d_0}$.

Finally, let us conclude by mentioning two issues to be addressed in future research. First, actual human behavior can be more complex than that in our model. Although the present study considers a greater variety of reactive strategies than in a previous study [19], the strategy set can be further broadened. Our strategy set, $\mathsf{\Omega }$, does not include, for example, continuous strategies that engage in partial cooperation (see [62]), or those that make decisions referring to their own past behaviors ([63,64,65,66,67,68]. In addition, in our analysis, the benefit, $b$, and the cost, $c$, of cooperation are assumed to be the same for all individuals. Namely, we assumed that there are no differences among agents in this sense. In reality, however, individuals may have different values of $b$ and $c$, and these variations could affect the model outcome [69]. Secondly, we have assumed that perfect information regarding group members’ behaviors is always available. It has been shown, however, that results of related models may be qualitatively altered depending on the accessibility of the relevant information [45,64,70,71,72,73,74,75].