1. Introduction

Natural systems exhibit both ecological and evolutionary dynamics. Nevertheless, many studies focus only on either the ecological or the evolutionary dynamics, assuming the other nonchanging and fixed. There are many excellent exceptions (see, e.g., refs. 1 and 2). Here, we study the ecological and evolutionary dynamics in both ecologically stable as well as periodically fluctuating populations of predator and prey, using the Rosenzweig–MacArthur predator–prey model (3) (see also refs. 45678–9). Our overall aim is to understand under what conditions evolutionarily stable strategies (ESSs) will exist. Specifically, we focus on evolutionary stability in stable as well as periodically fluctuating ecological systems.

The concept ESS was originally coined by Maynard Smith (10) (see also Smith and Price) (11), as an extension of Hamilton’s (12) “unbeatable strategy.” Maynard Smith (10) defined it as a strategy that, once adopted by a majority of the members of a population, cannot be overturned by any alternative strategy that is initially rare. It is essentially a Nash equilibrium that is evolutionary stable (13). The concept of ESS was used initially in behavioral ecology and sociobiology but Lawlor and Smith (14)—see also Roughgarden (15) and Nowak (16)—extended its use to ecological dynamics of competing populations.

Many ecological systems are characterized by fluctuating dynamics, some of which have periodic fluctuations: the famous lemming and small rodent cycles (see, e.g., refs. 1718–19) and the North American hare–lynx cycle (see, e.g., ref. 20). The predator–prey models of the kind presented by Rosenzweig and MacArthur (17) are therefore of broad ecological interest. Several studies have analyzed these—and related—models from an ecoevolutionary perspective (5, 2122232425–26). However, no one has addressed this from an ESS perspective, particularly for the limit-cycle case. This we do in the present contribution.

Models for predatory–prey interactions have played a major role within the field of ecology since the pioneering contribution by Lotka (27, 28) and Volterra (29) (see, e.g., ref. 30). Depending on the choice of parameters, such models are known to accommodate equilibrium populations that are constant or fluctuating (including limit-cycle dynamics).

The biological contribution of this paper will be the theoretical question regarding the existence of ESS in periodically fluctuating populations exemplified by Rosenzweig–MacArthur predator–prey systems. Specifically, we establish sufficient conditions for the existence of ESSs in such models. We consider populations that become stable (constant) over time and periodically fluctuating populations or systems exhibiting stable limit-cycle dynamics. Extending the approach of Reed and Stenseth in ref. 31, we show that ESSs occur at values of the evolutionary strategies that are local optima of certain functions of the model parameters. We identify these functions and show that they have a similar form for both stable and fluctuating populations. These results are illustrated with a concrete example.

The main mathematical contribution is that we prove Lyapunov stability through an adaptation of classical linearization arguments, variation of parameter formulas, and Floquet-type analysis. The main finding is the case of limit cycles, where we establish stability of two-dimensional limit cycles in a four-dimensional system of ordinary differential equations. Within the approach of Reed and Stenseth (31) to evolutionary games and ESS, we then show that these stability results lead to sufficient conditions for ESSs for both types of equilibrium solutions. These sufficient conditions could be of independent interest in game theory.

In section 2, we state the model and results about equilibrium solutions. Then, in section 3, we combine ecology and evolution: evolutionary strategies, an extended ecological model, and the concept of ESS are introduced. The strategies are model parameters that can be changed over time by the populations to ensure a more beneficial evolution, and the extended model includes mutant populations for predators and prey satisfying the same model equations at different (i.e., mutant) values of the strategies. An ESS is introduced as an extension of the definition of Reed and Stenseth (31) to include periodically fluctuating populations—it is a choice of evolutionary strategies for which mutant populations cannot survive in the long run. Evolution can be seen as a competition between different species and be modeled as a game (13). An ESS is then a Nash equilibrium (32), in the sense that no population can improve their own survivability acting alone. A game can only move away from an ESS when the original populations change their strategies. An ESS is therefore a type of locally optimal evolutionary strategy for the original populations.

When time becomes large, the populations will approach limit populations, a (stable) equilibrium solution of the model. Examples of equilibrium solutions are constants (limit points) and periodic solutions (limit cycles). In sections 4 and 5, we study such solutions for the extended ecological model. We first observe that equilibrium solutions of the two-population model remain equilibrium solutions in the extended model when the mutant populations are zero. Then, we establish conditions to guarantee the (Lyapunov) stability of these solutions. Stability here means that solutions starting near an equilibrium solution will over time converge to the equilibrium solution. These results extend earlier results for models with two populations. By taking into account the dependence on the strategies in our extended model, we use the stability results and our definition of ESS to give conditions that guarantee ESS. In section 4, we give the results for equilibrium points (constant populations in the limit), and in section 5, we give the results for limit cycles (periodic limit populations). ESSs are shown to occur at values of the evolutionary strategies that are local optima of certain functions of the model parameters. These results are the main contribution of our paper. In section 4, we also discuss examples of functional dependencies that lead to an ESS: loosely speaking, if we take the predation rate to be an increasing bilinear function of the evolutionary strategies, then we find that the rate constants of prey growth and predator mortality have to be decreasing convex functions. We give a concrete example where the latter two rate constants are quadratic functions of the evolutionary strategies.

2. The Ecology of Predator–Prey Models

Let $$ x_1$$ and $$ y_1$$ denote the population sizes of prey and predator, respectively. We will assume that the dynamics follow a logistic Gause-type model:

This model goes into a long tradition of predator–prey models of the form

For this system, the first quadrant is an invariant region (i.e., the populations can never become negative). Under some reasonable assumptions, the system has three limit points and up to one limit cycle. These results along with stability results are given in the next theorem (also see Fig. 1).

Fig. 1.

The phase diagram of Eq. 1 with $$ a=2.8$$, $$ b_{11}=0.7$$, $$ c_1=1.35$$, $$ r_1=3.5$$, $$ k=1.5$$, and $$ K=1.4$$, which corresponds to case (iii) in Theorem 2.1. (A) The full phase diagram. Dashed curves as well as the coordinate axes are isoclines. The brown points give the stationary points. The red curve is the limit cycle. (B) A close-up of the full diagram in A. If $$ K>b_{11}+2\widehat{x}$$, then $$ \left(\widehat{x},ŷ\right)$$ is unstable and there exists exactly one limit cycle in the first quadrant in the $$ \left(x_1,y_1\right)$$ plane, which is an (asymptotically) stable limit cycle.

3. Evolution: The Extended Ecological Model

Evolutionarily, there is a “conflict” or “arms race” between the predator and the prey regarding predation: the prey “wants” to evolve to avoid being caught by the predator, whereas the predator “wants” to be able to catch prey as efficiently as possible, even when the prey is at low abundance. However, this will come at some costs: we assume that for the prey, a decreased predation rate will lead to a reduced growth rate, whereas for the predator, an increased predation rate may lead to an increased mortality rate. A natural way to analyze this situation is through (differential) game theory and concepts like Nash equilibriums and ESS; we refer to refs. 13, 14, 40, and 41 for more information. We will follow the simplified approach of Reed and Stenseth (31), an approach explicitly emphasizing that a new mutant will have to establish itself through competition, even though rare initially.

Let $$ {\alpha }_1$$ and $$ {\beta }_1$$ represent the strategies of prey and predator, respectively. All model parameters are assumed to depend on $$ {\alpha }_1$$ and $$ {\beta }_1$$, i.e., different mutations correspond to different values of the various constants in Eqs. 1–3. The idea is that over time, the populations can modify their own strategies in order to increase their chances of survival.

We now make some assumptions about how the different model parameters depend on $$ {\alpha }_1$$ and $$ {\beta }_1$$. We know from, e.g., studies of the hare–lynx cycle (42) that the predator and the prey mutually “disagree” on the value of the half-saturation constant; for instance, large $$ b$$ and hence low predation is beneficial for the hare, while the opposite is the case for the lynx. We therefore take

In the next step, we extend the model to include mutants. Let $$ x_1$$ and $$ y_1$$ denote the original prey and predator populations, respectively, and $$ x_2$$ and $$ y_2$$ denote the corresponding mutant populations. The extended mode then takes the form

We now introduce evolutionary strategies $$ \left({\alpha }_1,{\beta }_1\right)$$, $$ \left({\alpha }_2,{\beta }_2\right)$$ for $$ \left(x_1,y_1\right)$$, $$ \left(x_2,y_2\right)$$, respectively. As before, we assume for $$ i=\mathrm{1,2}$$ that

4. Equilibrium Points, Stability, and ESS

SI Appendix, Eq. 1 has three equilibrium points: $$ \left(\mathrm{0,0}\right)$$, $$ \left(K,0\right)$$, and $$ \left(\widehat{x},ŷ\right)$$, where

We show in SI Appendix that the eigenvalues $$ {\lambda }_3$$ and $$ {\lambda }_4$$ for the Jacobi matrix $$ dM$$ of the full system Eq. 9 are

Theorem 4.2 (Conditions for ESS—Equilibrium Point Case)

Assume Eq. 12. A constant evolutionary strategy $$ \left(\widehat{\alpha },\widehat{\beta }\right)$$ is an ESS for Eq. 14 if

(i)$$ \sigma =\frac{ak}{c\left(\widehat{\beta }\right)}>1$$,

(ii)$$ \widehat{x}<K<b\left(\widehat{\alpha },\widehat{\beta }\right)+2\widehat{x}$$,

(iii)the function $$ e_1\left({\alpha }_2\right)=r\left({\alpha }_2\right)\left(1-\frac{\widehat{x}}{K}\right)\left(b\left({\alpha }_2,\widehat{\beta }\right)+\widehat{x}\right)$$ has a strict local maximum at the point $$ {\alpha }_2=\widehat{\alpha }$$,

(iv)the function $$ e_2\left({\beta }_2\right)=c\left({\beta }_2\right)b\left(\widehat{\alpha },{\beta }_2\right){\left(ka-c\left({\beta }_2\right)\right)}^{-1}$$ has a strict local minimum at the point $$ {\beta }_2=\widehat{\beta }$$.

Proof:

Note that by the definition of $$ \widehat{x}$$, the second inequality in Lemma 4.1 can be written as $$ c_1{b}_{11}/\left(ak-c_1\right)<c_2{b}_{12}/\left(ak-c_2\right)$$ when $$ ak-c_2>0$$. The result then follows from Eq. 12, the definition of an ESS, and Lemma 4.1.

Now, we consider an example where $$ b$$, $$ r$$, and $$ c$$ are given as bilinear, quadratic, and rational functions, respectively, of $$ \alpha $$ and $$ \beta $$. In this case, we use Theorem 4.2 to find the values of the ESS. We also check that Eq. 13 is satisfied near these ESSs.

Example 4.3:

Assume $$ b\left(\alpha ,\beta \right)=B\alpha \beta $$, $$ c\left(\beta \right)=ak\left(C_1{\left(\beta -{\beta }_0\right)}^2+C_2\right)/\left(1+C_1{\left(\beta -{\beta }_0\right)}^2+C_2\right)$$, and $$ r\left(\alpha \right)=R{\left(\alpha -{\alpha }_0\right)}^2$$ for positive constants $$ B,C_1,C_2,R,{\alpha }_0,{\beta }_0$$. Then, the conditions of Theorem 4.2 are satisfied for $$ \left(\widehat{\alpha },\widehat{\beta }\right)$$, where

$\begin{array}{ll}\hfill \widehat{\alpha }& =\frac{ak-c\left(\widehat{\beta }\right)}{3ak-c\left(\widehat{\beta }\right)}{\alpha }_0,\hfill \\ \hfill \widehat{\beta }& =\frac{1}{3}\left(2{\beta }_0+{\left({\beta }_0^2-\frac{3C_2}{C_1}\right)}^{1/2}\right),\hfill \end{array}$

under the conditions

$\begin{array}{ll}\hfill {\beta }_0^2& >\frac{3C_2}{C_1},\hfill \\ \hfill 1<\frac{K}{B\widehat{\alpha }\widehat{\beta }}\left(\frac{ak}{c\left(\widehat{\beta }\right)}-1\right)& <1+\frac{ak}{c\left(\widehat{\beta }\right)}.\hfill \end{array}$

Note that $$ \widehat{\alpha }\in \left(0,{\alpha }_0\right)$$ and $$ \widehat{\beta }\in \left(0,{\beta }_0\right)$$. We conclude that $$ \left(\widehat{\alpha },\widehat{\beta }\right)$$ is an ESS for Eq. 14. From a direct calculation, it now follows that

$\frac{dr}{d\alpha }\left(\widehat{\alpha }\right)<0,\hspace{1em}\frac{dc}{d\beta }\left(\widehat{\beta }\right)<0,\hspace{1em}\frac{d^{2}r}{d{\alpha }^2}\left(\widehat{\alpha }\right)>0,\text{and}\hspace{1em}\frac{d^{2}c}{d{\beta }^2}\left(\widehat{\beta }\right)>0.$

In other words, the natural relations Eq. 13 hold at the ESS, and the functions $$ r$$ and $$ c$$ are convex decreasing at the ESS (Figs. 2 and 3).

Fig. 2.

The function $$ r\left(\alpha \right)$$.

Fig. 3.

The function $$ c\left(\beta \right)$$. Note that $$ C_2\to 0$$ implies $$ \widehat{\beta }\to {\beta }_0$$.

Note that we need $$ C_2>0$$. If $$ C_2=0$$, then $$ \widehat{\beta }={\beta }_0$$, $$ c\left(\widehat{\beta }\right)=0$$, and $$ \frac{dc}{d\beta }\left(\widehat{\beta }\right)=0$$. However, this is unrealistic since the predator mortality rate constant $$ c$$ must be strictly positive. Eq. 13 is also violated. See SI Appendix, Remark 9.7 for further computations on this example.

Biologically, this example may be interpreted as follows. An ESS will exist when the evolutionary strategies $$ {\alpha }_1$$ and $$ {\beta }_1$$ are linked, respectively, to the logistic growth rate constant $$ r$$ for the prey and the mortality rate constant $$ c$$ for the predator, in a decreasing convex fashion, while at the same time, the predation half-loading constant $$ b$$ is bilinearly increasing.

5. Limit Cycles, Stability, and ESS

We now find conditions for when there exist (asymptotically) stable limit-cycle solutions of Eq. 9 with the two last components $$ x_2$$ and $$ y_2$$ equal to 0. Thereafter, we will look at the parameter dependent system Eq. 14 and identify ESSs. When $$ x_2$$ and $$ y_2$$ are identically 0, our model reduces to Eq. 1. To prove the existence of a limit cycle inside the first quadrant, both the position and the stability of the three equilibrium points $$ \left(\mathrm{0,0}\right)$$, $$ \left(K,0\right)$$, and $$ \left(\widehat{x},ŷ\right)$$ play an essential role. If there exists a limit cycle inside the first quadrant, it must surround at least one equilibrium point. In our case it must surround $$ \left(\widehat{x},ŷ\right)$$, which lies in the first quadrant by Theorem 2.1 if we assume that $$ {\sigma }_1=\frac{ak}{c_1}>1$$. By the same theorem, it follows that $$ \left(\widehat{x},ŷ\right)$$ is a stable node or spiral if $$ K<b_{11}+2\widehat{x}$$ and all phase paths inside the first quadrant end up in $$ \left(\widehat{x},ŷ\right)$$. In this case there cannot exist a limit cycle. If, on the other hand, $$ K>b_{11}+2\widehat{x}$$, then there exists at least one limit cycle surrounding $$ \left(\widehat{x},ŷ\right)$$. To establish the existence of an asymptotically stable limit cycle is much harder than establishing the existence of an asymptotically stable equilibrium point, but the main ideas behind are quite similar.

In ref. 37, it has been shown that every limit cycle surrounding $$ \left(\widehat{x},ŷ\right)$$ is asymptotically stable. As an immediate consequence it then follows that there is at most one limit cycle in the first quadrant, since an asymptotically stable limit cycle cannot be surrounded by another asymptotically stable limit cycle. The key ingredient in the argument is the following theorem, which is taken from ref. 48 (Corollary, p. 216) and adjusted to our notation and assumptions.

6. Red Queen Type of Continued Evolution or Stasis

Our analysis is related to the issue of Red Queen type of continued evolutionary evolution of stasis (cf. Van Valen) (49); see also Schaffer and Rosenzweig (50) and Rosenzweig et al. (51). In Example 4.3, we have assumed that both the predator and the prey have equal influence on the determination of the overall half-saturation parameter $$ b$$. For this system, we find that evolution will lead to an ESS without evolutionarily fluctuating dynamics. If the two interacting species (the predator and prey) had asymmetric influence on the overall half-saturation constant, this might not be the case [see, e.g., Nordbotten and Stenseth (52)]. A further analysis of this would indeed be worthwhile. The contributions by Dercole and coworkers (44, 45), Doebeli (46), and Cortez and Weitz (26) are all important stepping stones in such further analysis.

7. Discussion

We establish conditions that guarantee stability of equilibrium solutions of the extended ecological model. This model has four equations, and, for periodic solutions, such a result appears to be previously undescribed. We prove full Lyapunov stability of the periodic solutions. The proof relies on an adaptation of classical linearization arguments, variation of parameter formulas, as well as Floquet-type analysis. The approach of Reed and Stenseth in ref. 31 to evolutionary games and ESS is extended to include fluctuating populations. ESSs are then shown to occur at values of the evolutionary strategies that are local optima of certain functions of the model parameters. We identify these functions and express them in a similar way for both stable and fluctuating populations. A concrete example to illustrate our results is given. Mathematically, these results are obtained from the abovementioned stability results, and our characterizations of ESSs seem to be previously undescribed and could be of independent interest in game theory.

The concept of ESSs was developed with a stable ecological setting in mind. However, typically ecological systems vary in time, often with more or less periodic fluctuations. Predator–prey systems are such examples. If we are to link ecology and evolution, we must allow for varying ecological population fluctuations. With this contribution, we have extended the ESS concept to be applicable for periodically fluctuating ecological systems.

The application of ESS to ecology is similar to the “adaptive dynamics” approach in that it models the time evolution of populations. See Dieckmann et al. (53); see also Kang and Fewell (54). The “adaptive dynamics” approach focuses on how strategies evolved under changing ecological conditions. The ESS approach, on the other hand, focuses on finding the unbeatable fixed strategy within a population. To our knowledge, it has not been shown before that such unbeatable ESS strategies do exist when the ecological system exhibits a periodically fluctuating dynamics. With this contribution, we derive the mathematical conditions for such fixed ESSs to exist when the ecological system exhibits limit cycle-type dynamics.

By extending the approach presented by Reed and Stenseth (31), it furthermore becomes clear that evolution occurs through the ecological process of a variant type being able to invade an ecological system (or population) when it is at its stable state—be this a stable equilibrium or a limit cycle. Whenever such ecological invasion occurs (i.e., the wild type is not an ESS), evolution will occur—and the ecology of the system changes. This emphasizes the interlinkages between ecology and evolution: it is a matter of realizing both that “nothing in biology makes sense except in the light of evolution” (55) as well as “very little in evolution makes sense except in the light of ecology” (56).