Chapter 3. The Importance of Model Analysis Prior to Artificial Society Simulation and
3.3 Conclusion
By analyzing various relationships among model components, model scope can be defined, model complexity reduced, and appropriate parameter settings identified—all
helpful in terms of increasing simulation efficiency. Theoretical analyses can reduce and/or complement the weaknesses of agent-based simulations and computations. For simple models, analysis is required to determine why a simulation is needed and why certain parameters should be chosen. For complex models, analysis reduces unnecessary work and guides the direction of a simulation toward discovery. An analysis of relationships between or among model components provides a global view of a model’s scope, and helps to establish important strategies and appropriate parameter settings.
Although I want to emphasize the importance of pre-execution/simulation analysis, I am not claiming that analytical approaches can replace simulations, nor that they are superior to simulation approaches in any other manner. In each case, advantages and disadvantages are clear. The integration of simulative and analytical approaches will be an increasingly important topic in future execution/simulation studies for artificial societies.
Chapter 4.
The Analytical Framework for IPD Models
4.1 Definitions
In the framework described here, a finite state machine is used to represent interactions between deterministic memory-n strategies. The finite state machines (sometimes referred to as finite automata) are dynamic systems that only change their behavior at discrete moments under consideration. The system consists of a finite set of internal states and a transition function. The transition function determines a subsequent system state as a function of the current state plus input. In [93], a formal definition of a finite state was given as:
Definition 4.1 A finite-state machine (finite automata) consists of a 5-tuple (Q, ∑, q0, δ, A),
where
Q is a finite set (whose elements we will think of as states),
∑ is a finite alphabet set of input symbols,
q0 ∈ Q (the initial state),
A ⊆ Q (the set of accepting states), and
δ is a function from Q × ∑ to Q (the transition function).
For any element q of Q and any symbol a∈∑, δ(q,a) is interpreted as the state to which the
finite state machine moves if it is in state q and receives the input a.
A finite state machine can be represented in terms of a state transition diagram, described as:
Definition 4.2 A finite state machine (Q, ∑, q0, δ, A) can be represented as a state
transition diagram {V, E}, where
V is a finite set of vertices, and each vertex represents a state in set Q,
and (qi, qj) ∈ E if δ(qi,a) = qj, where a∈∑.
The diagram is a directed graph. The initial state q0 and the set of accepting states A are marked with specific notation. Each edge (qi, qj) is marked with an input symbol a, indicating that δ(qi,a) = qj.
Regarding the memory-n strategies, cooperative or defection moves are determined by the historical moves of two players. A memory-n strategy determines a move in correspondence to moves made during the last n round, which is represented as a history string (HS). Formally defined,
Definition 4.3 Let HS be the history string of Si in its interaction with Sj, where Si and Sj
are memory-n strategies. Then,
HS=h2nh2n-1…,hk+1hk,…h2h1.
The even and odd indices indicate the respective moves of a player and an opponent.
Here hk∈{C, D}. C represents a cooperative move and D represents a defection move.
Note that in this case, HS’ (the history string of Sj,) would be
HS’=h2n-1h2n…,hkhk+1,…h1h2.
In other words, the two strategies have different history strings. In all, there are 22n distinct history strings for a memory-n deterministic strategy. For convenience, the
following formal definition of the order of history strings for a memory-n strategy is offered.
Definition 4.4 {HSi/N | 0≤ i <N and N=22n } is the set of all possible history strings for a memory-n strategy, where HSi/N is the ith string in the lexicographic order:
HS0/N = CC…CC,
HS1/N = CC…CD,
HS2/N = CC…DC,
HS3/N = CC…DD,
…
HSN-1/N = DD…DD,
where |HS0/N|=|HS1/N|=…=|HSN-1/N|=2n.
A memory-n strategy is described in terms of the moves that are made according to their history strings.
Definition 4.5. A memory-n deterministic strategy S is represented as:
where Pk∈{C, D}, and N=22n. Pk represents the move S corresponding to history string HSk/N.
A memory-1 strategy is expressed as (P0, P1, P2, P3), where P0, P1, P2, P3 indicate moves when the results of the preceding round are CC, CD, DC, and DD, respectively.
For example, strategy (C D C D) indicates a cooperative move if the preceding round is either CC or DC; a defection move is indicated in all other cases.
After finishing a round, the history strings of both strategies are updated by deleting the two leftmost characters, and adding the result of the next round to the right. For example, if a current history string is CCDDCD and the last round is DC, then the history string becomes DDCDDC. Interactions between player strategies can be viewed as history string transitions. Interactions between two strategies can be expressed as a finite state machine, where each state represents a specific history string, and where transitions between states are dependent upon transitions between the strategies’ history strings.
Proposition 4.1 Interactions between two memory-n deterministic strategies can be
represented as a finite state machine.
Proof:
Assume two memory-n deterministic strategies, Si and Sj, represented as
Si = (P0, P1, …PN-2, PN-1), and
Sj = (P0’, P1’, …PN-2’, PN-1’),
where N=22n and Pk and Pk’ represent the moves of Si and Sj corresponding to history string HSk/N.
Let FSM(Si|Sj) be a finite state machine whose state transition diagram is:
D(Si|Sj) = {V, E}.
The set of vertices V and the set of edges E are defined as
V = {Vk | Vk is a vertex corresponding to history string HSk/N, 0≤k<N, N=22n }, and
E = {(Vu, Vv) | Vu, Vv ∈ V,
where Si’s history string may transit from HSu/N to HSv/N when interacting with Sj, and 0≤u,v<N }
In V, Vk is the vertex corresponding to history string HSk/N. Since there are N vertices, all possible history string permutations are contained.
In E, the edge (Vu, Vv) indicates that the history string of Si will transit from HSu/N to
Interactions between Si and Sj can be represented by D(Si|Sj) without loss of information. Thus, interactions between two memory-n deterministic strategies can be represented as a finite state machine.
Note that the history strings in V and E refer to the history strings of Si, which are different from those of Sj. D(Si|Sj) represents the interaction between Si and Sj from Si’s perspective. The state transition diagram that represents the interaction between Si and Sj
from Sj’s perspective is denoted as D(Sj|Si).
□
Definition 4.6 The finite state machine that represents interactions between two memory-n
deterministic strategies Si and Sj from Si’s perspective is denoted as FSM(Si|Sj). Its state transition diagram is D(Si|Sj). Thus,
D(Si|Sj) = {V, E},
where
V={Vk | Vk is a vertex corresponding to history string HSk/N, 0≤k<N, and N=22n}, and
E={(Vu, Vv) | Vu, Vv∈V, where
Si’s history string may transit from HSu/N to HSv/N when interacting with Sj for 0≤u<N }.
In common finite state machines, state transitions depend on the current state and input symbols. However, the input symbol in FSM(Si|Sj) is ignored; another way of saying this is that there is only one kind of input symbol, so the state transition depends only on the current state.
To give a simple example, if Si = (C, D, C, D) and Sj = (C, D, D, C), the interactions are represented as a finite state machine FSM(Si|Sj), whose state transition diagram D(Si|Sj) is shown as Figure 4.1. If Si and Sj both made defection moves in the previous round, Si
will make another defection move and Sj will make a cooperative move, resulting in a transition from state DD to state DC.
CC CD DC DD
Figure 4.1: The Interaction between Strategies (C, D, C, D) and (C, D, D, C).
4.2 Behavior Characteristics of Infinite Duration
Proposition 4.2 Let Si and Sj be two memory-n deterministic strategies. For the finite state machine FSM(Si|Sj), there is only one outgoing link for each state.
Proof:
Assume
Si = (P0, P1, …PN-2, PN-1) and
Sj = (P0’, P1’, …PN-2’, PN-1’),
where N=22n. In FSM(Si|Sj), each state represents a particular history string. Let the set of states be
Q={qk| qk represents history string HSk/N, and 0≤k<N }.
The state transition is determined by the transition of history strings. If the current history string of Si is HSk/N, then
HSk/N=h2nh2n-1…h2h1.
In the next round the history string of Si will transit to
h2n-2h2n-3…h2h1PkPk’.
Since Pk, Pk’ ∈{C,D}, the history string h2n-2h2n-3…h2h1PkPk’ is also an element in the set {HSk/N | 0≤i<N and N=22n }. Assume that
HSm/N=h2n-2h2n-3…h2h1PkPk’.
Each time Si interacts with Sj and Si’s history string is HSk/N, Si’s history string will always transit to HSm/N. This means that in FSM(Si|Sj), if the current state is qk, the next state will always be qm. Thus, there is only one outgoing link for each state in FSM(Si|Sj).
□
Following proposition 4.2, it is possible to prove that interactions between two memory-n deterministic strategies are periodically repeated.
Proposition 4.3 The behavior of a finite state machine with only one kind of input symbol
and only one outgoing link for each state is periodically repeated for any initial state; the maximum loop length equals the number of total states.
Proof:
Assume that FSM is a finite state machine which has only one kind of input symbol and one outgoing link for each state. Let N be the number of states and qs the initial state.
qsqs+1…qs+N-1qs+N….
According to the pigeonhole principle, since there are only N states, then qs+N=qs+k for some k which satisfies 0≤k<N. The state transition sequence is
qsqs+1…qs+kqs+k+1…qs+N-1qs+N…
From Proposition 4.2, we have qs+k+1=qs+N+1, qs+k+2=qs+N+2,…. The state transition sequence can therefore be expressed as
qsqs+1…qs+kqs+k+1…qs+N-1qs+kqs+k+1…qs+N-1….
The same sequence can be represented as
qsqs+1…(qs+kqs+k+1…qs+N-1)*,
where the asterisk indicates that the state transition is a repetition of (qs+kqs+k+1…qs+N-1).
Thus, the machine behavior is periodically repeated.
Furthermore, by letting the length of the loop (qs+kqs+k+1…qs+N-1) be l, then l=N-k.
Since 0≤k<N, l satisfies 0<l≤N, meaning that the maximum length of the loop is N.
□
Proposition 4.4 Interactions between two memory-n deterministic strategies are
periodically repeated for any initial condition; maximum loop length is 22n.
Proof:
Let Si and Sj be two memory-n deterministic strategies. FSM(Si|Sj) represents the interaction between Si and Sj from Si’s perspective. The input symbol in FSM(Si|Sj) is ignored, or it can be said that there is only one kind of input symbol. According to Proposition 4.2, there is only one outgoing link for each state, and according to Proposition 4.3, the behavior of FSM(Si|Sj) is periodically repeated. Furthermore, since the maximum number of states in FSM(Si|Sj) is 22n, the maximum loop length is also 22n.
□
The next formal definition is for the loop that FSM(Si|Sj) falls into for a specific initial state.
Definition 4.7. FSM(Si|Sj) represents interactions between memory-n strategies Si and Sj. Its state transition diagram is D(Si|Sj).
Let Lk be the loop which D(Si|Sj) will fall into if the initial vertex (initial state) is Vk. Then,
)
If the initial vertex is Vk, then after a sufficient period the probability that D(Si|Sj) is at state lk,m is 1/|Lk|. Taking all initial states into account, the probability that it is at Vm at
Definition 4.8 FSM(Si|Sj) is a finite state machine that represents interactions between strategies Si and Sj. Its state transition diagram is D(Si|Sj). The traversal probability for each state is:
PS(Si|Sj) = (PS0, PS1, … , PSN-1), where
∞
Recall that each state represents a specific history string whose final two characters represent the result of the previous round. Since these two characters indicate the payoff of the previous round, each state is mapped to one of the four payoffs. The traversal probability accumulates to derive the probability of each kind of payoff. That is,
FCC = PS0 + PS4 + … + PSN-4 is the probability that the result of the previous game was
When Si interacts with Sj, the behavior characteristic of Si is defined as
B(Si|Sj) = (FCC, FCD, FDC, FDD), where
FCC = PS0 + PS4 + … + PSN-4,
FCD = PS1 + PS5 + … + PSN-3,
FDC = PS2 + PS6 + … + PSN-2, and
FDD = PS3 + PS7 + … + PSN-1.
(PS0, PS1, … , PSN-1) is the traversal probability for each state in FSM(Si|Sj).
From the behavior characteristic, we can see how the two strategies interact. FCC
indicates the probability that both will cooperate when they interact; if FCC is high, so is the likelihood of cooperation between the two.
Definition 4.10 The payoff characteristic between Si and Sj from Si’s perspective is
E(Si|Sj) = FCC×R + FCD×S + FDC×T + FDD×P,
given that the payoff matrix is [R, S, T, P].
The payoff characteristic is the expected payoff when two strategies interact. In previous studies, the expected payoff between stochastic strategies is usually derived via the Markov process. In stochastic strategies, initial conditions (initial moves) do not
affect the expected payoff derived by the Markov process. However, when dealing with deterministic strategies, initial moves do affect the expected payoff. The payoff characteristics of deterministic strategies take into account all possible initial moves. The payoff characteristics denote the interaction between two strategies without information loss.
4.3 Two Criteria for Investigating Strategy Properties
Based on the behavior characteristic B(Si|Sj) and the payoff characteristic E(Si|Sj), two criteria were established for examining strategy properties: the ability to exploit others and the ability to form clone clusters.
4.3.1 Ability to Exploit Others
Assume strategies Si and Sj. Behavior characteristics between them are stated as:
B(Si|Sj)=(FCC, FCD, FDC, FDD), and
B(Sj|Si)=(FCC’, FCD’, FDC’, FDD’).
E(Si|Sj)= FCC×R + FCD×S + FDC×T + FDD×P, and
E(Sj|Si)= FCC’×R + FCD’×S + FDC’×T + FDD’×P.
The ability to exploit others is determined by the relationship between E(Si|Sj) and E(Sj|Si).
Definition 4.11 When Si interacts with Sj, Sj is said to exploit Si if E(Si|Sj)<E(Sj|Si). This is expressed as:
Si→Sj if E(Si|Sj)<E(Sj|Si), meaning that Sj exploits Si, and
Si=Sj if E(Si|Sj)=E(Sj|Si), meaning that Si and Sj receive the same payoff.
Note that the combination of Si→Sj and Sj→Sk does not imply Si→Sk; the relationship between Si and Sk is determined by E(Si|Sk) and E(Sk|Si). However,
Si→Sj implies E(Si|Sj) < E(Sj|Si), and
Sj→Sk implies E(Sj|Sk) < E(Sk|Sj).
From these two inequalities only, nothing is known about E(Si|Sk) and E(Sk|Si); the relationship between Si and Sk remains undetermined. Si→Sj, Sj→Sk, and Si→Sk denote Si→Sj→Sk.
The exploitation chain is used to indicate relationships among more than two strategies.
Definition 4.12. An exploitation chain is expressed as:
Ck
C
C S S
S → →...→
2 1
Strategy exploits if j<i.
Ci
S SCj
4.3.2 Ability to Form Clone Clusters
The ability of strategies to form clone clusters is determined by their behavior characteristics with their clones—i.e., E(Si|Si). If E(Si|Si) is high, then two players using the same strategy Si will receive relatively higher payoffs when they interact. This effect is more apparent in spatial models, in which strategies only interact with their neighbors [65-68].
According to Smith’s [60] definition, Si will invade Sj if
E(Si|Si) > E(Sj|Si), or if
E(Si|Si) = E(Sj|Si) and E(Si|Sj) > E(Sj|Sj).
The invasion relationship between the two strategies is determined by E(Si|Si), E(Sj|Sj), E(Sj|Si), and E(Si|Sj). A rather complex analytical task results, since the following are all possible:
Si invades Sj and Sj invades Si,
Si invades Sj and Sj can’t invade Si,
Si can’t invade Sj and Sj invades Si, or
Si can’t invade Sj and Sj can’t invade Si.
Furthermore, Si’s invasion of Sj and Sj’s invasion of Sk do not imply that Si will invade Sk. As the number of strategies increases, relationships among them eventually become too complex to analyze.
According to the proposed framework, the relationships between E(Si|Sj) and E(Sj|Si) and between E(Si|Si) and E(Sj|Sj) are investigated separately. Complex invasion relationships can be divided into two categories, which makes it easier to clarify relationships among the strategies, and which helps investigators to find and analyze hidden properties.
4.4 Conclusion
The three most important properties of the framework are:
a) It remains independent of payoff matrix values as long as they satisfy the constraint of the Prisoner’s Dilemma.
b) Interactions between strategies can be represented as a finite state machine. This kind of representation also provides an efficient means for deriving the expected payoff between deterministic strategies—a task that is difficult by using the Markov process because the result of deterministic strategies is sensitive to the initial state.
c) A strategy’s ability to exploit others and to form clone clusters provides stepping-off points for further analysis and simulation. Using these two criteria, relations among strategies can be identified.
Chapter 5.
Analysis of Memory-1 Deterministic Strategies in IPD
5.1 Exploitation Relation
A memory-1 strategy is notated as (P0, P1, P2, P3). The 24 deterministic strategies are named S0, S1, …, S15, representing (C, C, C, C), (C, C, C, D), …, and (D, D, D, D), respectively. The relation among strategies’ ability to exploit others is known as the exploitation relation. Exploitation relations between paired memory-1 deterministic
strategies are listed in Table 5.1.
Assume an element Tij in row i and column j.
Tij=0 if Si=Sj,
Tij=1 if Sj→Si, and
Tij=-1 if Si→Sj.
Table 5.1: Exploitation Relations between Paired Memory-1 Strategies
S0 S1 S2 S3 S4 S5 S6 S7 S8 S9 S10 S11 S12 S13 S14 S15 S0 0 0 -1 -1 0 0 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 S1 0 0 -1 -1 0 0 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 S2 1 1 0 -1 0 0 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 S3 1 1 1 0 1 0 0 -1 1 0 0 -1 0 -1 -1 -1 S4 0 0 0 -1 0 0 0 -1 0 0 -1 -1 0 0 -1 -1 S5 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 S6 1 1 1 0 0 0 0 -1 1 1 0 -1 0 0 -1 -1 S7 1 1 1 1 1 0 1 0 1 1 1 1 1 0 1 0 S8 1 1 1 -1 0 0 -1 -1 0 -1 -1 -1 0 -1 -1 -1 S9 1 1 1 0 0 0 -1 -1 1 0 0 -1 0 0 -1 -1 S10 1 1 1 0 1 0 0 -1 1 0 0 -1 0 -1 -1 -1 S11 1 1 1 1 1 0 1 -1 1 1 1 0 1 0 1 -1 S12 1 1 1 0 0 0 0 -1 0 0 0 -1 0 0 0 -1 S13 1 1 1 1 0 0 0 0 1 0 1 0 0 0 0 0 S14 1 1 1 1 1 0 1 -1 1 1 1 -1 0 0 0 -1 S15 1 1 1 1 1 0 1 0 1 1 1 1 1 0 1 0
5.1.1 Strategy Classification
WINk = {Si | Si→Sk},
LOSEk = {Si | Sk→Si}, and
DRAWk = {Si | Sk=Si}.
WINk is a set of strategies exploited by Sk. LOSEk is a set of strategies that may exploit Sk. DRAWk contains strategies that result in equal payoffs with Sk.
The exploitation relations among these 16 strategies are constructed by WINk, LOSEk, and DRAWk (0≤k≤15). Strategies are said to be equivalent if they have the same WIN, LOSE, and DRAW sets. There are three sets of equivalent strategies: { S0, S1 }, { S3, S10 }, and { S7, S15 }. The exploitation relations among them form an exploitation chain, shown as { S0, S1 }→{ S3, S10 }→{ S7, S15 }.
Statement 5.1 { S0, S1 }, { S3, S10 }, and { S7, S15 } are sets of equivalent strategies, and the exploitation chain they form is { S0, S1 }→{ S3, S10 }→{ S7, S15 }.
5.1.2 Exploitation Chains
Based on Statement 5.1:
Statement 5.2 { S0, S1 }→S2→S8→{ S3, S10 }→S11→S14→{ S7, S15 }.
Statement 5.3 S4→{ S3, S10 }→S11→S14→{ S7, S15 }.
Statement 5.4 { S0, S1 }→S2→S8→{ S3, S10 }→S13.
Statement 5.5 { S0, S1 }→S2→S12→S11→{ S7, S15 }.
Statement 5.6 { S0, S1 }→S2→S8→S9→S6→S11→S14→{ S7, S15 }.
S2 S8
S3 S10 S4
S12
S9 S6
S14 S11
S0 S1
S13
S7 S15
Figure 5.1: Exploitation Chains of Memory-1 Deterministic Strategies.
Relations among these strategies are shown in Figure 5.1. In most cases, if a path exists from Si to Sj, then Si→Sj. There is one exception: a path exists from S4 to S13, but S4=S13. This exception is described in the next section.
5.1.3 The Draw Case
All relations not specified in the preceding section are considered draw relations, meaning that two interacting strategies will receive the same payoff if no path exists between them. A path does exist from S4 to S13, but to describe the exception that S4=S13, a link must be added between the two strategies. In the case of Si and Sj, that link is Si=Sj. The exploitation relation between S4 and S13 is shown below.
S5, which receives the same payoff when it interacts with any other memory-1 deterministic strategy, is shown as an isolated vertex in Figure 5.2. This is the well-known strategy Tit-for-Tat.
S13 S4
S2 S8
S3 S10
S4 S13
S12
S9 S6
S14 S11 S5
S0 S1 S7 S15
Figure 5.2: The Exploitation Relation among Memory-1 Deterministic Strategies.
The exploitation relation of all memory-1 deterministic strategies is shown in Figure 5.2. The principle for determining the ability of Si and Sj to exploit others is:
Si→Sj if there is a traversal path from Si to Sj;
Sj→Si if there is a traversal path from Sj to Si; and
Si=Sj if there is no path from Si to Sj, or if there is a ‘=’ link between them.
5.2 Clustering Relation
The relation of the ability of strategies to form clone clusters is known as the clustering relation. Strategy Si’s ability to cluster is determined by E(Si|Si), the payoff characteristic resulting from interactions with its clones. Table 5.2 presents a list of E(Si|Si) for all memory-1 deterministic strategies.
Recall that R, S, T, and P in the payoff matrix of the Prisoner’s Dilemma satisfy T>R>P>S, and that 2R>S+T. According to this constraint, the clustering relations of all memory-1 deterministic strategies are:
If S+T > R+P:
If S+T < R+P:
The relations between {S1} and {S2, S4}, {S8, S14} and {S3, S5, S10, S12,}, and {S7}
Figure 5.3: The Ability to Form Clone Cluster for Memory-1 Deterministic Strategies.
It can be argued that the use of a value-independent analysis contradicts one of our stated reasons for performing a pre-simulation analysis in Chapter 3—that is, defining model scope—since model scope may depend on the payoff matrix value. The payoff matrix values indeed may affect the analytical result of relation between strategies.
However, I firmly believe that neglecting to perform an analysis of relations that are independent of payoff matrix values would make it difficult to determine whether results
value-independent analysis highlights relations based on the native properties of the IPD model, while a value-dependent analysis emphasizes how different values affect exploitation and clustering relations.
5.3 Properties of Memory-1 Deterministic Strategies
5.3.1 Relation among Memory-1 Deterministic Strategies
Before examining the important strategies revealed in Figure 5.2 and 5.3, relations between the two criteria are considered. A good strategy should exploit others in order to receive a higher payoff, and its clone cluster should be strong enough to prevent invasion of others. These two criteria are equally important. Strategies that easily exploit others but fail to form strong clone clusters will likely spread throughout an environment, but when that environment is saturated with those strategies, they will become susceptible to invasion by other strategies. For example, S15 (always defects, regardless of the opponent’s move, also referred as ALLD) is a typical strategy of this kind. Furthermore, strategies with a strong ability to from clone clusters but that are easily exploited by others cannot survive, since their being exploited by others usually occurs before their clone clusters are formed; S0 (always cooperates, regardless of the opponent’s move, also referred as ALLC) is one of this kind of strategies.
The order of strategies in Figure 5.2 is almost perfectly opposite to that in Figure 5.3, showing that strategies that exploit others tend to have difficulty getting along with their clones; on the other hand, a strategy that gets along well with its clones has a higher chance of being exploited. In memory-1 strategies, strength in one criterion usually implies weakness in the other. Thus, mid-order strategies in both Figure 5.2 and 5.3 are considered important because they show a certain degree of strength in both criteria. These strategies have attracted the greatest attention from Prisoner’s Dilemma researchers [53]; their names are listed in Table 5.3.
Table 5.3: Some Commonly Discussed Memory-1 Deterministic Strategies
Strategy Name Representation Description
S3 Stubborn (C, C, D, D) Repeats the first round move regardless of
S3 Stubborn (C, C, D, D) Repeats the first round move regardless of