Chapter 2. Related Work
2.2 The Prisoner’s Dilemma (PD)
2.2.3 Methods Used to Investigate PD
Currently the two predominant approaches to PD research are mathematical analysis [60, 62-64, 75, 81, 82] and evolutionary simulation [56, 65, 71, 74, 77, 78, 83-85]. Nowak [81] emphasizes the dynamic complexity and unpredictability of PD games despite their small number of strategies based on a simple set of rules. Besides, after using a spatial
concluded that limited computation resources produce simulation results that are heavily dependent upon the parameters involved. For these and additional reasons, PD games are usually analyzed under certain restrictions. For example, Nowak analyzed the dynamics of only three kinds of PD strategies [81]. In a separate project [82], he investigated a homogeneous population of strategies in an attempt to simulate strategy evolution.
Clear relationships among strategies (i.e., the specifics of player actions during a PD game) must be established prior to studying a PD problem. Furthermore, artificial society component categories underscore the importance of considering agent interactions.
Although the IPD model is used in this study of the public good/private interest conflict in artificial societies, an important distinction is the use of an agent viewpoint of interactions.
It is also important to note the important role played by simulations, regardless of the degree of understanding of model behavior or level of sophistication. Even in analytical studies, the majority of researchers have used simulations to collect comprehensive data on model behavior [76]. The framework described in this dissertation was used to analyze relationships between strategies—a crucial pre-simulation step that is often overlooked.
Most IPD studies make use of one of the payoff matrices shown in Table 2.4. The effect of payoff matrix choice on study results remains unclear. The model used in this study is based on the payoff matrix constraint shown in Table 2.3—in other words, it is
independent of the payoff matrix value.
Table 2.4: Some Commonly Used Payoff Matrices for the Prisoner’s Dilemma
Payoff Matrix Values [R, S, T, P]
Reference
[3, 0, 5, 1] [45, 46, 55, 65, 66, 68, 86]
[4, 0, 5, 1] [84]
[1, -2, 2, -1] [53]
[3, 1, 4, 2] [87]
Chapter 3.
The Importance of Model Analysis Prior to Artificial Society Simulation and Execution
Performing a model analysis prior to model simulation and execution is important regardless of the intended purpose of the artificial society in question. In this chapter I will offer reasons why it is important and present an illustrative example to show how it facilitates the use of artificial societies for problem solving.
Regarding the model execution/simulation process, common methodologies include the following steps: a) model creation (constructing a model based on an existing theory, hypothesis, or empirical data); b) model execution (running a model to produce data); and
c) model verification (assessing a model’s ability to operate as intended) and validation (analyzing data to ensure that a model is working as intended) [88].
Although it is rarely included in lists of model execution/simulation methodologies, theoretical and statistical model analyses play important roles. During the creation phase, model structures and relationships are mostly based on theories or hypotheses.
Theoretical variables are defined and quantified, and relationships among them are encoded [89]. In the words of Hanneman and Patrick, any model being constructed is
“one concrete realization of the prior theory” [10]. During the verification phase, the simulated results are statistically analyzed for purposes of interpretation and/or explanation [6].
There are few discussions in the literature of useful analytical tasks to be performed after a model is created but before simulation begins. At this point, it is important to determine appropriate model parameters or parameter sets based on empirical experience or existing data. The importance of analysis at this phase is the focus of this chapter.
We believe that pre-simulation model analysis can help reduce simulation complexity as well as assist in the identification of appropriate execution/simulation parameters.
3.1 Why Analyze before Model Simulation and Execution?
The most important motivation for a pre-execution/simulation model analysis is the assumption that the more one knows, the easier it will be to properly simulate or run the model. In this section I will describe how a pre-execution/simulation analysis helps in defining model scope, reducing model complexity, and choosing appropriate simulation/execution parameters.
3.1.1 Defining Model Scope
Even though defining a model’s scope is an important first step toward increasing model efficiency, it is surprising how often this step is overlooked by researchers.
Whenever an execution/simulation run provides significant findings, the data and the model clearly need to be inspected in terms of validity. But it is equally important to determine the conditions under which a particular model is successful, as well as the possibility of achieving success under other conditions.
Following model construction, concepts and entities are defined as parameters or variables. Prior to each new execution/simulation run, individual parameters must be set to specific values to satisfy some condition. An execution/simulation run is not equivalent
to a model. In this paper, a run is defined as an instance of the model. In theoretical terms, the comprehensive understanding of a model requires the execution/simulations of all possible model instances, but doing so is usually considered impractical.
A model M can be defined as
M=(P1, P2, … , Pn),
where P1, P2, … , Pn represent n parameters of M. Letting N denote the number of possible model instances and |Pi| denote the number of possible values of parameter Pi, then
N = |P1| • |P2| • … |Pn|
Each parameter has its own constraints. Examples of discrete parameters include the size of a population in a societal model and the number of nodes in a social network model [90]. Here the number of possible values is finite, but other parameters are considered continuous and infinite—for instance, tax rates in a simulation of tax and welfare systems.
Most artificial society models contain both discrete and continuous parameters; even in simple models, the number of instances is usually large or infinite. Each instance represents a tiny part of the model.
Since it is impossible to execute/simulate all model instances, it is important to choose an appropriate single model instance or set of model instances. I want to emphasize the importance of knowing the number of potential choices before making what appears to be the most appropriate choice, since the success of one model instance implies overall model success, but the failure of one model instance does not imply overall model failure. It is easier to figure out the relationship between a model and a model instance once its scope is defined.
3.1.2 Reducing Model Complexity
The second step toward successful execution/simulation involves reducing model complexity. Once the scope of a model is defined, it is no longer necessary to run all possible model instances. Unnecessary instances should be avoided in order to make the execution/simulation process more efficient. The two types of model instances that can be skipped are:
a) Unreasonable instances, meaning that a parameter setting does not match real
world conditions. These can be further divided into two categories: (1) instances with unreasonable parameter values, which are not under the constraints of the corresponding parameters; and (2) unreasonable parameter combinations, meaning that individual parameter values that are considered reasonable become unreasonable once they are
combined with other reasonable parameter values because of their correlational relationships.
b) Equivalent instances, meaning that instances may appear to be completely
different but nevertheless produce identical simulation results, or have identical meanings from the perspective of the model. An analysis of equivalent instances can provide information about whether or not an instance should be executed or simulated. It may be unnecessary to execute/simulate reasonable or important instances in cases where the results from equivalent instances are identified.
Analyses of unreasonable or equivalent instances reduce the number of potentially appropriate model instances. Using the metaphor of a highway map, the scope of a model provides the number of possible ways to get from point A to point B, while reduced model complexity provides answers to questions such as: “Which routes will not get us from point A to point B?” and “Which individual routes lead to the same destination?” By reducing the numbers of unreasonable and equivalent instances, it becomes easier to choose the appropriate parameter settings for successful execution/simulation.
3.1.3 Choosing Appropriate Model Instances
The final analytical step before execution/simulation is to determine appropriate model instances that resemble most other instances or that otherwise have some significant importance. In the literature, most model instance selections are based on empirical data or the testing of hypotheses. Nevertheless, an execution or simulation instance with significant results must still be tested to determine if it is representative of other instances and produce identical or similar outcomes. Answering such questions becomes more difficult when model instance determinations are not based on theoretical or statistical analyses.
3.2 Case Study
The example presented in this section is based on Azuaje’s [91] efforts to use a GA to evolve game strategies and cooperation. In my analytical approach the model is greatly simplified, which allows optimal solutions to be obtained more quickly and easily. Even though the robustness of artificial societies designed for problem solving (including evolutionary approaches) makes them popular among researchers in various disciplines, two important considerations are frequently overlooked: the importance of pre-run model
analysis, and the need to make a conscious decision between evolutionary and analytical approaches to individual problems based on their specific characteristics.
3.2.1 Model Description
In [91], Azuaje proposes an approach to the co-evolution of competing virtual creatures to model the emergence of cooperation in game strategy. His artificial life model considers two kinds of organisms, X and Y. Their individual decisions about whether or not to approach a food source are presented in the form of an IPD game. If X approaches a food source and Y doesn’t, X gets 5 points and Y 0, and vice versa. If neither organism approaches a food source, both X and Y get 3 points, and if both approach the source, they each get 1 point.
Azuaje stated that Y is represented by a genetic code that determines its sequence of moves against X. For example, if a Y organism is encoded as:
001000,
that means it will make defection moves in the first two rounds, followed by a single cooperation move, followed by three defection moves. Furthermore, Y cannot recognize individuals or store (remember) previous events. In contrast, X is a more sophisticated
organism that can perform basic cognitive and memory functions; it follows a Tit-for-Tat strategy of mimicking each move that its opponent made in the preceding round.
Azuaje used GA approach to evolve his game strategies. In his experiment, all Y organisms were randomly produced, and the length of each Y was 30. Each Y organism interacted with an X organism to obtain its score, after which GA operators (including reproduction, crossover, and mutation) were used with the population of Y organisms.
His results showed that it was possible for the evolved Y organisms to outperform the Tit-for-Tat strategy followed by the X organisms. After 50 generations, the most successful Y organism followed this code:
000000000000000000000000000001.
Its final point total was 92, versus 87 for the X organism.
According to this model, a) cooperative behavior can emerge from an evolutionary and unsupervised learning process, and b) evolving organisms are capable of achieving individual success by developing strategies that are more effective than Tit-for-Tat [91].
3.2.2 An Analytical Approach to the Model
I purpose an analytical approach that is more efficient than the evolutionary approach.
Y-type organisms evolve according to two basic genetic algorithm (GA) operators:
crossover and mutation. Even though there are two kinds of organisms, only Y is subject to the forces of artificial selection. X, whose primary function is to evaluate Y’s score, cannot evolve. It is unnecessary to select “100 fittest individuals from each type of organism to be included in the next generation” [91], that selection process can be limited to Y-type organisms.
Azuaje’s model can be formulated to the following problem: find an optimal binary string yopt that maximizes F(y), where a) the binary string y represents a Y-type organism;
b) 0’s encoded in y represent “do not approach food” and 1’s represent “approach food”;
and c) a fitness function F(y) denotes a y score when it plays with X (the Tit-for-Tat strategy) n times, with n equal to the length of y.
Let y=s1s2s3…sn. y’s moves from round 1 to round n are expressed as:
s1, s2, s3,…, sn-1, sn
Since X simply repeats its opponent’s preceding moves, its moves are expressed as:
0, s1, s2…, sn-2, sn-1
Moves of y and X from round 1 to round n are shown in Table 3.1.
Table 3.1: Moves of y and X from Round 1 to Round n.
1 2 … n
y s1 s2 … sn
X 0 s1 … sn-1
The interaction between y and X during a single round is represented as:
(aX, ay),
where aX stands for X’s move, and ay stands for y’s move. There are four possible combinations for (aX, ay). For each combination, y receives a score (in IPD terminology, a payoff) P(aX, ay). Possible scores are listed in Table 3.2.
In Table 3.2, [R, S, T, P] denotes four IPD payoff values. In [91]’s model, [R, S, T, P]
= [3, 0, 5, 1].
Table 3.2: Potential Scores of y for Each Combination of (aX, ay).
(aX, ay) (0, 0) (0, 1) (1, 0) (1, 1)
y’s score
P(aX, ay) R T S P
An interaction history of y and X moves from round 1 to round n is denoted as a sequence of (aX, ay) pairs:
(0, s1), (s1, s2), (s2, s3), (s3, s4), …, (sn-1, sn) The problem can be re-formulated as follows:
Find a binary string y=s1s2s3…sn, to maximize
) s , s
( 1 m
n
P m
∑
−m=1
,
where s0=0, P(0, 0) = 3, P(0, 1) = 5, P(1, 0) = 0, and P(0, 0) = 1
To solve this problem, define Bm as a pattern with m consecutive 1’s sandwiched
B2=0110,
…
Bn-1=011…10, with (n-1) 1’s
The longest pattern is represented as Bn-1. Since n is the length of y, after adding s0 at the beginning of y, the maximum number of consecutive 1’s must be (n-1) in order to satisfy the constraint that it starts and ends with 0.
Figure 3.1: String Combinations Can Be Represented as Bm Patterns Connected by Consecutive 0’s.
Bj
Bi Bk
… 0 0 1 … 1 0 0 … 0 0 1 … 1 0 0 … 0 0 1 … 1 0 0 …
Let the number of B1, B2, … , Bn-1 patterns in y be b1, b2, …. , bn-1. Most strings with a random combination of 0’s and 1’s can be represented as the b1, b2, …, and bn-1 of B1, B2, … and Bn-1 patterns connected by arbitrary numbers of 0’s (Figure 3.1). Strings that start or end with 1 cannot be represented in this form. These cases will be addressed later in this section.
The reason for choosing Bm patterns to represent binary strings is to compute y‘s score. In cases of consecutive 0’s, the y score will be the sum of consecutive R’s. At the
appearance of the first 1, the score of that round changes from R to T, since (aX, ay) is (0, 1).
The score of subsequent round will be S if that 1 is followed by a 0 and P if it is followed by another 1. Bm pattern scores for m=1 to (n-1) are listed in Table 3.3. They can be formulated as: PP(Bm)= (T+(m-1)P+S).
Table 3.3: Bm Scores for m=1 to (n-1).
y‘s score can be computed by adding the scores of consecutive 0’s and the summation
of Bm patterns’ scores from m=1 to (n-1). However, this kind of representation does not work when the string ends with 1, therefore two cases must be considered:
0{1, 0}*0
Case 2: Strings that end with x number of consecutive 1’s, whose regular expressions
are:
0{1, 0}*1+
Note that the cases that strings start with 1 are skipped since s0=0 in problem formulation of this section.
The y score: (s ,s )can be accumulated as follows: consecutive 0’s that connect them. Thus:
∑
Case 2: The string includes an additional section consisting of the last x consecutive
1’s, where x≥1. Thus:
)
Based on constraint of IPD: 2R>T+S and R>P, maximizing
∑
which is greater than 0 if 3
<5
x . Since x≥1, the optimal yopt is obtained when x=1:
with a score of
yopt = 000000000000000000000000000001
with a score of
This result is exactly the same as that produced by Azuaje’s evolutionary approach [91].
Figure 3.2: Mathematical Procedure for Obtaining Optimal y Score.
The mathematical procedure implemented to obtain optimal y score: is
shown in Figure 3.2.
)
3.2.3 Discussion of This Example
My approach emphasizes an important question: What should be done before using evolutionary approach to solve a problem? I believe that too many researchers overlook the importance of model analysis. The model in the current example is complex on the surface because it contains two kinds of organisms. However, since X cannot evolve, it only serves as an evaluation tool for Y—that is, the problem actually addresses only one kind of organism. When using an evolutionary approach, only Y organisms need to be selected for the next generation, cutting the number of artificial selection operations in half and drastically reducing computation time. My main point here is that pre-run analytical work can increase one’s understanding of problem scope, reduce model complexity, and help in the search for appropriate parameters—in short, increase the efficiency of a search for appropriate solutions.
My result was the same as [91]’s, but my analytical approach was faster and simpler.
In this model, my approach is appropriate in cases with different Y lengths, but the
evolutionary approach, since they clearly have complementary advantages and disadvantages. An analytical approach is much less effective than an evolutionary approach when problem spaces exceed a certain size or complexity threshold. This was not the case in [91]’s example.
Next, the model results are discussed from a game theorists’ perspective. In [91], Azuaje states that Y was able to achieve individual success “by learning to approach the source at the end of a contest.” This strategy was more successful than Tit-for-Tat. He also makes the claim that “information about the length of the games was not provided to the creature.” I believe game length was implied in Y’s encoding, since n is the length of Y and Y always plays with X n times. Thus, the important “shadow of the future”
assumption of stable cooperation no longer holds; in Axelrod’s words, “if you are unlikely to meet the other person again, or if you care little about future payoffs, then you might as well defect now and not worry about the consequences for the future” [46]. The behavior of an evolved solution for [91]’s model has already been discussed and verified in the literature [46].
Azuaje also wrote that “the emergence of cooperation did not require special assumptions about the individuals and the game environment.” I suggest that the emergence of cooperation is actually determined by the X organism, which uses the
Tit-for-Tat strategy in his model. The strategy encourages the evolution of Y toward a strategy that is equal to or better than Tit-for-Tat, which in turn encourages mutual cooperation. Assuming that X follows an “always defect” strategy (ALLD), then Y will also defect, and cooperative behavior will not emerge.
Furthermore, it is generally accepted that no evolutionarily stable strategy (ESS) exists for traditional IPD games, meaning that no prevalent strategy exists for extended IPD interactions [62]. Game theorists are less concerned with finding a dominant IPD strategy than with investigating relationships among strategies [92] and identifying conditions under which strategies become evolutionarily stable [61]. [91]’s model would be very interesting if X used more than one strategy or if X were also capable of evolving.
Either case would result in complex evolutionary dynamics, underscoring the weaknesses of the analytical approach and emphasizing the strengths of the evolutionary approach.
My analytical approach to the problem described in [91] is a faster and easier alternative to the evolutionary approach. I also emphasized two important considerations that are frequently overlooked by users of the evolutionary approach: the importance of pre-run model analysis, and the need to make a conscious decision between an evolutionary or analytical approach to solving a problem. Some game theory
also addressed.
My approach can be expanded to solve more sophisticated problems. For example, in cases where X-type organisms use other kinds of strategies, y’s score function may change; at a certain level of sophistication for X and y encoding, interactions between them may become too complex to be represented as a string. In such cases, a finite state machine representation may be useful for representing interactions between the two strategies [92].
Furthermore, my approach can be applied to other form of two-person matrix games (e.g., chicken games), and Prisoner’s Dilemma derivatives (e.g., N-person or N-choice Prisoner’s Dilemma). It is also important to investigate the extent to which the analytical approach is useful for non-deterministic strategies (strategies with slight chances of deviations in moves). For those sophisticated models, we believe a combination of analytical and evolutionary approaches may be more efficient than relying on either one
Furthermore, my approach can be applied to other form of two-person matrix games (e.g., chicken games), and Prisoner’s Dilemma derivatives (e.g., N-person or N-choice Prisoner’s Dilemma). It is also important to investigate the extent to which the analytical approach is useful for non-deterministic strategies (strategies with slight chances of deviations in moves). For those sophisticated models, we believe a combination of analytical and evolutionary approaches may be more efficient than relying on either one