Process validity measures the correspondence between a simulation model’s
running processes and the courses of action of the social issue being simulated.
Unfortunately, there is no simple method for evaluating the process validity of a
simulation model. In the absence of a direct method, researchers can use an indirect
method that addresses three process validity examination requirements:
It examines face validity. In other words, it addresses the issue of whether the
model is reasonable and acceptable in the minds of experts or scholars who are
familiar with the issue being studied. Face validity is the most commonly applied
means for examining a simulation model. A stricter standard entails an extended
version of the Turing test (Turing, 1950), which is based on the claim that if
individuals using independent judgment cannot tell the difference between the
operating processes or outcomes of a simulation model and real systems, the
simulation model can be said to exhibit process validity. Sensitivity analysis (to be
discussed in a later section) can also be used to verify face validity by systematically
changing the value of one important parameter and holding the others at a fixed value,
then examining whether or not the simulation model behavior matches the
expectations of an expert or modeler. Of course, such an examination requires that
modelers collect the required data inputs, behavioral modes, and outputs for certain
conditions. Although such a method requires prior knowledge, there is no need to
have a comprehensive understanding of all behavioral modes and outputs under all
conditions.
It directly examines simulation model hypotheses and uses those that are proven
to be correct to justify the model’s operating processes and degrees of acceptance.
Supposing that certain hypotheses cannot be justified after evaluation and analysis
(since certain simulation models are overly simplified and exaggerated representations
of real world scenarios), modelers must cautiously evaluate the degree to which
inaccurate hypotheses affect behavioral modes and outcomes. Decisions need to be
made as to whether inaccurate hypotheses should be revised or discarded. Again,
sensitivity analysis is very useful for this function. Modelers can use it to analyze
which elements, variables, parameters, rules, or relations affect behavioral modes and
outcomes. According to causal theory, knowing how many errors an invalid
hypothesis can cause helps to determine how simulation model process validity is
impaired.
It uses outcome validity (especially multi-level) tests to examine the process
validity of a simulation model. Horizontal, vertical, or other classifications (e.g.,
object-oriented, even-driven, or multi-agents) can be used to break down the running
process into several sub-processes, and each sub-process can be further divided into
more detailed sub-processes. Theoretically, a simulation model can be broken down to
the most basic level, with each sub-process corresponding to an independent and
indivisible component. Outcome validity tests can then be used to determine whether
or not sub-process outcomes are correct and fulfill expectations. Although such tests
are not completely equivalent to actual examinations of simulation model running
processes, they do increase operating process validity. However, tests that examine
individual components are very costly in terms of computation time and resources,
and require the collection of large real-world samples in order to fulfill test
requirements. Furthermore, even when test sample accuracy requirements are very
demanding, collected samples may contain extra information to the extent that it
cannot be used to evaluate simulation model sub-processes. In spite of these problems,
such examinations are considered the most effective way to evaluate the process
validity of a simulation model.
Internal Validity, Reliability Analysis, and Sensitivity Analysis
As mentioned above, social simulation projects require at least five steps: a)
proposing a theory or hypothesis to explain a social phenomenon, b) developing a
formal model of said theory or hypothesis, c) using a computer language or modeling
development tool to construct a simulation model or simulation system that
corresponds to the formal model, d) verifying and validating the simulation model,
and e) executing the simulation model and collecting and analyzing the generated data.
Note that the simulation model/system, theory/hypothesis and formal model are
considered equivalent and interchangeable—in other words, a simulation model or
system should faithfully and reliably represent the theory or hypothesis and formal
model. If not, the simulation model is meaningless. Experience dictates that the more
complex the theory or hypothesis, the greater the likelihood of making mistakes when
establishing a formal or simulation model, which leads to inconsistency among the
three elements. Therefore, it is necessary to examine the internal validity of a
simulation model before examining its outcome and process validity. However, this
raises an important issue: how to evaluate a simulation model that can fully represent
its corresponding theory/hypothesis and formal model.
The first step in evaluating internal validity is one that every modeler should use
during a simulation model’s development stage: determining if the basic structure of
the model (i.e., elements, variables, parameters, relations, rules, and components) are
equivalent to the theory, hypothesis, and formal model that it intends to represent. To
give an example, cognitive psychologists have traditionally depicted human cognitive
activity as an information processing system consisting of a long-term memory
component and a working memory component. Simulating human cognitive activity
requires the application of a data structure or another method (abstract or practical) to
represent the two memory components; otherwise, one cannot claim that the system
corresponds to cognitive theory. If a modeler claims that he or she has created an
information process system based on cognitive theory, how should we evaluate the
model to support or refute that claim?
There are two ways to solve this problem. First, a modeler can use face validity to
evaluate the simulation model or models and periodically consult with experts or
scholars familiar with such a theory to determine if the model actually captures the
spirit of the theory. Second, a modeler can use existing cases to examine simulation
models and to determine whether or not (under specific conditions) the simulation
model running process and outcomes fulfill the expectations of an expert or scholar.
Cases used to examine models must be valid, but they do not necessarily have to be
accurate or complete. Furthermore, they may represent a theory or hypothesis in a
very simplified or specialized format.
Applying existing cases to examine simulation models is similar to another
method of examining internal validity: reliability analysis, which focuses on whether
or not outcomes produced by repetitive simulation executions are consistent according
to simple statistical tests. In other words, for simulation models that involve stochastic
uncertainty, model designers must run simulation models many times (depending on
the statistical method employed) to examine outcome consistency. From the
perspective of reliability, the value of a simulation model with high stochastic
uncertainty should be measured according to whether or not there is a sufficient
number of executions to determine outcome consistency. If a modeler wants to
examine a simulation model that can produce output with probabilistic outcomes, it is
a much easier task: all the modeler needs to do is systematically compare the
outcomes of each execution until certain robust estimate values converge. In order to
prove that their model is stable and reliable, Jones, Radcliff, Taber, and Timpone
(1995) applied every possible set of initial parameters to their simulation model and
performed one million executions per parameter set. This example also serves as an
example of a clear examination method that is worth noting for future reference: no
matter how many stochastic components a simulation model owns, the model can be
executed repeatedly on a computer until the modelers make an evaluation. Modelers
need to take advantage of their computing tools to perform many executions, revising
or controlling certain components or procedures while holding the others at fixed
values in order to examine how revisions affect a model. In short, reliability analysis
gives social simulation modelers and researchers the ability and tools to understand
whether a vital stochastic component in a simulation model is interactive or additive.
A third means of examining the internal validity of a simulation model is
sensitivity analysis. Converting a theory, hypothesis, or formal model into a
simulation model entails deciding the domain of each parameter, the scope of initial
conditions, the sampling technique for stochastic elements, and probability
distributions. During sensitivity analysis, modelers systematically change parameter
values or other component settings (e.g., probability distribution) and examine how
simulation model performance or outcomes are affected. This type of analysis allows
modelers to distinguish between two kinds of parameters: when model performance or
outcomes are influenced by slight changes in a parameter value, this is referred to as a
sensitive parameter. Other simulation models are not influenced, regardless of how
much a parameter value is changed (within a reasonable range).
As long as the internal validity of a simulation model is assured, modelers can
use sensitivity analysis to examine a simulation model hypothesis. Simulation models
are usually viewed as simplifications of social issues for exploration purposes, in
which hypotheses are proposed and tested. Under such circumstances, sensitivity
analysis can be used to examine how different degrees of simplification affect the
model. If a simulation model’s constraints are loosened but its qualitative analysis
conclusion remains the same or stays consistent, then the hypothesis is plausible; if
not, the hypothesis requires special attention. During the last stage of simulation
model development, sensitivity analysis can serve as a warning mechanism or
guidance procedure. As mentioned earlier in this chapter, sensitivity analysis is also
suitable for examining outcome and process validity.
2.2. Beauty
Although some argue that the standards of a “beautiful” simulation model vary
from person to person, we should not ignore aesthetic criteria when evaluating a
simulation model. According to Lave and March (1993), three aesthetic characteristics
need to be acknowledged: simplicity, fertility, and surprise. These characteristics are
not only a matter of personal taste, but also ones that any researcher who uses
computational modeling and simulation to explore social issues should consider
seriously.
Simplicity
Computational modeling generally focuses on simplified understanding through
exploring systems designed to explain the real world. However, simulation models try
to filter the real world and address a pivotal theory by eliminating unnecessary details
according to the Occam’s razor recommendation that “Things should not be
multiplied without good reason” (Starfield, Smith, and Starfield 1990). Applied to
computational modeling and simulations, concise and succinct simulation models
should be accurate and valid.
In physics and other physical sciences, succinctness is regarded as a useful
exploration principle, but some biologists (including Crick, 1988) and social scientists
view an emphasis on succinctness as misleading. This leads to the question, “Is it
reasonable to apply Occam’s razor to social simulation models?” If a simulation
model is simplified too much, its process or outcomes may become so confusing that
they cannot be properly examined, thus making the associated theory or system
inaccurate or unhelpful in understanding real world phenomena. On the other hand,
some simulation models are too complex to be useful in understanding a system or
theory. The following principle can be inferred from this situation: if a researcher
cannot clearly track the operation or outcomes of a simulation model, the model will
create more problems than it solves. According to Occam’s razor, evaluating the
conciseness of a simulation model requires a determination of whether or not it
precisely expresses all of a theory’s important procedures, followed by a
determination of whether any procedures can be removed without affecting proper
model operation. In other words, simplicity has value in terms of both aesthetics and
practicality.
Fertility
Fertility refers to the implications that a simulation model conveys, how many
theories it covers, and how broadly it applies. When two models are used to test the
same theory, the model that generates a greater number of predictions is more highly
valued than the one that generates fewer. The simulation model that generates more
predictions is said to have a richer framework for theoretical deductions, thus making
it easier to investigate its predictions. Fertility is also related to simplicity in terms of
strictly controlling the number of assumptions during evaluation—that is, a complex
simulation model with more assumptions is only considered fertile if it generates
more predictions or implications.
Furthermore, a complex simulation model is said to generate more in the same
manner that merchandise quality increases with price. A very complex simulation
model that generates only a few predictions is generally viewed with suspicion; on the
other hand, after a simulation model is simplified it should be examined in terms of
whether or not it maintains a high level of prediction quality. A complex model that
produces more detailed predictions is preferable to a simple model that generates
predictions that are not as well defined. It is also important to consider how broadly a
model’s predictions or implications can be applied; a model that accounts for a larger
number of scenarios is more valuable than one that explains only a few. Again,
aesthetic standards and practicality merge within the fertility criterion.
Surprise
Another important aspect of fertility is surprise. An effective simulation model
often produces unexpected but conceptually applicable and easily examined
predictions. A prediction with strong implications may be surprising in situations
where a researcher does not expect an outcome or outcomes to be generated from a
simulation model, yet it produces data that fits well with facts or other evidence. In
other situations, an outcome may contradict a researcher’s intuition or appear to be
estranged from facts or other evidence, but the outcome turns out to be correct based
on a logical deduction or analysis. Precision and surprise often coincides in social
simulations—that is, when a theory can be correctly expressed in a simple simulation
model, precise and surprising predictive results are sometimes generated. Compared
to complex models, simple simulation models are less likely to produce surprising
predictions because their outcomes cannot be directly applied without further
transformation and explanation. Theories that require computational modeling and
simulation are usually more complex, and therefore have greater potential to produce
surprising conclusions that extend our knowledge.
2.3. Summary
A perfect simulation model should be valid and beautiful, but very few achieve
these ideals. Researchers should instead look for a balance between perfection and
real-life obstacles. Although many model designers are familiar with the examination
procedures and standards discussed in this chapter, they still face the challenge of
properly applying them to test different simulation models with applications, purposes,
and methods. In this chapter I have proposed several principles in terms of timing and
examination methods that other researchers may find useful when applying
computational modeling and simulation techniques to social science issues.
Truth can be divided into internal, outcome, and process validity. When
examining internal validity, sensitivity analysis can be used to examine simulation
models, especially more complex models. Outcome validity relies on traditional
quantitative concepts to test simulation models, but the question of how to quantify a
simulation model remains, as well as the question of judging results after
quantification. Based on the existing literature, I have described several examining
tools to help modelers test their simulation models, and suggested other tools (e.g.,
face validity, directly testing hypothesis validity, sensitivity analysis, and multi-level
tests borrowed from outcome validity) for inspecting the process validity of a
simulation model.
Simplicity has traditionally been the most important criterion for simulation
model beauty. Conciseness is still a desirable goal, but when the theories to be tested
by a simulation model are very complex, modelers need to achieve truth before
pursuing conciseness and guarantee that model operating procedures and outcomes
are both correct and precise. Fertility and surprise are two reasons why social
scientists adopt computational modeling and simulation, since they are more likely
than other formal models to produce surprising implications.
Chapter 3. Related Computational Models and Concepts
3.1. Dynamic Simulation
Dynamic simulation is one of the earliest computational modeling and simulation
methods in the social sciences (Huckfeldt et al. 1982). Famous research projects that
applied this method to explore social phenomena include urban systems of Forrester
(1969), the global population of Meadows et al. (1972), and electoral systems of
McPhee (1963). Dynamic simulation is the process of constructing a mathematical
model of some real-world system and analyzing its behaviors and results through
computer-based experiments. In part because of the availability of special simulation
software (e.g., STELLA or GPSS), dynamic simulation remains one of the most
popular and productive computational modeling and simulation methods.
As note, dynamic simulation refers to the construction of and experimentation
with a computational model of a dynamic system. For example, to save time and
money, a network engineer might propose a network model with a novel connection
topology, using computers to construct a virtual network environment model under
certain conditions and simulating their research results. Likewise, an epidemiologist
may examine the transmission dynamics and growth tendency of an infected
population through a simulation model. As a result, the network engineer saves time
and money; while the epidemiologist gains experimental control and the ability to
manipulate the theoretical world.
For more detailed introductions to dynamic simulation, we recommend Forrester
(1980). This book includes many modeling techniques and research examples, as well
as introducing DYNAMO and its applications. In addition, please refer to the
researches of Kheir (1988), Hoover and Perry (1990), Whicker and Sigelman (1991),
and Hannon and Ruth (1994). In Garson (1994), there are many introductions and
evaluations regarding the application of dynamic simulation.