Small-scale SO problems

Small-scale SO problems are problems whose potential solutions are known or pres-elected (see Section 2.2.1). Such problems can be solved with ranking and selection procedures. There are also other methods used in the literature to solve these prob-lems which will be briefly mentioned in this section. The focus in this study, however, is on ranking and selection.

2.4.1 Ranking and selection

R&S procedures are statistical methods developed to select the best system design or a subset that contains the best system design from a set of n competing alternatives (Goldsman & Nelson, 1994). Efficient R&S procedures also aim, in the process, to minimise the total number of simulation replications required while preserving a de-sired confidence level. Two important R&S procedures dominate the literature: the indifference-zone (IZ) methods and the optimal computing budget allocation (OCBA) methods. They are discussed in this section.

R&S procedures (or algorithms) find their origin in the 1950s within the statistics community. Bechhofer (1954) was the first to introduce the concepts of indifference-zone and probability of correct selection P(CS). His work aimed to improve on the then

(and possibly still) popular method of analysis of variances (ANOVA) “deficiencies”.

Following his contribution, R&S drew the attention of the simulation community due to its potential usefulness in stochastic simulation output analysis and many researchers have since built upon the foundations laid byBechhofer(1954). In particular,Dudewicz

& Dalal (1975) then Rinott (1978) further improved on Bechhofer’s work, proposing more efficient IZ methods. Rinott’s method (Rinott, 1978) particularly, which is dis-cussed later in this section, is one of the simplest and well-known R&S procedures and will be used in this study to illustrate the basic concept behind IZ methods (Kim &

Nelson,2007).

2.4.1.1 Indifference-Zone methods

The main idea behind IZ methods is to guarantee, with a probability of at least P^∗, that the system design ultimately selected is the best (Bechhofer,1954). Kim & Nelson (2007) provide a comprehensive survey on recent advances on the topic and they discuss, in detail, a number of IZ methods. InYoon & Bekker(2017), which is another survey, a procedure byChen & Lee(2009) is presented that attempts to use the IZ concept in the MOO context. The study, however, remains an empirical study and does not guarantee the probability of correct selection requirement P(CS) ≥ P^∗for the final Pareto optimal set (Yoon, 2018). This was achieved in Yoon (2018), where the researcher presents a new IZ multi-objective R&S procedure with P(CS) ≥ P^∗ guaranteed.

IZ methods make use of a parameter δ, which is set by the experimenter or the decision-maker to be the smallest actual difference that is important to detect. If the difference between the estimated means of any two system designs is within δ, then the difference between them is viewed as being, for practical purposes, insignificant;

meaning that the decision-maker is indifferent (hence the name indifference-zone) in selecting or ignoring these system designs depending on how they compare with other competing system designs outside the IZ. To illustrate how the IZ methods work, the following two-stage IZ method, namely Procedure R byRinott(1978) is repeated here (Algorithm2).

The constant h^∗ in Step 4 is the solution to the following double integral equation:

Z ∞

Algorithm 2 Procedure R

1: Select the probability requirement P^∗, the indifference-zone value δ^∗, and the first-stage sample size n₀ ≥ 2.

, where dxe denotes the smallest integer greater than or equal to x, and h^∗ is the solution to (2.7).

5: Run additional N_i− n₀ simulation replications for system i (i = 1, ..., k).

6: Compute the overall sample means ¯Xi(Ni)(i = 1, ..., k) and present system b as the best system, where b = arg min_iX¯_i(N_i).

where f denotes the probability density function (pdf) of the χ² distribution with n₀− 1 degrees of freedom.

Procedure R as well as other IZ methods use the least favourable configuration (LFC) assumption, which prevents them from taking advantage of the sample mean information (Yoon,2018), making them more conservative than they should be. Yoon (2018) developed a more efficient IZ method based on Procedure R, theMY procedure, which follows the Bayesian probabilistic approach, instead of the LFC assumption, for its probability of correct selection formulation. The procedure is presented in Algorithm 3.

2.4.1.2 Optimal Computing Budget Allocation methods

OCBA methods have been developed to address the efficiency issue related to the many simulation replications that are often utilised during R&S procedures. OCBA methods follow the Bayesian probabilistic theory. The main idea here is to maximise the probability of correct selection P(CS) by intelligently controlling the number of simulation replications based on the mean and variance information in the face of limited computing budget (Lee et al.,2010). OCBA has also been successfully adapted for multi-objective problems. Lee & Goldsman (2004), for example, incorporated the concept of Pareto optimality in OCBA and used the method to find non-dominated system designs.

Many OCBA methods exist in the literature for single and multi-objective problems.

The survey byLee et al.(2010) lists a number of them and points to further references

Algorithm 3 ProcedureMY

1: Select the probability requirement P^∗ = 1 − α, the indifference-zone value δ^∗, and the first-stage sample size n₀ ≥ 2. Let I = {1, 2, ..., M } be the set of systems in competition, and let β = _{M −1}^α .

2: Simulate n₀ replications for all M systems, and calculate sample means ¯X_i(n₀) and sample variances S_i²(n0). Let Ni= n0 (i = 1, ..., M ), and let b = arg miniX¯i(Ni).

and delete system b from I if

N_b ≥ than or equal to x, and h is the solution of:

Z ∞

4: If |I| = 0, then stop and present system b as the best system. Otherwise, go to Step 5.

5: Take one additional observation Xi,Ni+1from each system i ∈ I, and set Ni ← N_i+1 (∀_i ∈ I). Set I = {1, 2, ..., M } and update ¯X_i(N_i), S_i²(N_i) and b = arg min_iX¯_i(N_i), go to Step 3.

for more.

2.4.2 Other algorithms for small-scale SO

Besides R&S methods, there are other procedures available for solving small-scale SO problems. These are often referred to as multiple comparison procedures. In these procedures a number of simulation replications are performed on all the potential de-signs, and conclusions are made by constructing confidence intervals on the performance metric (Amaran et al.,2014). (See alsoTekin & Sabuncuoglu (2004) and Rosen et al.

(2008).)