2. Moderator analysis
2.2 Sample characteristics
Kulik (1981) argued that individualized instruction has different effects on different people. Specifically, he argued that it has different effects on elementary school, secondary school, and college-level learners. Therefore, in his review, he separated students into three groups: k-8, 9-12 and college.
Kulik (1981) suggested that at the lower levels of instruction, learners need the stimulation and guidance provided by a highly reactive teaching medium. At the upper levels of instruction, a highly reactive instructional medium may not only be unnecessary but may even get in the way.
Most of meta-analysis set education stage as moderator (e.g. Kulik, 1981;
Kulik et al., 1983; Lee, 1999; Ke, 2009). Dekkers and Donatti (1981) set age instead of educational level as moderator. Wouters et al. (2013) thought the question can be raised whether age is a moderator. The meta-analysis by Vogel et al. (2006), however, did not find differences between age groups in learning with serious games. Regard the finding of Vogel et al. (2006), they still listed it as a moderator because of time passing by and more articles are found. We should pay attention that Wouters et al. (2013) used age as a moderator, but actually it’s a mixture: the age separates into children, preparatory education students and adults. Vogel et al. (2006) listed out the age range and the suitable educational level together as: preschool (less than five years of age),
elementary (grades K-5, ages 6-11), middle (grades 6-8, ages 11-14), high (grades 9-12, ages 14-18), college (undergraduate study, ages 18-24), adult (25 years of age and older), unknown/unspecified.
Since studies collected for the present meta-analysis come from different countries, it is possibly not appropriate to set age as a moderator because each country has its own educational system for qualified age to study. The researcher divided the educational level into five levels: elementary school, junior high school, senior high school and university, and unspecified.
(2) Sample location
With geopolitical contexts, historical changes can have powerful influences on social programs that moderate or mediate their effects. Interventions that are
“exported” from one place to another do not always “travel” well because cultural, social, organizational, and political contexts can greatly shape the implementation and effects of interventions. However, in the meta-analysis the researcher has found, none of any meta-analysis set the sample location as a moderator. Because this research is an international research, the researcher thought it is worth setting sample location as a new moderator because each country has its own culture and context; moreover, it is worth paying attention if there is “digital divide”. Therefore, the researcher divided the sample location into continents: Africa, Asia, Europe, South America, North America and Oceania.
4.2.3 Research design characteristics
(1) Instrumentation
Lee (1999) listed two variables to report the outcome measures of the studies:
effect size by academic achievement and effect size by attitude toward simulation. However, Lee did not tell what kind of instrumentation the data he collects contained. Observing the meta-analysis the researcher has found, none listed out what kind of instrumentation the data they collected contained. Is the assessment test used really measure the learning outcome that is aimed at?
Dede, Clarke, Ketelhut, Nelson and Bowman (2005) demonstrated a better command of cognitive skills for a game group when measured with an evaluation letter to the mayor, but not with traditional test items. Most serious games were situated in specific contexts that may yield learning outcomes that were contextualized as well. Assessment methods that took the context of learning into account (e.g., an evaluation letter to the mayor) may reveal differences in performance that would be undisclosed with traditional assessment methods.
An additional argument for reconsidering the traditional assessment methods followed from the results of Belanich, Sibley and Orvis (2004) who found that items with visual information were better recalled than written information.
Video games are highly visual and may favor the acquisition of visually encoded knowledge. In that case visually-oriented assessment may reveal learning of knowledge that would probably not have been found with a text-based assessment method. So, the type of instrumentation may affect the effect of learning outcome according to Dede et al. (2005) and Belanich et al.
(2004). Therefore, it is worth listing it as a new moderator. In accordance with the reason above, instrumentation to measure students’ learning effect includes standardized test, self-compiled test, mixed, and unspecified.
(2) Instructor bias
The role teacher plays may affect the learning outcome. Sitzmann (2011) argued that trainees in the comparison group were often taught via a different instructional method as a substitute for utilizing the simulation game. However, studies differed in terms of whether the comparison group learned by means of active (e.g., Hughes, 2001; Mitchell & Savill-Smith, 2004; Willis, 1989) or passive (e.g., Bayrak, 2008; Frear & Hirschbuhl, 1999; Shute & Glaser, 1990) instructional methods. Therefore, it’s not only what experiment group does is important variables, what comparison group receives is equally important. In the comparison group, a teacher may play a dominant role, while in the experiment group; the role teacher play varies according to the usage of the game. The game implementation is as supplement or as substitute will affect the role teacher plays. Kulik et al. (1983) thought that computer applications can be typed as five categories: drill and practice, tutoring, computer-managed teaching, simulation, and programming the computer to solve problems. In studies in which the computer served as a substitute, it replaced teacher presentations, readings, assignments, or some combination of these. In studies where the computer served as a supplement, it did not replace regular course elements, but instead served as an additional resource for students. Because the function of game may serve different purpose and may conduct different result, it is necessary to list the role teacher play as a moderator. Therefore, Kulik et al.
(1983) concerned the instructor bias and listed it as a moderator. The instructor bias can be separated in to two parts: different instructor and same instructor. In this study, the researcher compared DGBL vs. non-DGBL, which means the comparison group will contain no digital game inside. However, the role teacher played still is important because it will help us know the portion of
teacher involved in the study. So, instructor bias moderator can be separated into six parts: the same teacher, different teacher, mixture, comparison group has no teacher involved and game serves as teacher in the experimental group, comparison group has teacher involved and game serves as teacher in the experimental group, unspecified.
(3) Experiment design
Sitzmann (2011) stated that randomly assigning trainees to experimental conditions and utilizing a rigorous study design would allow researchers to rule out alternative explanations for differences in learning between simulation game and comparison groups. However, in the real world, if the standard is too strict, the data you get may be small. Wouters et al. (2013) listed randomization and experimental design as their moderators. They took randomization into account whether a pure or a quasi-experimental design was used. In the latter case, participants were not randomized between the conditions, which may allow alternative explanations for the results that are found. And about the experimental design, they considered whether a posttest-only design or a pretest-posttest design was used. Kulik et al. (1980) and Kulik et al. (1983) not only cared about the random assignment of experiment group, but listed random assignment of comparison group as moderator. Wouters et al. (2013) set the experiment design as posttest only and pre-posttest. In the meantime, they also examined randomization. In their finding, they found that the beneficial effect of serious games is contingent on the experimental rigor:
random assignment decreased the effect of serious games. In fact, in studies with randomization, serious games are not more effective than conventional instruction methods. And, the experimental design of the study does not have
an impact on the magnitude of the effect size.
In this research, the researcher decided to use a broad view in order to get a broad data. Therefore, the research design contained: one-group pretest posttest design; pretest-posttest control group design, posttest-only control group design; the nonequivalent pretest-posttest designs and nonequivalent posttest only.