6. Set study variables and develop coding sheet accordingly
6.6 Digital game characteristics
In an intrinsic game, the content is an integral part of the game structure and extrinsic games are games where the content and the game structure are less tightly linked, or not linked at all. The researcher set the moderator as unspecified, intrinsic game, extrinsic games.
6.6.2 Tightly Linked vs. Loosely Linked
A tightly linked game is one that is constructed specifically around a fixed set of content. A loosely linked game is one in which the content is essentially separate from the game, but there are "hooks" in the game which bring the two together, and send the player from the game to the content, and back again. The researcher set the moderator as unspecified, tightly linked game, loosely linked game.
6.6.3 Hard-Wired vs. Engines and Templates or Shells
Everything in the game is designed and optimized around the game, the content, and the player experience is hard-wired game. In the template or shell, the content, be it text, graphics, video clips, or whatever, sits somewhere external to the game, is "read-in" or "called" by the program at the appropriate time, and is displayed onscreen. The researcher set the moderator as unspecified, hard-wired game, engines and templates or shells game.
6.6.4 Reflective vs. Action
Nonstop action games, offer the least opportunity for reflection in themselves, while role-playing, adventure, simulation, strategy, and puzzle games often
proceed at a slower pace and offer more built-in reflective "space". The researcher set the moderator as unspecified, reflective game, action game.
6.6.5 Synchronous vs. Asynchronous
Synchronous means real-time and asynchronous means turn-based games. In real time game, the game continues whether or not you are playing; stop playing long enough and you lose. In a turn-based game, the machine will wait forever for you to figure out your next move, unless you're "playing by the clock". The researcher set the moderator as unspecified, synchronous game, asynchronous game.
6.6.6 Single-Player vs. Multiplayer
Most DGBL to date has been single player, except the goal is to link people.
The researcher set the moderator as unspecified, Single-Player, Multiplayer.
6.6.7 Session-Based Games vs. Persistent-State Games
Session-based games exist only for as long as the initial players are playing. In persistent-state games, the world of the game never goes away. The researcher set the moderator as unspecified, session-based games, persistent-state games.
6.6.8 Video-Based vs. Animation-Based
Video is absolute realism. Animated characters and graphics, on the other hand, allow designers a great deal of freedom. The researcher set the moderator as unspecified, video-based game, animation-based game.
6.6.9 Narrative-Based vs. Reflex-Base
Narrative and characters can add emotional impact to a game, which may aid in the recall of certain content. Reflex is the ability to react very quickly to a stimulus and Language learning is one example. The researcher set the moderator as unspecified, narrative-based game, reflex-based game.
7. Analyze Qualified Articles and Perform Data Coding.
The researchers analyzed the selected articles based on the inclusion criteria and perform coding. In order to achieve inter-coder reliability, each article was co-examined by two coders. The calculation formula was identical+differentidentical × 100%.
The inter-coder reliability was 92%. About the disagreement items, the two coders consulted professional meta-analysis researcher to get advice and assistance until agreement was achieved.
8. Calculate the Effect Sizes for Each Article.
Because the effect size came from different articles and their samples and instrumentations vary, the researcher needed to use a standardized equation in order to achieve the same standard. This study conducted Hedges and Olkin (1985) equation and after calculating the effect sizes of each article, the researcher coded the data and performed statistical analysis.
The mean and standard deviation formula was the first option. However, if the collected data lacked of mean and standard deviation data but provided F or t and the sample size of experiment and control group instead, the researcher used the formula of t and samples or F and samples to calculate the ES. Then, the researcher corrected
the small sample bias.
After the ES was calculated, the researcher key in the value to calculate the variance.
After that, the researcher calculated the confidence interval (95% CI) of each ES.
After calculating each ES, each ES would be weighted by the inverse of its sampling variance. Thus, more weight would be given to findings that would be based on larger sample sizes. The weighted ES would then aggregate to form an overall weighted mean estimate of the treatment effect (d+). The sampling variance of 𝜎(𝑑2𝑖) was defined as the inverse of the sum of weights. The significance of the mean ES was judged by its 95% confidence interval (95% CI).
9. Proceeding Homogeneity of Variance Tests.
The homogeneity of effect size was calculated to examine whether all studies shared a common treatment effect. When all findings shared the same population ES, QT had an approximate χ2 distribution with k – 1 degrees of freedom, where k was the number of ES. If the obtained QT was larger than the critical value, the findings were determined to be significantly heterogeneous, meaning that there was more variability in the ESs than chance fluctuation would allow. (Hedges et al., 1985).
Next, each coded study was tested through two homogeneity statistics, between-class homogeneity (Qbetween) and within-class homogeneity (Qwithin). A QB
tests for
homogeneity of ES across classes. If QB is greater than the critical value, it indicates a significant difference among the classes of ES. When a moderator had more than two classes, post hoc comparisons will be performed to control type I error.10. Report the Findings and Write the Thesis.
A meta-analysis research should contain the finding of descriptive statistics and
inferential statistics. In descriptive statistics, information such as the publish date, ES, weighted average effect size, 95% CI should be shown. However, how big should the weighted average effect size be to be meaningful? According to Cohen (1988), an effect size of .2 is a small effect, an effect size of .5 is a medium effect, and an effect size of .8 represents what may be considered a large effect.
As for the inferential statistics part, the researcher presented the effect caused by each moderator, for example, the ES of each moderator and class, QB and QW, interpreted the finding and composed the finding as doctorial thesis according to the research purpose and research questions.
Chapter Four Results 1. Descriptive Analysis
Following the research procedure, the researcher got 96 articles that fitted the criteria of the study. In publication year, the publication year is from 1982 to 2015. Before 2000, there were only eight articles related to the DGBL topic, but after 2001, the number of DGBL articles roared as time passing by. During 2001 to 2005, eight articles studied on DGBL; during 2006-2010, 29 articles studied on DGBL; during 2011-2015, 51 articles studied on DGBL (see figure 4.1).
Note. The researcher made the figure.
Figure 4.1 Distribution figure of publication year.
In educational level, 38 articles studied on university sample, 29 articles studied on elementary sample, 14 articles studied on junior high, and 13 articles studied on senior high (see figure 4.2).
Note. Two unspecified articles are not included in the figure. The researcher made the figure.
Figure 4.2 Distribution figure of educational level.
In sample location, 34 articles studied on Asia, 33 articles studied on North America, 25 articles studied on Europe, two articles studied on South America, and one article studied on Oceania (see figure 4.3).
Note. One unspecified is not included in the figure. The researcher made the figure.
Figure 4.3 Distribution figure of sample location.
Educational Level
Sample Location
From the data above, the distribution of publication year showed that the number of DGBL articles increased rapidly as time passing by. The tendency showed that DGBL is a hot and an important topic. In educational levels, university had the most articles and the second was elementary. The nationality of the sample crossed Asia (Hong Kong, Iran, Korea, Malaysia, Taiwan, Thailand, Turkey), Europe (Belgium, Cyprus, Czech Republic, France, Germany, Greece, Italy, Netherlands, Norway, Scotland and Spain), South America (Chile), North America (American and Canada) and Oceania (Austria), showing that the sources were worldwide. The result showed that most of the DGBL studies were in Asia, Europe and North America.
2. Academic Achievement Analysis
2.1 Main effect size analysis on academic achievement
Seventy-six articles studied on the academic achievement. The researcher performed random model and the weighted mean ES (d+) = .69, 95% CI [.54, .84], standard deviation= .59. The range of the weighted ES was from -1.47 to 4.55. However, the homogeneity statistic (QT =138.17, p<.05) is significant, meaning that these articles are heterogeneous. The researcher excluded three standard deviations outliers (2.47 ~ -1.09). The excluded articles are Wang (2008), ES =4.55; Hummel et al. (2011), ES
=4.41; Squire et al. (2004), ES =3.06; Lu and Liu (2014), ES =2.97; Jackson et al.
(2011), ES = -1.48. Five articles are excluded and the details of 71 articles are showed in Table 4.1.
After deleting the outlier, the range of the weighted ES was from -.58 to 2.29. The researcher performed the homogeneity statistic again. The homogeneity statistic (QT =86.20, p >.05) is insignificant after excluding five articles, meaning that data in this research is homo, and the total weighted mean ES(d+) =.58, 95% CI [.45, .70].
The result indicated that the effect of applying DGBL on students’ academic achievement is significantly better than the effect of applying Non-DGBL on students’
academic achievement and achieved medium level.
Homogeneity statistic QT before excluding the outliers is significant, meaning that the included articles are heterogeneous; therefore, moderator analysis is necessary.
As for the publication bias examination, the Fail-safe N=2261.4 >Tolerance level (5×71+10=365), meaning that publication bias is unlikely to occur.
Table 4.1
Calculations of Academic Achievement N=9556
Year Author Total N Subject matter Educational level Sample location ES(d) SD
1982 Stockburger 54 Math University America 1.24 .09
Year Author Total N Subject matter Educational level Sample location ES(d) SD
Year Author Total N Subject matter Educational level Sample location ES(d) SD
2013 Hummel 19 Classroom management
dilemma University Netherlands -.08 .21
2013 Garneli 80 STEM Junior Greece -.42 .10
2014 Lan 38 English University Taiwan 1.38 .13
Note. The researcher compiled the table. The list only showed the first author’s last name.
2.2 Moderator analysis on academic achievement
The homogeneity statistic is significant on academic learning; therefore, it is necessary to examine the possible moderators. The researcher separated moderators into six categories: study characteristics, sample characteristics, research design characteristics, program characteristics, digital game categories, and digital game characteristics to perform the analysis.
2.2.1 Examination of study characteristics on academic achievement
The researcher separated study characteristics into three moderators: subject matter, publication Type and publication year and then examined possible moderators (see Table 4.2).
(1) Subject matter
The researcher separated subject matter into language, math, nature and science, social study, health and PE, computer science, and others. Research results showed that each subject matter weighted mean ES is positive and 95% CI is significantly different from zero, meaning that no matter in which subject matter,
the effect of applying DGBL on students’ academic achievement is significantly better than the effect of applying Non-DGBL on students’ academic achievement.
Because subject matter (QB = 88.60, p<.05) achieved significant level, it meant that the moderator affected academic achievement. The post hoc showed that the effects of language (ES=.84) and nature and science (ES=.81) are significantly higher than other kind of subject matters while math (ES=.27) is the smallest.
Except social study (Qw =1.49, p>.05), each subject matter Qw achieved significant level. The results showed that there are heterogeneous inside each group, meaning that unknown moderators still exist and need further analysis.
Therefore, the researcher will perform secondary moderator cross analysis.
(2) Publication type
The researcher separated publication type into journal and proceedings.
Research results showed that both weighted mean ES are positive and 95% CI are significantly different from zero, meaning that in both journal and proceedings, the effect of applying DGBL on students’ academic achievement is significantly better than the effect of applying Non-DGBL on students’
academic achievement.
Because publication type (QB = 1.21, p＞.05) is insignificant, it meant that different publication type will not affect the result. Both Journal and proceedings Qw are significant (p<.05), meaning that unknown moderators still
exit.
(3) Publication year
The researcher separated publication year into 1981-1985, 1986-1990, 1991-1995, 1996-2000, 2001-2005, 2006-2010, and 2011-2015. Due to the reason that the data before 2000 are few (k=7), these period will not be discussed. Research results showed that 2001-2005, 2006-2010, 2011-2015 weighted mean ES are positive and 95% CI are significantly different from zero, meaning that in these periods, the effect of applying DGBL on students’ academic achievement is significantly better than the effect of applying Non-DGBL on students’ academic achievement.
Because publication year (QB = 38.80, p<.05) achieved significant level, it meant that it is the moderator of academic achievement. Post hoc showed that 2011-2015 (ES=.57) and 2006-2010 (ES=.54) are significantly higher than 2001-2005 (ES=.33). Each publication year Qw is significant (p<.05), meaning that unknown moderators still exit.
Table 4.2
Note. *: significant at the .05 level; d+:
weighted mean effect size; k: number of articles; CI: confidence interval; #: d
+positive and 95% CI significantly different from zero ;(): post hoc.
2.2.2 Examination of sample characteristics on academic achievement
The researcher separated sample characteristics into two moderators:
educational level and sample location, and then examined possible moderators.
(1) Educational level
The researcher separated educational level into elementary, junior high, senior
high, and university. Research results showed that each weighted mean ES is positive and 95% CI is significantly different from zero, meaning that in elementary, junior high, senior high, and university, the effect of applying DGBL on students’ academic achievement is significantly better than the effect of applying Non-DGBL on students’ academic achievement.
Because educational level (QB = 46.59, p<.05) achieved significant level, it meant that educational level is the moderator of academic achievement. Post hoc showed that the effects on university (ES=.70), senior high (ES=.60) and junior high (ES=.58) are significantly better than elementary (ES=.37)
Each educational level Qw is significant (p<.05), meaning that unknown moderators still exit and further analysis is necessary. Therefore, the researcher will perform secondary cross analysis.
(2) Sample location
The researcher separated sample location into Asia, Europe, Africa, South America, North America, and Oceania. Due to the reason that the data of Africa, South America and Oceania are few, Africa, South America and Oceania will not be discussed. Research results showed that Asia, Europe and North America weighted mean ES are positive and 95% CI are significantly different from zero in sample location, meaning that in Asia, Europe and North America, the effect of applying DGBL on students’ academic achievement is significantly better than the effect of applying Non-DGBL on students’ academic achievement.
Because sample location (QB =81.27, p<.05) achieved significant level, it meant
that sample location is the moderator of academic achievement. Post hoc showed that the effects on Asia (ES=.77) and North America (ES=.70) are significantly higher than Europe (ES=.26).
Asia, Europe and North America Qw are significant (p<.05), meaning that unknown moderators still exit and further analysis is necessary. Therefore, the researcher will perform secondary cross analysis.
Note. *: significant at the .05 level; d+:
weighted mean effect size; k: number of articles; CI: confidence interval; #: d
+positive and 95% CI significantly different from zero; (): post hoc.
2.2.3 Examination of research design characteristics on academic achievement
The researcher separated research design characteristics into three moderators:instrumentation, instructor bias and experiment design, and then examined possible moderators.
(1) Instrumentation
The researcher separated instrumentation into self-compiled test and standardized test. Research results showed that both weighted mean ES are positive and 95% CI are significantly different from zero, meaning that in self-compiled test and standardized test, the effect of applying DGBL on students’ academic achievement is significantly better than the effect of applying Non-DGBL on students’ academic achievement.
Because instrumentation (QB = 13.05, p<.05) achieved significant level, it meant that instrumentation is the moderator of academic achievement, and that self-compiled test (ES=.56) is significantly better than standardized test (ES=.32). Each instrumentation Qw is significant (p<.05), meaning that unknown moderators still exit.
(2) Instructor bias
The researcher separated instructor bias into the same teacher (ST), different teacher (DT), mixture (MI), control group has no teacher involved and game as teacher in the experimental group (CNT>), control group has teacher involved and game as teacher in the experimental group (CT>), and unspecified. The MI data is none, so it will not be discussed. Research results showed that each weighted mean ES is positive and 95% CI is significantly different from zero, meaning that in ST, DT, CNT> and CT>, the effect of applying DGBL on students’ academic achievement is significantly better than the effect of applying Non-DGBL on students’ academic achievement.
Because instructor bias (QB = 84.03, p<.05) achieved significant level, it meant that instructor bias is the moderator of academic achievement. Post hoc
showed that CT and GT (ES=.73) and CNT and GT (ES=.67) are significantly higher than ST (ES= .48) and DT (ES=.27); ST is significantly higher than DT.
(3) Experiment design
The researcher separated experiment design into one-group pretest posttest design (OPP), pretest-posttest control group design (PPC), posttest-only control group design (POC), the nonequivalent pretest-posttest design (NPP), the nonequivalent posttest only design (NPO) and unspecified.
Research results showed that except POC, each weighted mean ES is positive and 95% CI is significantly different from zero, meaning that in OPP, PPC, NPP, NPO ,the effect of applying DGBL on students’ academic achievement is significantly better than the effect of applying Non-DGBL on students’
academic achievement.
Because experiment design (QB = 68.55, p<.05) achieved significant level, it meant that experiment design is the moderator of academic achievement. Post hoc showed that OPP (ES=1.03) is significantly higher than other kind of experiment designs; NPO (ES=.79) is significantly higher than PPC (ES=.47) and NPP (ES=.45).
Table 4.4
Note. *: significant at the .05 level; d+:
weighted mean effect size; k: number of articles; CI: confidence interval; #: d
+positive and 95% CI significantly different from zero; (): post hoc.
2.2.4 Examination of program characteristics on academic achievement
The researcher separated program characteristics into four moderators: duration of treatment, purpose of treatment, group size in experiment group and strategy involved, and then examined possible moderators.
(1) Duration of treatment
The researcher separated duration of treatment into less than 7 days, 8~30 days, above 31 days and unspecified. Research results showed that each weighted
mean ES is positive and 95% CI is significantly different from zero, meaning that in less than 7 days, 8~30 days, above 31 days and unspecified, the effect of applying DGBL on students’ academic achievement is significantly better than the effect of applying Non-DGBL on students’ academic achievement..
Because duration of treatment (QB = 793.34, p<.05) achieved significant level, it meant that duration of treatment is the moderator of academic achievement.
However, the conclusion is not confirmed because unspecified (ES=.81) is significantly higher than other duration of treatment while others do not have significant difference. Each duration of treatment Qw is significant (p<.05), meaning that unknown moderators still exit.
(2) Purpose of treatment
The researcher separated purpose of treatment into superior, complement, and replace. Research results showed that each weighted mean ES is positive and 95% CI is significantly different from zero, meaning that in superior, complement, and replace, the effect of applying DGBL on students’ academic achievement is significantly better than the effect of applying Non-DGBL on students’ academic achievement..
Because purpose of treatment (QB = 7.59, p<.05) achieved significant level, it meant that purpose of treatment is the moderator of academic achievement. Post hoc showed that superior (ES=.56) is significantly higher than replace (ES=.30).
Each purpose of treatment Qw is significant (p<.05), meaning that unknown moderators still exit.
(3) Group size in experiment group
The researcher separated group size in experiment group into individual and group. Research results showed that both weighted mean ES are positive and 95% CI are significantly different from zero, meaning that in both individual and group, the effect of applying DGBL on students’ academic achievement is significantly better than the effect of applying Non-DGBL on students’
academic achievement.
Because group size in experiment group (QB = 24.94, p<.05) achieved significant level, it meant that group size in experiment group is the moderator of academic achievement and that individual (ES=.57) is significantly higher than group (ES=.24). Each group size in experiment group Qw is significant (p<.05), meaning that unknown moderators still exit.
(4) Strategy involved
The researcher separated strategy involved into both does not have strategy (BNS), both have strategies (BHS), control group has strategy but experimental group doesn’t apply any strategy (CHEN), and experimental group has strategy but control group has no strategy (EHCN).
Research results showed that excluding BHS (k=0), only BNS weighted mean ES is positive and 95% CI is significantly different from zero, meaning that if both groups have no strategy involved, the effect of applying DGBL on students’ academic achievement is significantly better than the effect of applying Non-DGBL on students’ academic achievement. However, the conclusion is not confirmed because the article numbers are seriously
unbalanced (both CHEN and EHCN are below five articles). Each strategy involved Qw is significant (p<.05), meaning that unknown moderators still exit.
Table 4.5
Results of Program Characteristics Examination
Q
B k d+ 95%CIQ
wDuration of treatment 793.34* 76
1.Under 7 days 31 .41# [.34,.49] 335.06*
2.8~30 days 17 .48# [.39,.58] 262.75*
3.Above 31 days 12 .45# [.33,.56] 30.52*
9.Unspecified 16 .81# [.71,.90] 111.43*
Purpose of treatment 7.59* 76
(1> 3)
1. Superior 62 .56# [.51,.60] 624.94*
2. Complement 8 .51# [.37,.64] 145.11*
3. Replace 6 .30# [.06,.53] 15.70*
Group size in experiment
group 24.94* 76
1. Individual 66 .57# [.53,.62] 733.51*
2. Group 10 .24# [.09,.38] 34.89*
Strategy involved 73.60* 76
1. BNS 71 .53# [.48,.57] 705.99*
3. CHEN 2 2.36 [-.50, 5.21] 11.92*
4. EHCN 3 .2 [-.33,.72] 1.84
Note. *: significant at the .05 level; d+:
weighted mean effect size; k: number of articles; CI: confidence
interval; #: d
+positive and 95% CI significantly different from zero; (): post hoc.
2.2.5 Examination of digital game categories on academic achievement
The researcher separated digital game categories into action games, adventure games, fighting games, puzzle games, role-playing games, simulation games, sports games, strategy games, and unspecified. Due to the reason that the data of action games, adventure games, fighting games and sports games are few; these
The researcher separated digital game categories into action games, adventure games, fighting games, puzzle games, role-playing games, simulation games, sports games, strategy games, and unspecified. Due to the reason that the data of action games, adventure games, fighting games and sports games are few; these