A meta-analysis research should contain the finding of descriptive statistics and
inferential statistics. In descriptive statistics, information such as the publish date, ES, weighted average effect size, 95% CI should be shown. However, how big should the weighted average effect size be to be meaningful? According to Cohen (1988), an effect size of .2 is a small effect, an effect size of .5 is a medium effect, and an effect size of .8 represents what may be considered a large effect.
As for the inferential statistics part, the researcher presented the effect caused by each moderator, for example, the ES of each moderator and class, QB and QW, interpreted the finding and composed the finding as doctorial thesis according to the research purpose and research questions.
Chapter Four Results 1. Descriptive Analysis
Following the research procedure, the researcher got 96 articles that fitted the criteria of the study. In publication year, the publication year is from 1982 to 2015. Before 2000, there were only eight articles related to the DGBL topic, but after 2001, the number of DGBL articles roared as time passing by. During 2001 to 2005, eight articles studied on DGBL; during 2006-2010, 29 articles studied on DGBL; during 2011-2015, 51 articles studied on DGBL (see figure 4.1).
Note. The researcher made the figure.
Figure 4.1 Distribution figure of publication year.
In educational level, 38 articles studied on university sample, 29 articles studied on elementary sample, 14 articles studied on junior high, and 13 articles studied on senior high (see figure 4.2).
Note. Two unspecified articles are not included in the figure. The researcher made the figure.
Figure 4.2 Distribution figure of educational level.
In sample location, 34 articles studied on Asia, 33 articles studied on North America, 25 articles studied on Europe, two articles studied on South America, and one article studied on Oceania (see figure 4.3).
Note. One unspecified is not included in the figure. The researcher made the figure.
Figure 4.3 Distribution figure of sample location.
Educational Level
Sample Location
From the data above, the distribution of publication year showed that the number of DGBL articles increased rapidly as time passing by. The tendency showed that DGBL is a hot and an important topic. In educational levels, university had the most articles and the second was elementary. The nationality of the sample crossed Asia (Hong Kong, Iran, Korea, Malaysia, Taiwan, Thailand, Turkey), Europe (Belgium, Cyprus, Czech Republic, France, Germany, Greece, Italy, Netherlands, Norway, Scotland and Spain), South America (Chile), North America (American and Canada) and Oceania (Austria), showing that the sources were worldwide. The result showed that most of the DGBL studies were in Asia, Europe and North America.
2. Academic Achievement Analysis
2.1 Main effect size analysis on academic achievement
Seventy-six articles studied on the academic achievement. The researcher performed random model and the weighted mean ES (d+) = .69, 95% CI [.54, .84], standard deviation= .59. The range of the weighted ES was from -1.47 to 4.55. However, the homogeneity statistic (QT =138.17, p<.05) is significant, meaning that these articles are heterogeneous. The researcher excluded three standard deviations outliers (2.47 ~ -1.09). The excluded articles are Wang (2008), ES =4.55; Hummel et al. (2011), ES
=4.41; Squire et al. (2004), ES =3.06; Lu and Liu (2014), ES =2.97; Jackson et al.
(2011), ES = -1.48. Five articles are excluded and the details of 71 articles are showed in Table 4.1.
After deleting the outlier, the range of the weighted ES was from -.58 to 2.29. The researcher performed the homogeneity statistic again. The homogeneity statistic (QT =86.20, p >.05) is insignificant after excluding five articles, meaning that data in this research is homo, and the total weighted mean ES(d+) =.58, 95% CI [.45, .70].
The result indicated that the effect of applying DGBL on students’ academic achievement is significantly better than the effect of applying Non-DGBL on students’
academic achievement and achieved medium level.
Homogeneity statistic QT before excluding the outliers is significant, meaning that the included articles are heterogeneous; therefore, moderator analysis is necessary.
As for the publication bias examination, the Fail-safe N=2261.4 >Tolerance level (5×71+10=365), meaning that publication bias is unlikely to occur.
Table 4.1
Calculations of Academic Achievement N=9556
Year Author Total N Subject matter Educational level Sample location ES(d) SD
1982 Stockburger 54 Math University America 1.24 .09
Year Author Total N Subject matter Educational level Sample location ES(d) SD
Year Author Total N Subject matter Educational level Sample location ES(d) SD
2013 Hummel 19 Classroom management
dilemma University Netherlands -.08 .21
2013 Garneli 80 STEM Junior Greece -.42 .10
2014 Lan 38 English University Taiwan 1.38 .13
Note. The researcher compiled the table. The list only showed the first author’s last name.
2.2 Moderator analysis on academic achievement
The homogeneity statistic is significant on academic learning; therefore, it is necessary to examine the possible moderators. The researcher separated moderators into six categories: study characteristics, sample characteristics, research design characteristics, program characteristics, digital game categories, and digital game characteristics to perform the analysis.
2.2.1 Examination of study characteristics on academic achievement
The researcher separated study characteristics into three moderators: subject matter, publication Type and publication year and then examined possible moderators (see Table 4.2).
(1) Subject matter
The researcher separated subject matter into language, math, nature and science, social study, health and PE, computer science, and others. Research results showed that each subject matter weighted mean ES is positive and 95% CI is significantly different from zero, meaning that no matter in which subject matter,
the effect of applying DGBL on students’ academic achievement is significantly better than the effect of applying Non-DGBL on students’ academic achievement.
Because subject matter (QB = 88.60, p<.05) achieved significant level, it meant that the moderator affected academic achievement. The post hoc showed that the effects of language (ES=.84) and nature and science (ES=.81) are significantly higher than other kind of subject matters while math (ES=.27) is the smallest.
Except social study (Qw =1.49, p>.05), each subject matter Qw achieved significant level. The results showed that there are heterogeneous inside each group, meaning that unknown moderators still exist and need further analysis.
Therefore, the researcher will perform secondary moderator cross analysis.
(2) Publication type
The researcher separated publication type into journal and proceedings.
Research results showed that both weighted mean ES are positive and 95% CI are significantly different from zero, meaning that in both journal and proceedings, the effect of applying DGBL on students’ academic achievement is significantly better than the effect of applying Non-DGBL on students’
academic achievement.
Because publication type (QB = 1.21, p＞.05) is insignificant, it meant that different publication type will not affect the result. Both Journal and proceedings Qw are significant (p<.05), meaning that unknown moderators still
exit.
(3) Publication year
The researcher separated publication year into 1981-1985, 1986-1990, 1991-1995, 1996-2000, 2001-2005, 2006-2010, and 2011-2015. Due to the reason that the data before 2000 are few (k=7), these period will not be discussed. Research results showed that 2001-2005, 2006-2010, 2011-2015 weighted mean ES are positive and 95% CI are significantly different from zero, meaning that in these periods, the effect of applying DGBL on students’ academic achievement is significantly better than the effect of applying Non-DGBL on students’ academic achievement.
Because publication year (QB = 38.80, p<.05) achieved significant level, it meant that it is the moderator of academic achievement. Post hoc showed that 2011-2015 (ES=.57) and 2006-2010 (ES=.54) are significantly higher than 2001-2005 (ES=.33). Each publication year Qw is significant (p<.05), meaning that unknown moderators still exit.
Table 4.2
Note. *: significant at the .05 level; d+:
weighted mean effect size; k: number of articles; CI: confidence interval; #: d
+positive and 95% CI significantly different from zero ;(): post hoc.
2.2.2 Examination of sample characteristics on academic achievement
The researcher separated sample characteristics into two moderators:
educational level and sample location, and then examined possible moderators.
(1) Educational level
The researcher separated educational level into elementary, junior high, senior
high, and university. Research results showed that each weighted mean ES is positive and 95% CI is significantly different from zero, meaning that in elementary, junior high, senior high, and university, the effect of applying DGBL on students’ academic achievement is significantly better than the effect of applying Non-DGBL on students’ academic achievement.
Because educational level (QB = 46.59, p<.05) achieved significant level, it meant that educational level is the moderator of academic achievement. Post hoc showed that the effects on university (ES=.70), senior high (ES=.60) and junior high (ES=.58) are significantly better than elementary (ES=.37)
Each educational level Qw is significant (p<.05), meaning that unknown moderators still exit and further analysis is necessary. Therefore, the researcher will perform secondary cross analysis.
(2) Sample location
The researcher separated sample location into Asia, Europe, Africa, South America, North America, and Oceania. Due to the reason that the data of Africa, South America and Oceania are few, Africa, South America and Oceania will not be discussed. Research results showed that Asia, Europe and North America weighted mean ES are positive and 95% CI are significantly different from zero in sample location, meaning that in Asia, Europe and North America, the effect of applying DGBL on students’ academic achievement is significantly better than the effect of applying Non-DGBL on students’ academic achievement.
Because sample location (QB =81.27, p<.05) achieved significant level, it meant
that sample location is the moderator of academic achievement. Post hoc showed that the effects on Asia (ES=.77) and North America (ES=.70) are significantly higher than Europe (ES=.26).
Asia, Europe and North America Qw are significant (p<.05), meaning that unknown moderators still exit and further analysis is necessary. Therefore, the researcher will perform secondary cross analysis.
Note. *: significant at the .05 level; d+:
weighted mean effect size; k: number of articles; CI: confidence interval; #: d
+positive and 95% CI significantly different from zero; (): post hoc.
2.2.3 Examination of research design characteristics on academic achievement
The researcher separated research design characteristics into three moderators:instrumentation, instructor bias and experiment design, and then examined possible moderators.
(1) Instrumentation
The researcher separated instrumentation into self-compiled test and standardized test. Research results showed that both weighted mean ES are positive and 95% CI are significantly different from zero, meaning that in self-compiled test and standardized test, the effect of applying DGBL on students’ academic achievement is significantly better than the effect of applying Non-DGBL on students’ academic achievement.
Because instrumentation (QB = 13.05, p<.05) achieved significant level, it meant that instrumentation is the moderator of academic achievement, and that self-compiled test (ES=.56) is significantly better than standardized test (ES=.32). Each instrumentation Qw is significant (p<.05), meaning that unknown moderators still exit.
(2) Instructor bias
The researcher separated instructor bias into the same teacher (ST), different teacher (DT), mixture (MI), control group has no teacher involved and game as teacher in the experimental group (CNT>), control group has teacher involved and game as teacher in the experimental group (CT>), and unspecified. The MI data is none, so it will not be discussed. Research results showed that each weighted mean ES is positive and 95% CI is significantly different from zero, meaning that in ST, DT, CNT> and CT>, the effect of applying DGBL on students’ academic achievement is significantly better than the effect of applying Non-DGBL on students’ academic achievement.
Because instructor bias (QB = 84.03, p<.05) achieved significant level, it meant that instructor bias is the moderator of academic achievement. Post hoc
showed that CT and GT (ES=.73) and CNT and GT (ES=.67) are significantly higher than ST (ES= .48) and DT (ES=.27); ST is significantly higher than DT.
(3) Experiment design
The researcher separated experiment design into one-group pretest posttest design (OPP), pretest-posttest control group design (PPC), posttest-only control group design (POC), the nonequivalent pretest-posttest design (NPP), the nonequivalent posttest only design (NPO) and unspecified.
Research results showed that except POC, each weighted mean ES is positive and 95% CI is significantly different from zero, meaning that in OPP, PPC, NPP, NPO ,the effect of applying DGBL on students’ academic achievement is significantly better than the effect of applying Non-DGBL on students’
academic achievement.
Because experiment design (QB = 68.55, p<.05) achieved significant level, it meant that experiment design is the moderator of academic achievement. Post hoc showed that OPP (ES=1.03) is significantly higher than other kind of experiment designs; NPO (ES=.79) is significantly higher than PPC (ES=.47) and NPP (ES=.45).
Table 4.4
Note. *: significant at the .05 level; d+:
weighted mean effect size; k: number of articles; CI: confidence interval; #: d
+positive and 95% CI significantly different from zero; (): post hoc.
2.2.4 Examination of program characteristics on academic achievement
The researcher separated program characteristics into four moderators: duration of treatment, purpose of treatment, group size in experiment group and strategy involved, and then examined possible moderators.
(1) Duration of treatment
The researcher separated duration of treatment into less than 7 days, 8~30 days, above 31 days and unspecified. Research results showed that each weighted
mean ES is positive and 95% CI is significantly different from zero, meaning that in less than 7 days, 8~30 days, above 31 days and unspecified, the effect of applying DGBL on students’ academic achievement is significantly better than the effect of applying Non-DGBL on students’ academic achievement..
Because duration of treatment (QB = 793.34, p<.05) achieved significant level, it meant that duration of treatment is the moderator of academic achievement.
However, the conclusion is not confirmed because unspecified (ES=.81) is significantly higher than other duration of treatment while others do not have significant difference. Each duration of treatment Qw is significant (p<.05), meaning that unknown moderators still exit.
(2) Purpose of treatment
The researcher separated purpose of treatment into superior, complement, and replace. Research results showed that each weighted mean ES is positive and 95% CI is significantly different from zero, meaning that in superior, complement, and replace, the effect of applying DGBL on students’ academic achievement is significantly better than the effect of applying Non-DGBL on students’ academic achievement..
Because purpose of treatment (QB = 7.59, p<.05) achieved significant level, it meant that purpose of treatment is the moderator of academic achievement. Post hoc showed that superior (ES=.56) is significantly higher than replace (ES=.30).
Each purpose of treatment Qw is significant (p<.05), meaning that unknown moderators still exit.
(3) Group size in experiment group
The researcher separated group size in experiment group into individual and group. Research results showed that both weighted mean ES are positive and 95% CI are significantly different from zero, meaning that in both individual and group, the effect of applying DGBL on students’ academic achievement is significantly better than the effect of applying Non-DGBL on students’
academic achievement.
Because group size in experiment group (QB = 24.94, p<.05) achieved significant level, it meant that group size in experiment group is the moderator of academic achievement and that individual (ES=.57) is significantly higher than group (ES=.24). Each group size in experiment group Qw is significant (p<.05), meaning that unknown moderators still exit.
(4) Strategy involved
The researcher separated strategy involved into both does not have strategy (BNS), both have strategies (BHS), control group has strategy but experimental group doesn’t apply any strategy (CHEN), and experimental group has strategy but control group has no strategy (EHCN).
Research results showed that excluding BHS (k=0), only BNS weighted mean ES is positive and 95% CI is significantly different from zero, meaning that if both groups have no strategy involved, the effect of applying DGBL on students’ academic achievement is significantly better than the effect of applying Non-DGBL on students’ academic achievement. However, the conclusion is not confirmed because the article numbers are seriously
unbalanced (both CHEN and EHCN are below five articles). Each strategy involved Qw is significant (p<.05), meaning that unknown moderators still exit.
Table 4.5
Results of Program Characteristics Examination
Q
B k d+ 95%CIQ
wDuration of treatment 793.34* 76
1.Under 7 days 31 .41# [.34,.49] 335.06*
2.8~30 days 17 .48# [.39,.58] 262.75*
3.Above 31 days 12 .45# [.33,.56] 30.52*
9.Unspecified 16 .81# [.71,.90] 111.43*
Purpose of treatment 7.59* 76
(1> 3)
1. Superior 62 .56# [.51,.60] 624.94*
2. Complement 8 .51# [.37,.64] 145.11*
3. Replace 6 .30# [.06,.53] 15.70*
Group size in experiment
group 24.94* 76
1. Individual 66 .57# [.53,.62] 733.51*
2. Group 10 .24# [.09,.38] 34.89*
Strategy involved 73.60* 76
1. BNS 71 .53# [.48,.57] 705.99*
3. CHEN 2 2.36 [-.50, 5.21] 11.92*
4. EHCN 3 .2 [-.33,.72] 1.84
Note. *: significant at the .05 level; d+:
weighted mean effect size; k: number of articles; CI: confidence
interval; #: d
+positive and 95% CI significantly different from zero; (): post hoc.
2.2.5 Examination of digital game categories on academic achievement
The researcher separated digital game categories into action games, adventure games, fighting games, puzzle games, role-playing games, simulation games, sports games, strategy games, and unspecified. Due to the reason that the data of action games, adventure games, fighting games and sports games are few; these game categories will not be discussed.
Research results showed that each digital game categories weighted mean ES is positive and 95% CI is significantly different from zero, meaning that in puzzle games, role-playing games, simulation games, strategy games,, the effect of applying DGBL on students’ academic achievement is significantly better than the effect of applying Non-DGBL on students’ academic achievement.
Because digital game categories (QB = 79.04, p<.05) achieved significant level, it meant that digital game categories is the moderator of academic achievement.
Post hoc showed that the effects of role-playing (ES=.98) is significantly higher than simulation games (ES=.74), strategy games (ES=.46) and puzzle games (ES=.36); simulation games is significantly higher than puzzle games and strategy games.
Each digital game categories Qw is significant (p<.05), meaning that unknown moderators still exit and further analysis is necessary. Therefore, the researcher will perform secondary cross analysis.
Table 4.6
Results of Digital Game Categories Examination
Q
B k d+ 95%CIQ
wDigital game categories 79.04* 71
(5>4,6,8; 6>4,8)4. Puzzle games 34 .36# [.30, .42] 298.93*
5. Role-playing games. 6 .98# [.71, 1.24] 86.97*
6. Simulation games 21 .74# [.66, .83] 161.45*
8. Strategy game 10 .46# [.32, .61] 67.31*
Note. *: significant at the .05 level; d+:
weighted mean effect size; k: number of articles; CI: confidence interval; #: d
+positive and 95% CI significantly different from zero ;(): post hoc.
2.2.6 Examination of digital game characteristics on academic achievement
The researcher separated digital game characteristics into intrinsic vs. extrinsic, tightly linked vs. loosely linked, hard-wired vs. engines and templates or shells, reflective vs. action, synchronous vs. asynchronous, single-player vs.
multiplayer, session-based games vs. persistent-state, video-based vs.
animation-based and narrative-based vs. reflex-based and then examined possible moderators.
(1) Intrinsic vs. Extrinsic
Research results showed that both weighted mean ES are positive and 95% CI are significantly different from zero, meaning that in intrinsic or extrinsic, the effect of applying DGBL on students’ academic achievement is significantly better than the effect of applying Non-DGBL on students’ academic achievement.
Because intrinsic vs. extrinsic (QB =20.64, p<.05) achieved significant level, it meant that intrinsic vs. extrinsic is the moderator of academic achievement, and
that intrinsic (ES=.69) is significantly better than extrinsic (ES=.47).
(2) Tightly linked vs. Loosely linked
Among 76 articles, there are 74 articles studying on tightly linked and two articles studying on loosely linked. Because loosely linked only has two article, and the 95% CI is not different from zero, the comparison is impossible.
(3) Hard-wired vs. Engines and templates or shells
Research results showed that both weighted mean ES are positive and 95% CI are significantly different from zero, meaning that in hard-Wired or Engines and Templates or Shells, the effect of applying DGBL on students’ academic achievement is significantly better than the effect of applying Non-DGBL on students’ academic achievement.
Because hard-wired vs. engines and templates or shells (QB = 33.57, p<.05) achieved significant level, it meant that hard-wired or engines and templates or shells is the moderator of academic achievement, and that engines and templates or shells (ES= .61) is significantly higher than hard-wired (ES=.32).
(4) Reflective vs. Action
All 76 articles belong to Reflective. Therefore, comparison is impossible.
(5) Synchronous vs. Asynchronous
Research results showed that both weighted mean ES are positive and 95% CI are significantly different from zero, meaning that in synchronous or asynchronous, the effect of applying DGBL on students’ academic
achievement is significantly better than the effect of applying Non-DGBL on students’ academic achievement.
Because synchronous vs. asynchronous (QB =19.59, p<.05) achieved significant level, it meant that synchronous vs. asynchronous is the moderator of academic achievement, and asynchronous (ES=.78) is significantly better than synchronous (ES=.50).
(6) Single-player vs. Multiplayer
Research results showed that both weighted mean ES are positive and 95% CI are significantly different from zero, meaning that in single-player or multiplayer, the effect of applying DGBL on students’ academic achievement is significantly better than the effect of applying Non-DGBL on students’
academic achievement.
Because single-player vs. multiplayer (QB =17.88, p<.05) achieved significant level, it meant that single-player vs. multiplayer is the moderator of academic achievement, and multiplayer (ES=.79) is significantly better than single-player (ES=.50).
(7) Session-based games vs. Persistent-state games
Research results showed that both weighted mean ES are positive and 95% CI are significantly different from zero, meaning that in session-based games or persistent-state games, the effect of applying DGBL on students’ academic achievement is significantly better than the effect of applying Non-DGBL on students’ academic achievement.
Because session-based games vs. persistent-state games (QB = 6.20, p<.05) achieved significant level, it meant that session-based games vs.
persistent-state is the moderator of academic achievement, and session-based (ES=.59) is significantly better than persistent-state games (ES=.48).
(8) Video-based vs. Animation-based
All 76 articles belong to Animation-Based. Therefore, comparison is impossible.
(9) Narrative-based vs. Reflex-based
Only narrative-based weighted mean ES is positive and 95% CI is significantly different from zero, meaning that in narrative-based game, the effect of applying DGBL on students’ academic achievement is significantly better than the effect of applying Non-DGBL on students’ academic achievement.
Table 4.7
Results of Digital Game Characteristics Examination
Q
B k d+ 95%CIQ
wDigital game characteristics
Intrinsic vs. Extrinsic 20.64* 76
1. Intrinsic 24 .69# [.61,.78] 218.64*
Synchronous vs. Asynchronous 19.59* 76
1. Synchronous 63 .50# [.45,.545] 559.71*
2. Asynchronous 13 .78# [.65,.91] 214.05*
Single-Player vs. Multiplayer 17.88* 76
1. Single-Player 67 .50# [.46,.55] 696.04*
Note. *: significant at the .05 level; d+:
weighted mean effect size; k: number of articles; CI: confidence
interval; #: d
+positive and 95% CI significantly different from zero; (): post hoc.
2.3 Secondary moderator analysis on academic achievement
In the first stage moderator analysis, the result showed that some categories Qw are significant, meaning that the ES in those groups are not homogenous and further analysis is necessary. Therefore, the researcher set three criteria to elicit proper data:
firstly, the Qw is significant; secondly, k>20; thirdly, researcher’s concern. Eight groups fitted the criteria: nature and science, elementary, university, Asia, Europe, North America, puzzle games and simulation games. The research results are as follows:
2.3.1 Nature and science group on academic achievement (1) Nature and science and educational level
As Table 4.8 shows, different educational level is the reason to cause the heterogeneity of academic achievement in nature and science. Research results showed that except junior high, each weighted mean ES is positive and 95% CI is significantly different from zero, meaning that in elementary, senior high and university, the effect of applying DGBL on students’ nature and science academic achievement is significantly better than the effect of applying Non-DGBL on students’ nature and science academic achievement.
As Table 4.8 shows, different educational level is the reason to cause the heterogeneity of academic achievement in nature and science. Research results showed that except junior high, each weighted mean ES is positive and 95% CI is significantly different from zero, meaning that in elementary, senior high and university, the effect of applying DGBL on students’ nature and science academic achievement is significantly better than the effect of applying Non-DGBL on students’ nature and science academic achievement.