行政院國家科學委員會補助專題研究計畫 █成果報告
（計畫名稱）
以經驗式學習理論為基礎之軟體工程遊戲性學習系統發展與
評估
計畫類別：■ 個別型計畫 □ 整合型計畫
計畫編號：NSC 99－2511－S－151－002－MY2
執行期間： 99 年 08 月 01 日至 101 年 07 月 31 日
計畫主持人：吳文雄
共同主持人：黃仕斌
計畫參與人員：黃思涵、李松霖
成果報告類型(依經費核定清單規定繳交)：□精簡報告 ■完整報告
本成果報告包括以下應繳交之附件：
□赴國外出差或研習心得報告一份
□赴大陸地區出差或研習心得報告一份
■出席國際學術會議心得報告及發表之論文各一份
□國際合作研究計畫國外研究報告書一份
處理方式：除產學合作研究計畫、提升產業技術及人才培育研究計畫、
列管計畫及下列情形者外，得立即公開查詢
□涉及專利或其他智慧財產權，□一年□二年後可公開查詢
執行單位：國立高雄應用科技大學
中 華 民 國 101 年 10 月 22 日
摘要
本研究是以軟體工程課程為基礎，開發兩個遊戲式學習系統並對學習者進行學習成效與態 度之評估。第一個遊戲式學習系統是以 role play 方式(如程式設計師)，而第二個遊戲式學 習系統是以 drill-and-practice 方式來學習軟體開發過程。透過實驗法本研究發現，學習成效 的測驗是有差異(F=5.166, p<.05)。例如，在 drill-and-practice 方式學習的平均分數為 72.86 分而 role play 方式則為 61.90 分。再者，在學習態度方面，學生在透過系統進行學習意圖 與其滿意度則無差異(F=.016, p>.05; F=1.854, p>.05)。最後，本研究對上述的結果進行討論。 關鍵詞：遊戲式學習、軟體工程、量化評估、實驗法Abstract
This study developed and evaluated two varied game-based learning systems in a software engineering class. The first system adopted a role-playing gaming strategy for students to learn about the process of software development effort estimation in a team-based environment. They played different characters, such as a project leader, a system analyst, a system designer, or a programmer. The second system used a traditional drill-and-practice gaming strategy to guide design. An experimental study involving human subjects was conducted to compare the effects of both systems on students’ learning performance and attitude. Univariate analyses revealed that the game-based learning had a significant main effect on dependent variables in the knowledge test (F=5.166, p<.05), but not in the intention and satisfaction measures (p>.05). The results showed that students in the drill-and-practice gaming group achieved significantly higher scores in the knowledge test (M=72.86) than the role-playing gaming group (M=61.90). The measures of students’ intention and satisfaction were in the range of a moderate to high level. However, no significant differences were found between the two groups about their affective measures: (1) intention to use gaming platforms (F=.016, p>.05) and (2) satisfaction toward the assigned gaming platforms (F=1.854, p>.05). The effect of students’ motivational factors in both gaming groups was about the same.
Keywords: game-based learning; software engineering education; quantitative evaluation;
experimental study
1. INTRODUCTION
Engineering education is rapidly evolving as the advancement of technological innovations accelerates. With the prevalence of game-based technologies in engineering education, instruction and learning are changing from a teacher-centered approach to a student-centered learning environment. Many research studies in engineering education revealed that the use of instructional gaming strategies in classrooms facilitated students’ learning in a variety of engineering disciplines [1-2]. Compared to traditional teacher-centered learning, in a game-based learning environment, students may perform better academic achievement not only in obtaining
factual information (lower-level thinking), but also in creating reasoning-related knowledge (higher-order thinking) [3-4].
Mayo [3] suggested that game-based learning could be a good pedagogical approach to compensate traditional teacher-centered learning in engineering education. Mayo considered that game-based learning has the potential to improve overall learning outcomes of engineering students by providing the following six features: (1) experiential learning (learning by doing), (2) inquiry-based learning (experimentation or simulation), (3) self-efficacy (decision making), (4) goal setting, (5) cooperation (team work), and (6) cognitive modeling (continuous feedback). However, very few research studies explore whether the use of gaming learning technologies will result in positive student learning achievement, which warrants a further investigation.
Some other educators have performed resource-intensive work and attempted to integrate a game-based learning system into their classrooms. For example, in medical education, Mann et al. [5] developed an interactive game system to teach surgical management algorithms by using Microsoft Visual Basic. Several interactive 3-D physical examination simulations were implemented in the study. Roubidoux et al. [6] developed an interactive web-based breast imaging game by using JavaScript. In computer science education, Baker et al. [7] developed an educational card game that simulated software engineering development methodology based on a waterfall model that included these seven phases: (1) setup, (2) turn structure, (3) requirements, (4) design, (5) implementation, (6) integration, and (7) product delivery. In engineering education, a review of existing literature showed that the number of studies integrating game-based learning into college curricula was limited.
In engineering education, a review of existing literature showed that the number of studies integrating game-based learning into college curricula was limited. Many previous studies focused on computer science-related subjects. A summary of game-based learning case studies in engineering education is shown in Table 1.
Table 1. Game-based learning cases in engineering education
Author(s) Focused Content Area in Engineering
Education
Ford & Minsker [8] Development of a game for teaching data structure
Computer science
Martin [9] Design of a game for teaching information systems development
Computer science
Lawrence [10] Teaching data structure by using a game-based learning strategy
Computer science
Yeh [11] Teaching programming by using a game
Computer Science
Veronese et al. [12] Design of a model to support for simulation-based training games
Software engineering
Dantas et al. [13] A simulation-based game for software project manager
Software engineering
Hoek [14] (SimSE) for software engineering education
Connolly et al. [15] Development of a game
(SDSim) for teaching concepts in software engineering
Software engineering
Wu et al. [16] Development and evaluation of a game-based software engineering educational system
Software engineering
Baker et al. [7] Development of a card game that simulates the software
engineering process
Software engineering
Drappa & Ludewig [17] Development of a simulator for teaching software engineering
Software engineering
Ye et al. [18] Integrating 3-D online game (Second Life) into course instruction
Software engineering
Ebner & Holzinger [19] Integrating a online game into instruction
Civil engineering
Although researchers found that instruction incorporating game features may lead to improved teaching and learning, development and evaluation of a game-based learning system by professionals in higher education still needs further investigation. Therefore, this study designed, developed, and evaluated varied game-based learning systems in a software engineering course. Two different types of game-based learning systems were investigated. The first system adopted a role-playing gaming strategy for students to learn about the process of software development effort estimation in a team-based environment. They played different characters, such as a project leader, a system analyst, a system designer, or a programmer. The second system used a traditional drill-and-practice gaming strategy to guide design. An experimental study involving engineering undergraduate students was conducted to compare the effects of both systems on students’ learning performance and attitude.
Based upon the purpose of the study, one research null hypothesis is drawn: no statistically significant differences in student knowledge test achievement, intention to use systems, and satisfaction toward systems when they learn by playing in the two different types of gaming systems. The course content for developing the game-based learning systems is one unit of instruction in an undergraduate software engineering course - the process of software development effort estimation, including the stages of initial estimation, cost estimation, time estimation, risk estimation, and final decision making.
2. SYSTEM FRAMEWORK
The two game-based learning environments (role-playing game and drill-and-practice game) proposed in this study adopt the following system framework (Fig. 1). This system framework
consists of four different conceptual layers. The first layer, e-Learning Entrance, provides learners with instructional resources used in the course contents. The learners can operate the system functions at their own pace. The second layer, Instructional Game Design, provides learners with instructional tutorials about how to install the game-based learning tools and play. The third layer, XNA Framework, provides the development tools used in the game design including different audios, graphics, videos, and learning contents. The last layer, .Net
Framework, uses Visual C# as the development language that connects ADO.NET and XNA.
Fig.1. System framework of the game-based learning design
3. METHOD
3.1 Participants
Forty-two engineering undergraduate students in a software engineering class participated in the study. Out of the participants, Thirty-one (74%) were male; eleven (26%) were female. Their age levels ranged from 20 to 24.
3.2 Instructional materials
The course content for developing the game-based learning systems is one unit of instruction in an undergraduate software engineering course - the process of software development effort estimation, including the stages of initial estimation, cost estimation, time estimation, risk estimation, and final decision making.
3.3 Independent/dependent variables
One independent variable with two varied levels was studied. These two levels were the two different types of game-based learning systems. The first system adopted a role-playing gaming strategy for students to learn about the process of software development effort estimation in a team-based environment. They played different characters, such as a project leader, a system analyst, a system designer, or a programmer. The second system used a traditional drill-and-practice gaming strategy to guide design.
Three dependent variables were measured: (1) a knowledge achievement test, (2) intention to use gaming platforms, and (3) satisfaction toward gaming learning environments. A knowledge
test was given after the students learned from the assigned experimental treatments. A questionnaire was designed to measure these two student affective constructs: (1) intention to use gaming platforms, and (2) satisfaction toward the assigned gaming platforms. The two affective constructs were measured by a seven-point Likert scale. In order to guarantee the validity of the two dependent measures, the test items and the questionnaire were reviewed by subject matter experts.
3.4 Research Design
The research design of the study was a 1 x 2 randomized post-test design. Since the dependent variables are typically related statistically and conceptually, a multivariate analysis of variance (MANOVA) is performed to analyze the results. The real value of using MANOVA is in controlling Type I error (also known as false positive error) while analyzing multiple dependent variables simultaneously. According to the research design, two gaming treatments were created: (1) role-playing gaming strategy and (2) a traditional drill-and-practice gaming strategy. A brief description of the two instructional treatments follows.
3.4.1 Treatment 1: Role-playing gaming (RPG) strategy. In this treatment, a unit of interactive materials was developed using RPG strategy. Students learned about the process of software development effort estimation in a team-based environment. They played different characters, such as a project leader, a system analyst, a system designer, or a programmer. 3.4.2 Treatment 2: Drill-and-practice (DAP) gaming strategy. This treatment used a traditional drill-and-practice gaming strategy to guide design. Students learned about the course knowledge and skills through repetitive practices.
3.5 Experimental procedures
To avoid a sampling bias, all participants were randomly assigned into two groups. Before conducting the experiment, researchers explained to students about the purpose of experiment and demonstrated them how to use varied types of gaming platforms. To reduce interferences between different treatment groups, the RPG group and the DAP group were arranged in separate rooms. After being assigned to their experimental locations, the participants were presented with their assigned instructional treatments. They were allowed to play the assigned games in 45 minutes at their own pace. After this playing session was completed, the participants were first asked to take a knowledge test and then fill out questionnaires on their intention and satisfaction of using the games.
4. FINDINGS
4.1 Descriptive Statistics
A summary of descriptive statistics for all dependent measures is shown in Table 2.
Table 2. Treatment means and standard deviations of dependent measures
Dependent Measures Role-Playing Game M/SD* Drill-and-Practice M/SD* Knowledge Test Intention Satisfaction 61.90/16.32 5.13/0.64 4.69/0.68 72.86/14.89 5.15/0.72 5.04/0.96 *M: Mean; SD: Standard Deviation
4.2 Results of the Multivariate Analysis of Variance (MANOVA)
Table 3 shows the value of Wilks’ Lambda was .760, which was significant at the p-value of .05. That is, a significant effect of gaming strategies was found (Lambda (1,40) = .760, p<. 05). .
Table 3. Results of multivariate tests
Effect Wilks’ Lambda F P Intercept Treatment Groups .010 .760 881.910 2.916 .000* .034* * Significant at .05 level
According Cronk [20], Wilks’ Lambda determines whether an independent variable has any effect on dependent variables. Table 3 indicated that a significant effect of instructional gaming strategies was found (Lambda=.760, p< .05). To further investigate the effect of the varied gaming strategies on dependent variables, univariate analyses (Table 4) revealed that the independent variable had a significant main effect on dependent variables in the knowledge test (F=5.166, p<.05), but not in the intention and satisfaction measures (p>.05). The results showed that students in the drill-and-practice gaming group achieved significantly higher scores in the knowledge test (M=72.86) than the role-playing gaming group (M=61.90) when they were assigned to varied gaming environments. The measures of students’ intention and satisfaction were in the range of a moderate to high level (Means range: 4.69~5.15). However, no significant differences were found between the two groups about their affective measures: (1) intention to use gaming platforms (F=.016, p>.05) and (2) satisfaction toward the assigned gaming platforms (F=1.854, p>.05). The effect of students’ motivational factors in both gaming groups was about the same.
Table 4. Tests of between-subjects effects
Source Degrees of
Freedom
Sum of Squares F-ratio p-value
Knowledge Test Groups Error Total Intention Groups Error Total Satisfaction Groups Error Total 1 40 42 1 40 42 1 40 42 1259.524 9752.318 201700.000 0.008 18.563 1128.785 1.295 27.936 1023.734 5.166 0.016 1.854 .028* . .899 .181 * Significant at .05 level;
The result of the study confirms Ebner and Holzinger’s findings [19] on implementing user-centered games in higher education that students seem to enjoy the game-based learning environment, when designed properly. The result is also in accordance with Foss and Eikaas’ finding [21] that the learners favor a combined use of a traditional exercise with an interactive
gaming environment, as opposed to a traditional teacher-centered learning environment.
An interesting result was found that students in the drill-and-practice gaming group yielded a higher satisfaction score (M=5.04) than in the role-playing gaming group (M=4.69). A possible explanation was that the students in the role-playing gaming group may take longer time and need extra efforts in learning the process of software development effort estimation in a team-based environment by playing different roles, such as a project leader, a system analyst, a system designer, or a programmer. During such a short period of time in experiment for the current study, they may need extra assistance in getting familiar with the system.
In terms of the knowledge test, the drill-and-practice gaming group yielded a significantly better performance on the test than the role-playing gaming group. It may be due to that fact that this knowledge test of the experiment only consists of factual and conceptual knowledge, not involving higher-order thinking skills including procedures and other meta-cognitive problem solving skills. Therefore, Future research should consider assessing more advanced students’ learning outcomes such as procedures, principles, and problem solving skills.
5. CONCLUSION
This study explored the effects of varied game-based learning systems in a software engineering course. Forty-two undergraduate students majoring in Electronic Engineering participated in the study. Two game-based learning systems were developed and evaluated by considering human participants. The role-playing gaming group learned about the process of software development effort estimation in a team-based environment by playing different characters, such as a project leader, a system analyst, a system designer, or a programmer. The drill-and-practice gaming group learned the same course contents through playing repetitive practices.
The results indicated that the students in both gaming groups expressed a positive intention to use the systems and were satisfied with the gaming platform features, which is consistent with other gaming research findings [11 16, 21]. A simple game design (drill-and-practice) yielded a significantly better learning outcome than a complex game design (role-playing gaming) in terms of learning factual and conceptual knowledge. This study continues Wu et al.’s assertion [16] that future gaming research should focus on assessing students’ authentic learning achievement instead of simply reporting research data via surveys and questionnaires. The expected learning outcomes should include facts, concepts, comprehensions, problem solving skills and other higher critical thinking skills while the students use such a game-based learning system.
According to the findings of the study, future research should continue to investigate the impact of game-based learning technologies along with different instructional strategies on engineering students’ learning achievement, such as facts, concepts, comprehensions, problem-solving, and critical-thinking skills. In addition, future studies should consider human factors in a game-based learning environment, such as learners’ individual differences, learning styles, etc. Many of the independent variables associated with the study of aptitude-treatment interactions should be taken into account in the design of game-based learning environment. Learners’ prerequisites and prior competencies should be considered to further investigate how
they interact with a game-based learning system.
REFERENCES
1. J. Lean, J. Moizer, M. Towler, and C. Abbey, Simulations and games: Use and barriers in higher education, Active Learning in Higher Education, 7(3), 2006, pp. 227-242.
2. R.T. Hays, The effectiveness of instructional games: A literature review, 2004-2005 Technical
Report, Naval Air Warfare Center, Orlando, FL, 2005.
3. M.J. Mayo, Games for science and engineering education, Communications of ACM, 50(7), 2007, pp.30-35.
4. C.A. Steinkuehler, Why game (culture) studies now? Games and Culture, 1(1), 2006, pp. 97-101.
5. B.D. Mann, B.M. Eidelson, S.G. Fukuchi, S.A. Nissman, S. Robertson, and L. Jardines, The development of an interactive game-based tool for learning surgical management algorithms via computer, The American Journal of Surgery, 183, 2002, pp. 305-308.
6. M.A. Roubidoux, C.M. Chapman, M.E. Piontek, Development and evaluation of an interactive Web-based breast imaging game for medical students, Academic Radiology, 9(10), 2002, pp. 1169-1178.
7. A. Baker, E.O. Navarro, and A. van der Hoek, An experimental card game for teaching software engineering processes, Journal of Systems and Software, 75, 2005, pp. 3-16.
8. C.W. Ford and S. Minsker, TREEZ - An educational data structures game, Journal of
Computing Sciences in Colleges, 18(6), 2003, pp. 180-185.
9. A. Martin, The design and evolution of a simulation/game for teaching information systems development. Simulation and Gaming, 31(4), 2000, pp. 445-463.
10. R. Lawrence, Teaching data structures: Using competitive games. IEEE Transactions on
Education, 47(4), 2004, pp. 459-466.
11. K.C. Yeh, Using an educational computer game as a motivational tool for supplemental instruction delivery for novice programmers in learning computer programming. Proceedings of
SITE Conference, Charlton, South Carolina, USA, 2009, pp. 1611-1616.
12. G. Veronese, M. Barros, and C. Werner, Model support for simulation-based training games: From behavioral modeling to user interactions, In J. Munch & D. Pfahl (Eds), Software Process
Simulation and Modeling Workshop (ProSim), St Louis, USA, 2005, pp. 9-15.
13. A.R. Dantas, M.O. Barros, and C. Werner, Simulation models applied to game-based training
for software project managers. In J. Munch & D. Pfahl (Eds), Software Process Simulation and Modeling Workshop (ProSim), St Louis, USA, 2005, pp. 110–116.
14. E. Oh Navarro and A. van der Hoek, Software process modeling for an educational software engineering simulation game, Software Process Improvement and Practice, 10(3), 2005, pp. 311–325.
15. T.M. Connolly, M. Stansfield, and T. Hainey, An application of games-based learning within software engineering, British Journal of Educational Technology, 38(3), 2007, pp. 416-428. 16. W.H. Wu, W.F. Chen, T.L. Wang, and C.H. Su, Developing and evaluating a game-based software engineering educational system, International Journal of Engineering Education, 24(4),
2008, pp. 681-688.
17. A. Drappa, and J. Ludewig, Simulation in software engineering training, Proceedings of the
2000 International Conference on Software Engineering, Limerick, Ireland, 2000, pp. 199-208.
18. E. Ye, C. Liu, and J. Polack-Wahl, Enhancing software engineering education using teaching aids in 3-D online virtual worlds, Proceedings of IEEE Frontiers in Education Conference, Milwaukee, WI, USA, 2007, pp. T1A/1-6.
19. M. Ebner, and A. Holzinger, Successful implementation of user-centered game based learning in higher education: An example from civil engineering, Computer & Education, 49, 2007, pp. 873-890.
20. B.C. Cronk, How to Use SPSS, Glendale, CA: Pyrczak Publishing, 1999.
21. B.A. Foss and T.I. Eikaas, Game play in engineering education: Concept and experimental results, The International Journal of Engineering Education, 22(5), 2006, pp. 1043-1052.
