An investigation into effectiveness of different reflective learning strategies for learning operational software

(1)

An investigation into effectiveness of different re

ﬂective learning

strategies for learning operational software

Chorng-Shiuh Koong

a,*

_{, Tzu-I Yang}

b

_{, Chao-Chin Wu}

c

_{, Han-Tai Li}

c

_{, Chien-Chao Tseng}

b a_{Department of Computer Science, National Taichung University of Education, No.140, Min-Sheng Rd, Taichung 403, Taiwan, ROC}

b_{Department of Computer Science, National Chiao Tung University, No.1001, University Rd., Hsinchu 300, Taiwan, ROC} c_{Department of Computer Science and Information Engineering, National Changhua University of Education, No. 2, Shi-Da Rd.,}

Bao-Shan Campus, Changhua 500, Taiwan, ROC

a r t i c l e i n f o

Article history:

Received 20 February 2013 Received in revised form 3 October 2013

Accepted 7 November 2013 Keywords:

Reﬂective learning strategies Operational software Learning effectiveness

Computer-assisted learning (CAL) ACT-R model

a b s t r a c t

Skill certification promotion is one of the main policies facilitated by the technological and vocational education, where application software instruction is regarded as the core curriculum to foster skill certification. With its close connection with problem-solving learning, application software instruction relies heavily on hands-on operation incorporating information technology to adequately unravel the challenges where living or working application is simulated as problem situations. According toDewey (1963)andEdwards (1996), the process of reflection is characterized by the inference course where learners attempt to analyze and solve the problems. However, more evidence is needed to decide what reflective learning strategies are effective for students’ learning. Application software operation is categorized as procedural knowledge. Repeated drills are requisite to reach the ultimate goal of spon-taneous reaction without thinking. Features of CAL system offer a well-rounded environment to meet the demands. The purposes of this study were 1) to investigate how different reflective learning strategies can affect learning effectiveness of operational application software acquisition, 2) to identify effective learning strategies and to incorporate the CAL approach with instructional practices to foster learning performance. Aiming at characteristics of operational software, this study proposed operational software learning strategy theory model based on reflective learning and Adaptive Character of Thought-Rational (ACT-R) model theories. The proposed model modified the reflective learning theory and added cyclical loop into CAL tofit for operational software instruction. The CAL system is developed and incorporated into learning activities of reflective learning theory strategy model by collecting frequent operation er-rors made in thefirst-year experiment as the source drill items. This study is conducted in a two-year sequence. A total of 172 second-grade students was recruited from a vocational high school. Different reflective learning strategies, individual and group reflective learning strategy, are implemented on two experimental groups in thefirst year. CAL strategy is later added into the experimental groups in the second year. The results suggest that group reflective learning strategy can enhance learning effective-ness of the holistic and medium-score group students. When reflective learning strategy is incorporated with CAL, in addition to maintaining thefirst-year learning effectiveness, learning effectiveness of the holistic and low-score group students can be benefited by individual reflective learning strategy. Furthermore, reflective learning incorporating with CAL has greater learning effectiveness than the learning without CAL.

1. Introduction

The process of reﬂection is usually characterized by the inference course where learners attempt to identify, analyze, and solve the problems (Dewey, 1963; Edwards, 1996; Park & Son, 2011). It is the mental and emotional activities that individual engages in searching and probing for prior experiences in the attempt to solve the problems (Boud, Keogh, & Walker, 1985). During the process, learners are allowed to

* Corresponding author. Tel.: þ886 4 22183582 (Department ofﬁce); fax: þ886 4 22183580.

E-mail addresses:csko@mail.ntcu.edu.tw(C.-S. Koong),tiyang@cs.nctu.edu.tw(T.-I. Yang),ccwu@cc.ncue.edu.tw(C.-C. Wu),taiteacher@saihs.edu.tw(H.-T. Li),cctseng@ cs.nctu.edu.tw(C.-C. Tseng).

Contents lists available atScienceDirect

Computers & Education

j o u r n a l h o m e p a g e : w w w . e l s e v i e r . c o m / l o c a t e / c o m p e d u

(2)

face a dilemma and consider what is needed to address the problem through the steps of reflection, which are vital to learn (Henderson, Napan, & Monteiro, 2004; Park & Son, 2011; Potting, Sniekers, Lamers, & Reverda, 2010). Jay (1999)suggested that reflection can be treated as problem-solving strategy. Reflecting on the process of an action is beneficial to untangle a challenge.Boud et al. (1985)have described reflection as intellectual activities in which individuals engage to explore their experiences in order to generate new un-derstandings and appreciations. It may take place in individual or in-group settings. Many studies have recognized that reflection plays a (Gotoda, Sakurai, Matsuura, Nakagawa, & Miyaji, 2013, pp. 84–93) critical role in learning process in that it can foster learning effectiveness (Chi, DeLeeuw, Chiu, & Lavancher, 1994; Lee & Hutchison, 1998; McNamara, 2004), which include Motor learning (Gotoda et al., 2013), Professional education (Lyons, Halton, & Freidus, 2013), Medical education (Carek, Geiger, Oelklaus, James, & Karty, 2013; MacDermott, 2013), and Law education (Rué, Font, & Cebrián, 2013) for higher education. It can also be used in Mathematics Learning (Yu, 2013) and Language Learning (George, 2013, pp. 335–357) for young learners.

Approaches to reflection may involve reflective journals, logs, portfolios, and self-writing (Barney & Mackinlay, 2010; Carrington & Selva, 2010; Moon, 2004). They proffers a elaborate list of information that are intended to help learners understand how to learn reflectively. However, it is relatively hard to check if the students do actively reflect (Ryan & Ryan, 2013). Collaborative reflections, on the other hand, can help students actively re_{flect though different reflective skills. Through information sharing, helping each other, discussion, and evaluating} one another’s ideas can help students to improve the reflection efficiently. For the past decade, many researches had only reveal part of the information on effectiveness of using individual reflection (Gotoda et al., 2013; Rué et al., 2013) or only focus on self-report which may not be precise enough. It is relatively unclear that different reflective learning strategies will affect the performance of learning. Besides, how to apply reflective learning with appropriate strategies on different ability students is still not clear. Therefore, more evidence is needed to decide what reflective learning strategies are more useful for students’ learning.

The prominent position which application software instruction holds within the technological and vocational education is widely recognized in terms of computer science curriculum, skill certification, and promoting quality workforce. Acquisition of application software relates closely to problem-solving learning, because it may involve many problem-solving strategies which include lateral thinking and trial and error. Trial and error is a fundamental method of solving problems (Helman, 1989). It is characterized by repeated, varied attempts which are continued until success, or until the agent stops trying. Lateral thinking (Bono, 1967) is solving problems through an indirect and creative approach, using reasoning that is not immediately obvious and involving ideas that may not be obtainable by using only traditional step-by-step logic. Where living or working application is simulated as problem situations, learners are expected to adopt information technology integrating pertinent knowledge of computer science, goal-identi_{fication, and strategic steps to adequately unravel the} chal-lenges (Browning, 2010; Léger et al., 2011). Grounded on prior knowledge or experience, problem-solving demands the coordination with reflection to surmount a dilemma, process of which is a learning experience (Jeppesen & Lakhani, 2010; Wang & Chiew, 2010).Chi and Glaser (1985)have suggested that problem-solving is the process during which individuals strive tofind solutions to attain a specific goal.

In addition, application software operation is categorized as procedural knowledge, such as open thefile, insert picture from file, and create a new table. Procedural knowledge refers to having the understanding of the procedure of an action. In other words, it is the knowledge acquired by learning the sequence of an operation (Anderson, 1983). The function of procedural knowledge is that it can equip learners with rapid and mastery performance, which when adopting on specific knowledge domain can help cultivating specialized personnel (Lin, 2007; Luechtefeld & Watkins, 2011). The ultimate goal of procedural knowledge acquisition is automation, that is, spontaneous reaction without thinking. Once the level is reached, instead of being constantly attentive to certain messages, individuals can engage fast operation without thinking and thereby reducing working memory load (Chang, 1996). However, repeated drills on procedural knowledge are necessary before automation and learning transfer can be attained (Gagne & Briggs, 1992; Simpson, 1972). Accordingly to Adaptive Character of Thought-Rational (ACT-R) theory of knowledge representation, procedural knowledge operation is often accompanied by declarative knowledge. With their close interaction, procedural knowledge cannot be fully acquired without stressing the learning of declarative knowledge (Lin, 2007). Although reflection is widely recognized as beneficial to problem-solving related courses, incorporating procedural knowledge into the learning of operational software needs further empirical evidence to prove its validity on learning effectiveness.

Computer-Assisted Learning (CAL) is operated in the way that question items are put forward by computer for learners to engage repeated drills for the proficiency of the learning content (Hartley, 2010; Huang, Liu, & Chang, 2012; Yalcin & Celikler, 2011). Since op-portunity provided by CAL on the same concept or question is unlimited and feedback offered is instant, learners can be benefited and consequently enhance their learning effectiveness (Nirmalakhandan, 2007). For skill learning, adopting CAL on the underachievers have shown significant result on the enhancement of their learning effectiveness (Seo & Bryant, 2012). The abovementioned suggested that integrating CAL system is effective in the learning of operational software.

The purposes of this study were 1) to investigate how different reﬂective learning strategies can affect learning effectiveness of oper-ational application software acquisition, 2) to identify effective learning strategies and to incorporate the CAL approach with instructional practices to foster learning performance.

This study designed an operational software learning strategy theory model (shortened as OSLST-model) based on characteristics of operational software, Adaptive Character of Thought-Rational (ACT-R) model theory, and reflective learning strategies. Students’ frequent operation errors made in the process of learning Microsoft Word from thefirst-year experiment are identified and collected as the source drill items for the design of CAL system. Later, results from the two-year experiment are used to scrutinize how individual and group reflective learning strategy instructions, and how CAL system can affect students_{’ learning effectiveness in the acquisition of operational software.} 1.1. Research questions

To underpin the hypothesis, we investigate the following questions:

1. Does different learning effectiveness exist between different reﬂective learning strategies without CAL for holistic and different level students?

2. Does different learning effectiveness exist between different reﬂective learning strategies with CAL for holistic and different level students?

(3)

3. Does different learning effectiveness exist between different reﬂective learning strategies without and within CAL for holistic and different level students?

2. Literature review 2.1. Reﬂective learning

2.1.1. Connotation of reﬂective learning

The concept of reflection in the domain of education was originated from John Dewey’s experiential learning theory. Learners are ex-pected to conduct active thinking and integrate past and present knowledge before constructing new learning experiences.Dewey (1933) suggested that reflection is the action that one takes with active motivation, special care, and incessant quest over a specific problem. Reflection forms its gradual shape when one consciously and voluntarily engages recurrent and scrupulous thinking based on the conclusion he deducted from any conviction or presupposed knowledge. Reflection consists of two parts: 1) a sense of uncertainty and mental confusion; and 2) an action of inquiry in the attempt to_{find out a solution to the perceived dilemma.}Boud et al. (1985)posited that reflection is the mental and emotional activities that individual engages in searching and probing for prior experiences in the attempt to solve the problems.Soloman (1987)unveiled that reflection is an integrating process of combining past experiences, actions, and the theories received into a new value that is signi_{ficant to oneself. That is, reflection is one’s re-inquiry and reorganization over the established} knowledge for his own understanding and meaning.Carver and Scheier (1998)held that reflection is the act that individual takes to examine, evaluate, and clarify his thinking, feeling, and behavior.Boud et al. (1985)have described reflection as intellectual activities in which individuals engage to explore their experiences in order to generate new understandings and appreciations. It may take place in isolation or in-group settings.Schön (1987)proposed that learner’s reflection can be categorized into three types: reflection-for-action, reflection-in-action, and reflection-on-action.

Approaches to reflection may involve reflective journals, logs and portfolios (Barney & Mackinlay, 2010; McGuire, Lay, & Peters, 2009; Moon, 2004). They proffers a comprehensive list of ideas that are intended to help learners understand how to learn or write re flec-tively. However, the question remains as to whether student really do actively reflect (Ryan & Ryan, 2013). Collaborative reflections, on the other hand, can help students actively reflect though different reflective skills. Through information sharing, helping each other, discussion, and evaluating one another’s ideas can help students to improve the reflection efficiently. On the other hand, researches had only reveal part of the information on effeteness of using individual reflection (Gotoda et al., 2013; Rué et al., 2013) or only focus on self-report which may not be precise enough. It is relatively unclear that different reflective learning strategies will affect the performance of learning operational knowledge. The reflective learning strategies this study designed is implemented after assessment. Students are asked to engage individual and group reflection on their operative behaviors during the assessment and on the results attained afterward. The reflection adopted in this study was reflection-on-action.

2.1.2. Courses of reﬂective learning

Zimmerman (1998, 2000)andZimmerman and Schunk (1989)proposed three phases in his theory of self-regulated learning: fore-thought, performance or volitional control, and self-reflection. Among which, reflection is divided into four cyclical loops: self-evaluation, attributions, self-reactions, and adaptively, as shown inFig. 1.Montgomery (1993)raisedfive steps in his reflective learning process: do, look, think, evaluate, and plan. Learners in each step are expected to carry out reflection based on their performance in the previous step.Fig. 2shows Montgomery’s reflective processes. Reflective processes may be different between the two versions. However,

Fig. 1. Steps of self-reﬂection in the self-regulated learning theory.

(4)

both scholars emphasized the importance of cyclical feedback loop. Cyclical reflection was valued to hold the key to the demanding plight. Both theories coincide with Dewey’s assertion that reflection must be recurrent actions, and steps proposed by both scholars are closely jointed with one another. To integrate the self-reflection with the self-regulated learning theory in learning operational knowledge, it is required to modify the existed learning model accordingly. The reflective learning theory strategies developed in this study were conformed to the rationale and were implemented in the experiment accordingly.

2.2. Theories of knowledge representation 2.2.1. Implication of knowledge representation

Representation refers to any notation, sign, or set of symbol representing the external objective or internal subjective world of an object. Such object can either be a characteristic, an entity, or an imaginative existence (Da Silva Souza, Oberauer, Gade, & Druey, 2012; Franco et al., 2012; Mislevy et al., 2010; Zhao, Liu, & Hu, 2012). It is an expression that one adopts to describe trait, connection, symbolism, mental image, concept, and ideology. In other words, representation is how individual portrays a phenomenon, or attempts to conclude an explanation (Cadoli, Donini, Liberatore, & Schaerf, 2011; Huang et al., 2012; Lin, 2007). Knowledge representation is how message is presented in our long-term and working memory (Gagne, Yekovich, & Yekovich, 1998). The process of knowledge representation was shaped when individual transforms all sorts of knowledge into processable message. This transformation involves the evolution of replacing entity with concept, and diverse mental images generated from psychological activities (Chen, Xie, Zeng, & Li, 2011). Consequently, knowledge representation is the projection of how individual perceives messages from the external world, and how he copes with his learning state (Halford, 1993).

Many scholars tempt to subdivide knowledge into declarative knowledge and procedural knowledge. Declarative knowledge refers to the understanding of a specific subject, such as fact, theory, and object, which can be described in phrases. For instance, that Abraham Lincoln is the sixteenth president of the United States is a factual knowledge. Procedural knowledge, on the other hand, relates to how things are done, which can include motor skill, cognitive skill, and cognitive strategy. Giving a well-defined expression to such knowledge may be difficult since its intricate nature. For example, a person may be skilled in riding a bike, but may fail to explain how to keep balance, or how not to fall down from it when making a turn (Anderson & Lebiere, 1998, pp. 19–56;Best, 1989; Wärnestål & Lindqvist, 2012). In addition to defining difficulty, procedural knowledge is dynamic and diverse in essence. Repeated practice is required before mastery of such knowledge can be displayed in a series of actions, making the acquisition of it a lengthy process. It is for the same reason that rectifying a procedural knowledge is challenging for a learner once he has acquired the knowledge, and has reached the level of automation (Gagne et al., 1998).

Knowledge representation is presented differently among diverse disciplines.Rumelhart and Norman (1985)classi_{ﬁed the knowledge} representation into propositional, analogical, and procedural representation. Among which, knowledge in procedural representation is presented according to the production rule, which is condition-action rule. The composition of a series of production rules is essentially the projection of all inner or exterior behavioral procedure.

2.2.2. Model of ACT-R knowledge representation

Sternberg and Mio (2009)suggested that Adaptive Character of Thought-Rational (ACT-R) model proposed by J. R.Anderson (1983) portrays a complete paradigm among all theories combining different framework of knowledge representation. The model integrates both context representation of declarative knowledge and process representation of procedural knowledge (Sternberg & Mio, 2009). As shown inFig. 3, ACT-R model encompasses three major parts: declarative memory, procedural memory, and working memory (Marewski & Mehlhorn, 2011; Trafton, Altmann, & Ratwani, 2011). Declarative memory stores long-term memory into the structure of declarative knowledge. It is the pre-requisite condition to trigger production memory. Procedural memory is related to the memory zone of procedural knowledge. It coordinates the processing task of match and execution by the production rules. Meanwhile, production memory also im-plements the application process. Working memory works as the medium between declarative knowledge and procedural knowledge. It manages the coding input/input coding of the exterior messages, and retrieves both knowledge from declarative memory and commands from production memory. The messages are in a priming state, and will project the outcome through performance process.

Operative mechanism of ACT-R model consists of_{ﬁve phases (}Anderson, 1983; Lin, 2007): storage, retrieval, match, execution, and application: 1). Storage manages the integration of old and new messages, and places the results into long-term memory. 2). Retrieval refers to the process of transforming messages being stored in the long-term memory into an operative state. 3). Match supervises the mapping

(5)

task of requisite conditions of a speciﬁc production rule. Such production rule will be triggered once all conditions are under operation in working memory. 4). Execution implements the actions prescribed by production rules. 5). Application refers to the function that production memory will automatically replace the composition of conditions and actions into another or more types of rule.

Declarative knowledge, based on ACT-R model, is the pre-condition for procedural knowledge (production memory) to generate actions. In other words, declarative knowledge works as the ground basis for procedural knowledge. Consequently, procedural knowledge cannot be fully acquired without stressing the learning of declarative knowledge.

2.3. Signiﬁcance of computer-assisted learning

The evolution of information technology has made computer a dominant necessity of modern life. Features such as presenting diversified information, interacting with and providing feedbacks to learners are what make computer not only is widely adopted in educational practices, but also is valued highly as an instructional media (Hannafin, 1987). The term Computer-Assisted Instruction (CAI) was used when computer was regarded primitively as a tool for instruction. Learners were viewed as passive recipients whereas teaching took the leading role during the instructional activities. Cognitive psychologists later came to the realization that learners should be taking the initiative in learning instead, and should be taking the responsibility to integrate and construct comprehensive knowledge on their own. Application of computer then has transformed its role into the learner-centered CAL, where learning received more attention ever since (Berry & Chew, 2008; Blumberg, 2008; Kafai, Carter Ching, & Marshall, 1997; Meece, 2003; Yasa, 2012). Unlike other instructional approaches, learners under CAL are allowed to have more exposure of vivid multimedia, rich opportunities for drills, and reviews over specific subjects (Seo & Bryant, 2012).

Simonson and Thompson (1990)held that all computer-related learning activities can be deﬁned as CAL.Wu (1992)supported the use of CAL over CAI for its wider connotation, which can include CAI, Computer Directed Instruction (CDI), and Computer Enhanced Instruction (CEI). Software courseware of CAL can be divided into many categories.Alessi and Trollip (1991)have classiﬁed the courseware into tutorials, drill, simulation, games, and tests.

The self-developed system in this study can be classified as drill type CAL software. First, frequent operation errors made in the first-year experiment were collected as the basis for classification. Second, the system provides drill environment by presenting items of declarative knowledge and procedural knowledge. Finally, instant feedbacks of positive and negative reinforcement are delivered right after learners finish the answers.

3. Operational software learning strategy theory model

3.1. Introduction to operational software learning strategy theory model

This study proposed a reflective learning strategy theory OSLST-model which the CAL is introduced to engage learners themself to act the reflection activity to generate positive reaction, which learners are passive and less motivated to perform reflective learning activities (Ajeneye, 2005). Besides, to meet the demands of learning software applications, we bring up a new sub-loop between Adaptively and Self-evaluation, which modified and integrated self-regulated and self-reflective learning strategies. The proposed OSLST-model is divided into two parts: reflective learning and CAL. Reflective learning is an adaptation of Zimmerman (1998) self-regulated learning theory and Montgomery (1993)reflective learning process, which is implemented in the first-year experiment. However, both these two models are not applicable for operational software. Steps proposed by Zimmerman are adopted since they are more in tune with reflection-on-action. On the other hand, Self-reaction steps are proposed for the individual to examine his performance and progress toward the set goal, where one can generate positive reaction for himself. The intention is to have the individual more motivated to seek out adaptive strategies through attribution. However, the aptness of attribution is not included in the steps for inspection. Processes advised byMontgomery (1993), instead, incorporated evaluation and plan for learners to observe the aptness of their attribution before framing a problem-solving plan. Rationale of the latter is more to the objective of our design and is included in our model. Besides, according to many practitioners’ experience, instead of not having the correct concept, the failing of learners’ operational strategy usually results from minor carelessness such as skipping a vocabulary when highlighting the set area during the process of application software acquisition. Such small mistakes can often be corrected by taking more practices. Repetition of the whole reflective loop is not necessary as long as operational strategy is proven to be effective. For this reason, this study modified the reflective loop by adding a sub-loop between adaptively and self-evaluation. Learners are allowed to engage more try-and-errors before re-starting the reflective loop, thus shorten the time spent on problem-solving.

Our drill type CAL system is an integration of knowledge representation and CAL theories, which is developed and used in the second-year experiment. Aiming at learners’ frequent operation errors, the CAL system provides a drill environment after learners engaged reﬂective learning. With the bi-directional loops within the OSLST-model, learners are also allowed to engage reﬂective learning after drills. Fig. 4shows the OSLST-model.

Steps of the OSLST-model include self-evaluation, attribution, evaluation, plan, adaptivity, and CAL. Each step is described as follows: 1. Self-evaluation: Main focus of this step is to compare the answer results with the set goal. For individual re_{ﬂective learning strategy}

(shortened as I-RLS), students are asked to engage self-evaluation on errors being made and to make related list before result of the assessment is given for reference. For group reﬂective learning strategy (shortened as G-RLS), groups also engage self-evaluation before making relevant comparison. The aim for engaging self-evaluation after adaptivity is to conﬁrm whether or not problems are properly solved.

2. Attribution: For I-RLS groups, core task of this step is to identify reasons for failing the target after self-evaluation. Learners’ answering process is recorded with video software for further reﬂection and attribution. G-RLS groups also engage attribution in a group setting. 3. Evaluation: Main concern of this stage is aptness of attribution. For I-RLS groups, students are asked to evaluate aptness of their contribution and to compare change of concepts before and after this step. G-RLS groups conduct comparison between different points of view and develop the best option.

(6)

4. Plan: Conclusion derived from evaluation is used as the problem-solving plan. For I-RLS groups, students are asked to develop a feasible plan to self-solve the problems and to write the plan on a piece of paper. For G-RLS group, the plan is decided with the participation of all members.

5. Adaptivity: Adaptivity is conducted based on the plan. Learners are asked to make modification on their answer results according to their plans. Veri_{fication is required on the modified results. Re-operate the process based on the plans if the results are proven to be} incorrect.

6. CAL: Frequent operation errors collected from thefirst-year experiment are listed as the target for drills under our CAL system after reflective learning activity. Or, drills can also be engaged before reflective learning activity.

Flowcharts of I-RLS and G-RLS activities are shown inFigs. 5and6. 3.2. Computer-assisted learning system

The self-developed CAL system is subjoined into the second-year experiment. Theory background is detailed as follows.

1. Computer application software operation is categorized as procedural knowledge (Gagne & Briggs, 1992). The ultimate goal of proce-dural knowledge acquisition is automation (Chang, 1996). Repeated drills on procedural knowledge are necessary before automation and learning transfer can be attained (Gagne & Briggs, 1992; Simpson, 1972). The opportunity and feedback learners can be beneﬁted from CAL on the same concept is unlimited and instant (Nirmalakhandan, 2007). The practice button designed in the system is for learners to engage repeated drills. Meanwhile, instant feedbacks of positive and negative reinforcement are delivered right after learnersﬁnish the answers.

Fig. 4. Operational software learning strategy theory model.

(7)

2. According to ACT-R theory of knowledge representation, operation of procedural knowledge is often accompanied by declarative knowledge. With their close interaction, procedural knowledge cannot be fully acquired without stressing the learning of declarative knowledge (Lin, 2007). Hence, declarative items are included in the CAL system to foster learners’ declarative knowledge acquisition. 3. Irrelevant items are excluded from of the target practices to avoid unnecessary waste of time and decrease of learning motive (Seo &

Bryant, 2009).

Based on the abovementioned considerations, frequent operation errors collected from thefirst-year experiment are collected and included in the CAL system for drills. Results of the three classes participated in thefirst-year experiment were added up and classified into error number and error corrected number. Major error numbers and minor error corrected numbers are sorted inTable 1. Based on items listed inTable 1, items of declarative knowledge are created, answer of which is selected by one click, as shown inFig. 7. Items of procedural knowledge are created, answer of which is selected by a series of operative steps, as shown inFig. 8a. Besides, there may exist more then one approach to solve the same question, as shown inFig. 8b.

4. Experiment design

The purposes of the experiment were to investigate the learning effectiveness of different level students using different reﬂective learning strategies (individual and group reﬂective), incorporated with/without the CAL approach. The experiment result can be applied to instructional practices to foster learning performance, especially useful for the individualized instruction. To underpin the hypothesis, our research questions include:

1. Does different learning effectiveness exist between different reﬂective learning strategies without CAL for holistic and different level students?

2. Does different learning effectiveness exist between different reﬂective learning strategies with CAL for holistic and different level students?

3. Does different learning effectiveness exist between different reﬂective learning strategies without and within CAL for holistic and different level students?

This study is conducted in a two-year sequence. The aim of thefirst-year experiment is to examine how different reflective learning strategies, individual and group strategy, can affect learning effectiveness of the holistic and different learning achievers. Frequent operation errors collected from thefirst-year experiment are included as the drill items in the CAL system. CAL system is then added into the two experimental groups in the second year. Target of which is to investigate how CAL system can affect learning effectiveness of the holistic and different learning achievers.

4.1. Introduction to experiment design 4.1.1. Experiment process

Instructional content is divided into three units of courseware: C1–C3. The experiment consists of three instructional units and one summative assessment. Four phases are included in each instructional activity: step 1) giving in-class instructions; step 2) implementing a post-instruction formative assessment; step 3) conducting reﬂection on the assessment results using I-RLS and G-RLS; step 4) executing

(8)

another formative assessment using the same test sheet after reflection. Each summative assessment consists of three parts: step 1) using different test sheets to implement assessment based on the content of three instructional activities; step 2) conducting reflection on the assessment results; step 3) executing the second summative assessment using the same test sheets. CAL strategy is included into each reflective learning activity of the experimental groups in the second year. Flowcharts of the first and second experiments are shown in Figs. 9and10.

Fig. 7. Cal screenshots of declarative knowledge items. Table 1

Frequent operation errors.

Item First assessment

error number

Second assessment error number

Error corrected number First formative assessment Picture alignment (change the alignment of picture) 66 34 32

Insert page number 60 24 36

Picture height (change the height of the picture) 55 36 19 Picture width (change the width of the picture) 54 24 30 Specify date & time formats 49 11 38 Specify header & footer without border 45 21 24 Picture border (specify the border around the picture) 45 14 31 Align the width of header & footer to the width of paper 37 22 15 Picture wrapping style (specify the wrapping style of picture) 34 8 26 Second formative assessment Column (split text into two or more columns) 59 21 38 Decrease indent (decrease the indent level of the paragraph) 49 25 24

Font (change the font face) 49 14 35

Insert a blank line between paragraphs 48 30 18 Line and paragraph spacing (change the spacing between lines of text) 36 26 10 Paragraph border (change the border of paragraph) 32 10 22 Paragraph shading (change the shading of paragraph) 30 9 21 Text alignment (change the alignment of text) 29 6 23 Font style (choose a style of text) 25 11 14 Third formative assessment Input text into a table 66 55 11 Align the text of table (change the alignment of table text) 60 31 29 Table shading (change the shading of table) 56 11 45 Change the position of text in a table 51 19 32 Cell alignment (change the alignment of cells) 47 18 29 Merge cells (merge the selected cells into one cell) 44 12 32 Justify table margins (align the table margins to the width of text) 37 20 17 Table column width (change the width of table column) 31 25 6 Summative assessment Picture alignment (change the alignment of picture) 59 38 21

Input text into a table 54 35 19

Picture height (change the height of the picture) 48 24 24 Insert a blank line between paragraphs 45 29 16 Justify table margins (align the table margins to the width of text) 41 35 6 Line and paragraph spacing (change the spacing between lines of text) 37 26 11 Change the position of text in a table 37 11 26 Specify header & footer without border 35 22 13 Align the text of table (change the alignment of table text) 34 11 23 Picture width (change the width of the picture) 33 18 15 Table column width (change the width of table column) 33 12 21 Column (split text into two or more columns) 31 21 10 Font style (choose a style of text) 30 14 16

Font (change the font face) 27 14 13

Decrease indent (decrease the indent level of the paragraph) 26 14 12 Cell alignment (change the alignment of cells) 26 11 15 Paragraph shading (change the shading of paragraph) 25 11 14

(9)

4.1.2. Subject and grouping

The experiment is conducted within a two-year time frame. A total of 172 students from department of business administration is recruited as subjects of this study from a vocational high school. The subjects are around age of 16, they all have basic computer operation skill and also have attended the course of Introduction to Computer Science. Based on the original placement, three classes of second-grade female students are recruited each year. Three participating classes are randomly assigned to contrast, I-RLS and G-RLS experimental groups, three for each year and total six classes for two year. Subjects of G-RLS experimental groups are divided into six S-shape heterogeneous reﬂective learning groups based on their score of Introduction to Computer Science course of prior semester. They all ﬁnished the word operation courses, which include in-class lecture and operation practice in computer rooms.Table 2shows experiment grouping.

(10)

4.1.3. Experiment tool

Experiment instructional courseware: Scope of“National Skill Testing of Level C technician for computer software application” (shortened as CSA-level C) hosted by the Council of Labor Affairs (CLA) of the Executive Yuan of Taiwan was selected as the target instructional content. Assessment test sheet: Three self-edited formative assessment test sheets (F–C1, F–C2, F–C3) and one summative assessment test sheet (S) are used in the study. Expert validity is conducted under three experts with over 15 years of instructional experience on information technology. Modiﬁcation is conducted according to given suggestions.

Software tool

a) CAL system: The system is developed by VB6.0 programming language. Main system content is to provide an operative environment for learners to practice frequent operation errors collected from theﬁrst-year experiment, where both declarative and procedural knowledge items are included and instant feedback is given to learners after theirﬁnishing practices.

b) Video recorder software: Learners’ answering process is recorded by video recorder software as the reference for learners to engage individual or group reﬂective learning.

c) Expert assessment system on CSA-level C: The expert assessment system (Computer-Assisted Instruction Website, 2010) was used as the grading tool on each formative and summative assessment. This system is designed for assessing CSA-Level C test by using the collection of ofﬁcial test inventory. Each test sheet has a unique number as the identity. The students’ answer will be submitted to CSA with correspondent test sheet number, and then we can obtain the assessment results. And we collect all the assessment results from CSA while we perform the formative and summative assessments.

(11)

4.1.4. Experiment design description

This study used a quasi-experimental pre-test/post-test factorial design. The aim of the study is to examine how different learning strategies can affect students’ learning effectiveness in the acquisition of WORD application software.Fig. 11illustrates the experiment design.

1. Control variable

(1) Lecturer: All six classes participated in the experiment are given the instruction under the same lecturer.

(2) Instructional courseware: Three types of WORD courseware, C1–C3, are authored for experiment based on the scope of CSA-level C. (3) Assessment tools: All six classes participated in the experiment were given the same formative and summative assessment test

sheets by taking the examination on computer.

Fig. 10. Flow chart for the second-year experiment with cal system.

Table 2

Experiment groupings and number distribution.

Contrast group I-RLS group G-RLS group Total

First-year grouping 28 30 30 88

Second-year grouping 28 29 27 84

(12)

2. Independent variable: Different learning strategies are implemented on diverse groups. Individual and group reﬂective learning within the experimental groups are directed by the lecturer and are included in theﬁrst-year experiment. CAL system is then added into the experimental groups in the second-year experiment.

3. Dependent variable: Testing results are assessed under the Expert assessment system. 4.1.5. Data processing

Data collected is processed by SPSS Statistics 17.0 and is analyzed by the following methods:

1. One-way ANOVA is conducted to examine how different learning strategies can affect learning effectiveness of the holistic students; the analysis results in Section5.1.2.1and discussions were stated in Sections5.2.1.1 and 5.2.2.

2. One-way ANOVA is conducted to investigate how different learning strategies can affect learning effectiveness of students with diverse learning achievements; the analysis results in Sections5.1.2.2–5.1.2.4and discussions were stated in Sections5.2.1.2–5.2.1.4 and 5.2.2.

3. Independent-sample t test is conducted to compare the holistic learning effectiveness of the experimental groups within the two years; the analysis results in Section5.1.3and discussions were stated in Section5.2.3.

5. Analysis results and discussion 5.1. Data analysis

SPSS Statistics 17.0 is applied to engage one-way ANOVA and independent-sample t test on data collected. Formative assessment is used to compare progress record after learning strategy is employed, while summative assessment is adopted to analyze total scores of the examination being taken.

5.1.1. Homogeneous sample analysis

A total six classes of subjects from theﬁrst- and second-year experiments participated in the study. Subjects’ score of Introduction to Computer Science course of prior semester are used to categorize the experiment samples into groups. Test of homogeneity is conducted on the samples by one-way ANOVA. Scores of different learning achievers are used as the grouping determiner. The top 27% is grouped as high-score group, the middle 46% is medium-high-score group, while the bottom 27% low-high-score group.Table 3shows the test results. No signiﬁcant

Fig. 11. Design for theﬁrst- and second-year experiments.

Table 3

One-way ANOVA on the pre-test result.

SS df MS F P Holistic .511 .768 Between groups 286.473 5 57.295 Within group 18630.242 166 112.230 Total 18916.715 171 High 1.119 .367 Between groups 77.569 5 15.514 Within group 540.875 39 13.869 Total 618.444 44 Medium 1.962 .094 Between groups 170.979 5 34.196 Within group 1324.923 76 17.433 Total 1495.902 81 Low 25.852 1.288 .289 Between groups 129.260 5 Within group 782.518 39 20.065 Total 911.778 44

(13)

difference is indicated on students’ holistic scores, evidence of which is conformed to homogeneous assumption. Learners’ initial ability of each class can be regarded as the same (P¼ .768 > .05). Similarly,Table 3shows no signiﬁcant difference on test results acquired from high-, medium-, and low-score groups, which is also conformed to homogeneous assumption (high P¼ .367 > .05, medium P ¼ .094 > .05, and low P¼ .289 > .05).

5.1.2. One-way ANOVA on learning effectiveness

Learners’ progress records collected from formative and summative assessment are used to conduct one-way ANOVA for learning effectiveness comparison. Scheffe method then is applied to engage Post-Hoc comparison, if the results reach level of signiﬁcance. Detailed results are illustrated as follows.

5.1.2.1. Holistic learning effectiveness comparison. Table 4summarizes holistic learning effectiveness from the two-year experiments. For the first year, three formative assessments and the second summative assessment reached level of significance (P ¼ .042 < .05, P ¼ .014 < .05, P¼ .012 < .05, P ¼ .002 < .05). Results of Post-Hoc comparison indicated that learning effectiveness of the G-RLS group out-performed the contrast groups on three formative assessments. Similarly, the G-RLS groups also out-performed both the contrast groups and the I-RLS group on the second summative assessments. For the second year, all assessments reached level of significance (P ¼ .000 < .05, P¼ .000 < .05, P ¼ .002 < .05, P ¼ .000 < .05, P ¼ .000 < .05). Results of Post-Hoc comparison indicated that learning effectiveness of the G-RLS groups out-performed the contrast groups on all assessments. The I-G-RLS groups out-performed the contrast groups on thefirst and second formative assessments, and the second summative assessment.

5.1.2.2. Learning effectiveness comparison on high-score groups. We omitted the statistics table of high-score groups, because of the dif-ferences between each assessment did not reach level of signiﬁcance, indicating that no signiﬁcant difference was found in learning effectiveness of the high-score groups.

5.1.2.3. Learning effectiveness comparison on medium-score groups. Table 5summarizes learning effectiveness on medium-score groups from the two-year experiments. For thefirst year, three formative assessments and the second summative assessment reach level of sig-nificance (P ¼.037 <.05, P ¼.024 < .05, P ¼.030 < .05, P ¼.025 < .05). Results of Post-Hoc comparison indicated that learning effectiveness of G-RLS groups out-performed the contrast groups on three formative assessments and the second summative assessment. For the second year, all assessments reach level of significance (P ¼ .022 < .05, P ¼ .025 < .05, P ¼ .033 < .05, P ¼ .009 < .05, P ¼ .006 < .05). Results of Post-Hoc comparison indicated that learning effectiveness of G-RLS groups out-performed the contrast groups on all assessments. I-RLS groups out-performed the contrast groups on the second summative assessment.

5.1.2.4. Learning effectiveness comparison on low-score groups. Table 6summarizes learning effectiveness on low-score groups from the two-year experiments. For theﬁrst year, the second summative assessment reached level of signiﬁcance (P ¼.009 < .05). Results of Post-Hoc

Table 4

One-way ANOVA on the holistic learning effectiveness.

Year Item SS df MS F Signiﬁcant Post-Hoc

First year First formative assessment Between groups 731.797 2 365.898 3.283 .042* G> C Within group 9474.763 85 111.468

Total 10206.560 87

Second formative assessment Between groups 445.977 2 222.988 4.513 .014* G> C Within group 4200.098 85 49.413

Total 4646.075 87

Third formative assessment Between groups 893.315 2 446.658 4.624 .012* G> C Within group 8210.901 85 96.599

Total 9104.216 87

First summative assessment Between groups 778.332 2 389.166 2.628 .078 Within group 12588.131 85 148.096

Total 13366.463 87

Second summative assessment Between groups 1240.480 2 620.240 6.480 .002** G> C G> I Within group 8135.990 85 95.718

Total 9376.471 87

Second year First formative assessment Between groups 2104.129 2 1052.065 12.573 .000*** I> C G> C Within group 6777.906 81 83.678

Total 8882.036 83

Second formative assessment Between groups 2254.109 2 1127.055 9.961 .000*** I> C G> C Within group 9164.593 81 113.143

Total 11418.702 83

Third formative assessment Between groups 1468.476 2 734.238 6.839 .002** G> C Within group 8696.226 81 107.361

Total 10164.702 83

First summative assessment Between groups 1906.647 2 953.323 8.432 .000*** G> C Within group 9158.305 81 113.065

Total 11064.952 83

Second summative assessment Between groups 1543.046 2 771.523 9.522 .000*** G> C I> C Within group 6562.990 81 81.025

Total 8106.036 83 Note:*P < .05, **P < .01, ***P < .001.

(14)

Table 5

One-way ANOVA on the learning effectiveness of the medium-score group.

First year First formative assessment Between groups 439.063 2 219.532 3.587 .037* G> C Within group 2387.126 39 61.208

Total 2826.190 41

Total 2609.490 41

Total 6093.746 41

Total 4958.631 41

Second summative assessment Between groups 505.583 2 252.791 4.068 .025* G> C Within group 2423.580 39 62.143

Total 2929.163 41

Second year First formative assessment Between groups 604.286 2 302.143 4.235 .022* G> C Within group 2639.714 37 71.344

Total 3244.000 39

Total 7072.775 39

Total 4637.500 39

First summative assessment Between groups 1041.010 2 520.505 5.432 .009** G> C Within group 3545.390 37 95.821

Total 4586.400 39

Second summative assessment Between groups 622.032 2 311.016 5.944 .006** I> C G> C Within group 1935.868 37 52.321

Total 2557.900 39 Note:*P < .05, **P < .01.

–: No signiﬁcant difference; C: contrast group; I: I-RLS group; G: G-RLS group.

Table 6

One-way ANOVA on the learning effectiveness of the low-score group.

First year First formative assessment Between groups 198.913 2 99.457 .360 .702 Within group 5519.000 20 275.950

Total 5717.913 22

Second formative assessment Between groups 114.406 2 57.203 1.195 .323 Within group 957.351 20 47.868

Total 1071.757 22

Third formative assessment Between groups 178.090 2 89.045 1.077 .360 Within group 1653.919 20 82.696

Total 1832.009 22

Total 3297.800 22

Second summative assessment Between groups 1077.063 2 538.531 5.962 .009** G> C G> I Within group 1806.589 20 90.329

Total 2883.652 22

Second year First formative assessment Between groups 1698.312 2 849.156 8.879 .002** I> C G> C Within group 1817.143 19 95.639

Total 3515.455 21

Second formative assessment Between groups 1495.487 2 747.744 10.448 .001** I> C G> C Within group 1359.786 19 71.568

Total 2855.273 21

Total 4069.818 21

First summative assessment Between groups 898.344 2 449.172 5.771 .011* G> C Within group 1478.929 19 77.838

Total 2377.273 21

Second summative assessment Between groups 1542.312 2 771.156 9.305 .002** I> C G> C Within group 1574.643 19 82.876

Total 3116.955 21 Note:*P < .05, **P < .01.

(15)

comparison indicated that learning effectiveness of G-RLS groups out-performed the contrast groups and I-RLS groups on the second summative assessment. For the second year, all assessments reached level of signiﬁcance (P ¼ .002 < .05, P ¼ .01 < .05, P ¼ .040 < .05, P¼ .011 < .05, P ¼ .002 < .05). Results of Post-Hoc comparison indicated that learning effectiveness of G-RLS groups out-performed the contrast groups on all assessments. I-RLS groups out-performed the contrast groups on theﬁrst and second formative assessments, and the second summative assessment.

5.1.3. Independent-sample t test on two-year learning effectiveness of the experimental groups

Independent-sample t test is conducted on the progress record of the formative assessments taken by I-RLS groups within the two years. The aim is to compare whether or not the incorporation of CAL system has enhanced I-RLS groups’ learning effectiveness in the second-year than that of theﬁrst-year. According to previous research, CAL can help improve the learning effectiveness. Therefore, we make the assumption that the learning effectiveness of second year will improve signiﬁcantly, so we can directly use one-tailed test to perform analysis. Being a right-tailed test, the null hypothesis is set up at H0:

m

1

m

2. Research hypothesis H1:

m

1>

m

2 (

m

1: I-RLS groups’ second-year learning effectiveness;

m

2: I-RLS groups’ ﬁrst-year learning effectiveness). Later, divide the signiﬁcance level (two-tailed) by 2 and examine if the value is less than

a

(.05), which means the difference reaches signi_{ﬁcant level.}

5.1.3.1. Holistic learning effectiveness comparison. Table 7summarizes holistic learning effectiveness on the experimental groups of the two years. For I-RLS groups, all three formative assessments and the second summative assessment reached level of signiﬁcance (

a

¼ .001 < .05,

a

¼ .001 < .05,

a

¼ .0275 < .05,

a

¼ .022 < .05). Hence, the null hypothesis was rejected. Results of which indicated that the incorporation of CAL system significantly enhanced learning effectiveness of the second year than that of the first year. For G-RLS groups, all three formative assessments and the first summative assessment reached level of significance (

a

¼ .0055 < .05,

a

¼ .001 < .05,

a

¼ .0235 < .05,

Table 7

Independent-sample t test on the experimental groups’ learning effectiveness between the two-year experiments.

Group Item Year Number Mean SD t df Signiﬁcant (one-tailed) MD I-RLS group First formative assessment Second year 29 14.172 9.2082 3.170 57 .001* 8.1057

First year 30 6.067 10.3730

Second formative assessment Second year 29 12.828 10.0110 3.249 47.219 .001* 7.1243 First year 30 5.703 6.3691

Third formative assessment Second year 29 13.966 11.2297 1.958 57 .0275* 5.0555 First year 30 8.910 8.4538

First summative assessment Second year 29 78.862 9.9489 1.425 57 .080 4.1954 First year 30 74.667 12.4788

Second summative assessment Second year 29 88.276 8.5394 2.063 57 .022* 5.3192 First year 30 82.957 11.0589

G-RLS group First formative assessment Second year 27 18.41 11.067 2.646 55 .0055* 8.057 First year 30 10.35 11.837

Second formative assessment Second year 27 18.48 12.656 3.192 55 .001* 9.008 First year 30 9.47 8.431

First summative assessment Second year 27 84.33 9.454 1.836 55 .036* 5.063 First year 30 79.27 11.172

Second summative assessment Second year 27 91.41 6.053 1.054 55 .148 1.944 First year 30 89.46 7.671

Note:*a< .05.

Table 8

Independent-sample t test on the medium-score group’s learning effectiveness between the two-year experiments.

First year 14 6.38 5.812

Second formative assessment Second year 13 12.538 12.2584 1.578 18.237 .066 6.0599 First year 14 6.479 6.6727

Third formative assessment Second year 13 13.077 11.6581 .848 25 .2025 3.4341 First year 14 9.643 9.3359

G-RLS group First formative assessment Second year 13 15.54 9.803 1.287 25 .105 4.581 First year 14 10.96 8.697

Third formative assessment Second year 13 20.31 10.625 .920 25 .183 4.493 First year 14 15.81 14.321

Second summative assessment Second year 13 91.08 6.788 .929 25 .929 2.591 First year 14 88.49 7.637

(16)

a

¼ .036 < .05). Hence, the null hypothesis was rejected. Results of which indicated that the incorporation of CAL system signiﬁcantly enhanced learning effectiveness of the second year than that of theﬁrst year.

5.1.3.2. Learning effectiveness comparison on high-score groups. We omitted the statistics table of high-score groups, because of that none of the assessment reached level of signiﬁcance within the experimental high-score groups, suggesting that no signiﬁcant difference was made on learning effectiveness for the two years.

5.1.3.3. Learning effectiveness comparison on medium-score groups. Table 8shows learning effectiveness on the experimental medium-score groups of the two years. For I-RLS groups, theﬁrst formative assessment and the second summative assessment reached level of signiﬁcance (

a

_{¼ .028 < .05,}

a

_{¼ .030 < .05). Hence, the null hypothesis was rejected. Results of which indicated that the incorporation of CAL system} significantly enhanced learning effectiveness of the second year than that of the first year. For G-RLS groups, the second summative assessment reached level of significance (

a

¼ .0495 < .05). Hence, the null hypothesis was rejected. Results of which indicated that the incorporation of CAL system signiﬁcantly enhanced learning effectiveness of the second year than that of the ﬁrst year.

5.1.3.4. Learning effectiveness comparison on low-score groups. Table 9summarizes learning effectiveness on the experimental low-score groups of the two years. For I-RLS groups, theﬁrst and second formative assessments, and the second summative assessment reached level of signiﬁcance (

a

¼ .026 < .05,

a

¼ .0115 < .05,

a

¼ .0405 < .05). Hence, the null hypothesis was rejected. Results of which indicated that the incorporation of CAL system significantly enhanced learning effectiveness of the second year than that of the first year. For G-RLS groups, all three formative assessments reached level of significance (

a

¼ .0275 < .05,

a

¼ .0005 < .05,

a

¼ .0375 < .05). Hence, the null hypothesis was rejected. Results of which indicated that the incorporation of CAL system signiﬁcantly enhanced learning effectiveness of the second year than that of theﬁrst year.

5.2. Experiment result and discussion

5.2.1. Different reﬂective learning strategies without CAL

Table 10summarizes results of thefirst-year experiment, in which five assessments, three formative assessments and two summative assessments, gathered from each class are listed. A discussion of details follows. Summary of thefirst-year experiments results are as below. 5.2.1.1. Holistic performance.

1) Learning effectiveness of G-RLS groups out-performed the contrast groups. During the process of reflection, reflective ability can be enhanced if contrasting opportunity is properly provided (Knights, 1985). Rodgers (2002)suggested that benefits of collaborative reflection included affirmation of the value of one’s experience generated in isolation, seeing things newly, and support to engage in the process of inquiry. Similar resonance is found that students participating in G-RLS groups were more motivated in learning.

Table 9

Independent-sample t test on the low-score group’s learning effectiveness between the two-year experiments.

First year 8 5.337 16.3238

Third formative assessment Second year 8 19.750 13.3924 1.575 14 .069 9.2875 First year 8 10.463 9.9401

First summative assessment Second year 8 69.250 8.7137 .717 14 .2425 3.3875 First year 8 65.863 10.1324

G-RLS group First formative assessment Second year 8 28.29 10.436 2.107 13 .0275* 17.573 First year 8 10.71 19.727

First summative assessment Second year 7 76.57 9.325 .866 13 .201 4.959 First year 8 71.61 12.362

Second summative assessment Second year 7 89.29 5.936 .890 13 .195 3.448 First year 8 85.84 8.598

Note:*a< .05.

Table 10

Summary of theﬁrst-year experiment results.

Holistic High Medium Low

First, second, and third formative assessments G> C – G> C –

First summative assessment – – – –

Second summative assessment G> C – G> C G> C

G> I G> I

(17)

2) Compared with the contrast groups and G-RLS groups, holistic learning effectiveness on I-RLS groups, though was enhanced, failed to reach significant difference on most of the assessments. A possible explanation for this finding is that reflective learning strategy though was thought to be beneficial to students’ learning, some studies disclosed that students tended to be less motivated to engage individual re_{flective learning when the strategy was incorporated into curriculum (}Abrami et al., 2008; Cheng & Chau, 2012). This suggests that thinking effectiveness is determined by the extent of motivation (Simpson & Courtney, 2007). In addi-tion, the lack of peer members to engage collaborative learning in I-RLS groups makes it difficult for the students to develop problem-solving strategy when faced with bottlenecks, besides their tendency of being easily distracted during the process of experiment.

5.2.1.2. High-score group performance. Performance of high-score groups did not make significant difference. It is probable that learners in high-score groups held more positive learning attitude, and had more advantage in knowledge comprehension and acquisition over other learners. Since the scale for enhancing learning effectiveness is little, none of the learning strategies could make significant difference. 5.2.1.3. Medium-score group performance. With peer members directing their learning, learning effectiveness on G-RLS groups out-per-formed the contrast groups. However, in spite that performance was enhanced on the two groups, that is, G-RLS group vs. RLS group and I-RLS group vs. the contrast group, none of the performance of the two groups made signi_{ficant difference.}

5.2.1.4. Low-score group performance.

1) Learning effectiveness between the contrast groups and I-RLS groups failed to make signiﬁcant difference. This could be inferred that learning attitude of the low-score group students was passive and less motivated in engaging reﬂective learning (Ajeneye, 2005). It was also found that students in I-RLS groups tended to give up learning once the errors made had reached to certain amount.

2) Learning effectiveness on G-RLS groups did not signiﬁcantly outperform the contrast groups and I-RLS groups until the second sum-mative assessment. This may suggest that the use of G-RLS though may empower students with reﬂective ability and willingness to learn through the direction of peer members, more times and struggling efforts were demanded of students in low-score groups on knowledge comprehension and acquisition (Cheng & Chau, 2012; Hwang, Shi, & Chu, 2010; Vygotsky, 1978).

5.2.2. Different reﬂective learning strategies with CAL

Table 11summarizes results of the second year, in whichfive assessments gathered from each class were listed. Learning effectiveness of G-RLS groups was generally the same with that of thefirst year. Significant difference was reflected on I-RLS groups whether it was on holistic performance or on low-score groups. The incorporation of CAL also helped learning effectiveness of the groups out-performed the contrast groups. The ultimate goal of procedural knowledge acquisition is automation, that is, spontaneous reaction without thinking. Once the level is reached, instead of being constantly attentive to certain messages, individuals can engage fast operation without thinking and thereby reducing working memory load (Biggs & Tang, 2011; Chang, 1996; Leinhardt, Young, & Merriman, 1995; Sas, 2006; Wärnestål & Lindqvist, 2012). Repeated drills on procedural knowledge are necessary before automation and learning transfer can be attained (Gagne & Briggs, 1992; Simpson, 1972). Since opportunity provided by CAL on the same concept or question is unlimited and feedback offered is instant, learners can be benefited and consequently enhance their learning effectiveness (Nirmalakhandan, 2007). Incorporation of CAL makes significant difference in promoting skill learning, particularly in fostering learning effectiveness of the low-score groups (Seo & Bryant, 2012). It is assumed that individual reflective learning tends to take considerable time in the process. However, the assumption did not apply to learners in the low-score groups, who did not appear to favor reflective thinking in the experiment. This could result from

Table 11

Summary of the second-year experiment results.

First and second formative assessments I(CAL)> C – G(CAL)> C I(CAL)> C

G(CAL)> C G(CAL)> C

Third formative assessment G(CAL)> C – G(CAL)> C G(CAL)> C First summative assessment G(CAL)> C – G(CAL)> C G(CAL)> C Second summative assessment I(CAL)> C – I(CAL)> C I(CAL)> C

G(CAL)> C G(CAL)> C G(CAL)> C Note:–: no signiﬁcant difference; C: contrast group; I(CAL): I-RLS group with CAL; G(CAL): G-RLS group with CAL.

Table 12

Independent-sample t test on I-RLS group’s learning effectiveness of the two-year experiments.

First formative assessment 2nd> 1st – 2nd> 1st 2nd> 1st Second formative assessment 2nd> 1st – – 2nd> 1st

Third formative assessment 2nd> 1st – – –

First summative assessment – – – –

Second summative assessment 2nd> 1st – 2nd> 1st 2nd> 1st Note:–: no signiﬁcant difference; 1st: ﬁrst-year learning effectiveness; 2nd: second-year learning effectiveness.

(18)

the adoption of the self-developed CAL system. The repeated drill environment the system provided is preferred over the process of reﬂective thinking.

5.2.3. Different reﬂective learning strategies without and within CAL

Table 12summarizes independent-sample t test results of the two-year’s learning effectiveness of I-RLS groups.Table 13summarizes independent-sample t test results of the two-year_{’s learning effectiveness of G-RLS groups. As the results indicated, holistic learning} effectiveness of the second year, with the incorporation of both reflective learning and CAL, is significantly different than that of the first year both for I-RLS and G-RLS groups. In terms of different learning group achievers, medium- and low-score groups marked the major improvement whether it_{’s for I-RLS or for G-RLS groups.}

Researches indicated that CAL can improve the learning motivation, attitude (Shaffer, 2004; Spitzer & Scherzinger, 2006; Sugand, Abrahams, & Khurana, 2010), and achievements (Kish, Cook, & Kis, 2013; Kılıç, 2007; Koklu & Topcu, 2012; Liu, 2008) of students. Other researches also implied that the learning achievement by using CAL was decided by how and when the teachers integrated different kinds of teaching strategies (Akinsola & Animasahun, 2007; Brantmeier, Flores, & Romero-Ghiretti, 2006; Fatemi Jahromi & Salimi, 2013; Yang & Huang, 2008).

While conducting CAL teaching, one of the most important is to use the appropriate CAL software (Greenhalgh, 2001; Memon, 2009; Robinson, 2007), it is relatively difficult to develop adaptable software for CAL teaching (Al-Kahtani, 2004). In our study, the develop-ment of CAL by referencing to thefirst-year statistical results to perform the reflective learning is another important factor that why it can bring better learning achievement in out experiment.

In this researches, we proposed a new OLST-model with four important factors include: 1) to develop a new CAL software for correspondent courseware unit by referencing to the first-year statistical results; 2) to integrate the CAL teaching method; 3) to introduce the reflective strategies 4) to use the CAL software in the appropriate situation of reflective cycle. Therefore, the significant improvement of the experiments result can show the bene_{fit of combing these important factors. In addition, these findings also can be} inferred that learning effectiveness on the procedural knowledge can be enhanced significantly through the drills provided by the CAL system developed in the study.

6. Conclusion and future research

This study proposed operational software learning strategy theory model aiming at characteristics of operational software acquisition. Focus of thefirst-year experiment was to investigate how different reflective learning strategies can affect learning effectiveness of oper-ational software acquisition. Learners_{’ frequent operation errors made in the process of Word software acquisition from the first-year} experiment are collected as the basis for the design of the CAL system, which was incorporated into the second-year experiment. Learning effectiveness was proven to be enhanced substantially. Results from the analysis revealed that reflective learning strategy was vital to the enhancement of operational software acquisition. Furthermore, greater learning effectiveness was shown on reflective learning with the incorporation of CAL than that of without such instruction. G-RLS strategy is recommended when conducting operational software in-struction. Significant learning effectiveness was shown on learners in the medium-score group. Learning effectiveness of low-score groups was enhanced only when G-RLS strategy is incorporated with CAL system, though effect derived from which came slowly. When adopting I-RLS, it is suggested that CAL is incorporated to significantly enhance holistic learning effectiveness, particularly for the low-score groups. The feasibility of this study has been validated by specifically classifying learners’ frequent operation errors in the acquisition of Word software, and by developing a corresponding CAL system to improve students’ mistakes. Results of the study can provide applicable reference for instructional practices.

On a broader scale, further research should be undertaken in the following areas: 1) expanding investigation scope on learning effec-tiveness of operational software; 2) including analysis on learners’ time spent and numbers of error clicking in the answering process as the benchmark for mastery observation; 3) grouping the G-RLS groups by different learning achievers, among whom in-group interaction can be scrutinized and learning effectiveness can be recorded.

Acknowledgments

This research was supported by Grant NSC-101-2511-S-142-010 and NSC-100-2218-E-142-001 from the National Science Council of the Republic of China, Taiwan. The researchers also appreciate grant from the Department of International Affairs and Research Development of National Taichung University of Education, and Embedded Systems and Software Engineering project of Ministry of Education, Taiwan. References

Abrami, P. C., Abrami, P. C., Wade, C. A., Pillay, V., Aslan, O., Bures, E. M., et al. (2008). Encouraging self-regulated learning through electronic portfolios. Canadian Journal of Learning and Technology, 34(3).

Table 13

Independent-sample t test on G-RLS group’s learning effectiveness of the two-year experiments.

First formative assessment 2nd> 1st – – 2nd> 1st Second formative assessment 2nd> 1st – 2nd> 1st 2nd> 1st Third formative assessment 2nd> 1st – – 2nd> 1st

First summative assessment 2nd> 1st – – –

Second summative assessment – – – –