• 沒有找到結果。

Constructing and evaluating online goal-setting mechanisms in web-based portfolio assessment system for facilitating self-regulated learning

N/A
N/A
Protected

Academic year: 2021

Share "Constructing and evaluating online goal-setting mechanisms in web-based portfolio assessment system for facilitating self-regulated learning"

Copied!
13
0
0

加載中.... (立即查看全文)

全文

(1)

Constructing and evaluating online goal-setting mechanisms in

web-based portfolio assessment system for facilitating self-regulated

learning

Chi-Cheng Chang

a,*

, Kuo-Hung Tseng

b

, Chaoyun Liang

c

, Yueh-Mai Liao

d

aDepartment of Technology Application & Human Resource Development, National Taiwan Normal University, No. 162, He-Ping E. Road, Section 1, Taipei

106, Taiwan

bGraduate Institute of Business and Management, Meiho University, No. 23, Pingguang Rd., Neipu, Pingtung, Taiwan cDepartment of Information Communication, Yuan Ze University, No. 135, Yuan-Tung Road, Chung-Li, Taoyuan 32003, Taiwan dDepartment of Data Processing, Yifeng Senior High School, Taiwan

a r t i c l e i n f o

Article history:

Received 18 December 2012 Received in revised form 12 July 2013

Accepted 15 July 2013 Keywords:

Computer-mediated communication Human–computer interface Improving classroom teaching Interactive learning environments Teaching/learning strategies

a b s t r a c t

The purpose of the present study was to construct goal-setting mechanisms in a web-based portfolio assessment system (WBPAS), based on the self-regulated learning (SRL) process proposed by Zimmer-man, and to examine effects of these mechanisms on SRL. The participants were two classes of 11th graders taking the website design class in a vocational high school. The participants were assigned randomly to either an experimental group (n¼ 40) learning with a WBPAS or a control group (n ¼ 41) learning with a paper-based portfolio. The study results revealed the following: a) the quality of goal-setting mechanisms may facilitate SRL. b) Students goal-setting learning goals with the WBPAS demon-strated significantly better SRL than students setting learning goals with the paper-based portfolio.

Ó 2013 Elsevier Ltd. All rights reserved.

1. Introduction

1.1. Goal setting in web-based portfolio assessment and self-regulated learning

A web-based portfolio assessment (WBPA) aims at collecting and presenting students’ learning experiences, thoughts, work, progresses and self-reflections for assessment on their learning (Lougheed, Bogyo, Brokenshire, & Kumar, 2005). A web-based portfolio assessment system (WBPAS) usually includes goal setting, work in different stages, reflection, teacher assessment and feedback, peer assessment and feedback, students’ self-assessment and feedback, and progress data (Chang, Tseng, Yueh, & Lin, 2011). The learning goals are one of the most important items in a portfolio. Based on above, the mechanisms for goal setting may at least contain setting goals, viewing the progress and revising the goals. Additionally, students are able to browse others’ portfolios, to give feedback, to self-regulate, and to review their own progress by the system. Teachers are able to do the assessment for students according to the effort a student put during his learning, the progress, and the learning goals a student achieves.

Goal setting is one of the essential activities in the procedure of web-based portfolio assessment (WBPA) (Chang et al., 2011). In order to achieve self-set goals, students regulate their own learning as time goes by. Accordingly, goal-setting facilitates self-regulated learning (SRL) and is an important factor that affects SRL. Some studies confirmed that SRL is facilitated by goal setting (Latham & Locke, 1991;Zimmerman, 2008). Actually, the portfolio itself shares features with SRL. For example, during the development of portfolios, students improve them-selves through reflections which are often performed based on self-set goals. Portfolios guide students during a learning process, and

* Corresponding author. Tel.: þ886 2 77343439; fax: þ886 2 23921015.

E-mail addresses:samchang@ntnu.edu.tw(C.-C. Chang),gohome8515@gmail.com(K.-H. Tseng),iccylian@saturn.yzu.edu.tw(C. Liang),lovabler@msn.com(Y.-M. Liao).

Contents lists available atScienceDirect

Computers & Education

j o u r n a l h o m e p a g e : w w w . e l s e v i e r . c o m / l o c a t e / c o m p e d u

0360-1315/$– see front matter Ó 2013 Elsevier Ltd. All rights reserved.

(2)

continuous self-regulation is performed based on self-set goals (Heo, 2000).Abrami et al. (2008),Carneiro, Lefrere, Steffens, and Underwood (2011)argued that an e-portfolio is helpful to SRL. Consequently, goal-setting mechanisms are obviously crucial. In addition,Riedinger (2004)pointed out advantages of a web-based portfolio, including that: a) it is not restricted by time and space; b) it is convenient to be browsed by peers; c) it is easy for peers to share with one another and to give feedback; and d) it is convenient for peers to view others’ learning goals. Therefore, how online goal-setting mechanisms can be implemented with the advantages of the Internet to enhance stu-dents’ SRL is an important issue.

1.2. Online goal-setting mechanisms

Since goal setting facilitates SRL, goal-setting mechanisms in a web-based portfolio assessment system (WBPAS) are apparently crucial. There is a lack of relevant studies about goal-setting mechanisms that facilitate students’ SRL. However, recent studies about formative assessment systems or SRL systems can be the reference for researchers.Wang (2011)enhanced students’ SRL and learning performance by review and feedback mechanisms in a formative assessment system.Arsal (2010)adopted a diary as a tool of SRL for pre-service science teachers. The attributes of the diary are similar to the portfolio. Both the diary and the portfolio allow students to review their learning progress. Actually, the diary is appropriate for keeping track of students’ development of SRL (Neber & Schommer-Aikins, 2002;Schmitz & Wiese, 2006;

Winne, 2005). A WBPAS can facilitate SRL more because it features the attributes of diary and formative assessment.Hwang et al. (2007)

enhanced students’ learning performance through SRL mechanisms in an online SRL system and online learning activities. Their study re-sults revealed that SRL process was helpful to their learning. Hence, the system did facilitate students’ SRL and learning performance.

There is some software for goal setting, but most software is for task management not for students’ learning. For example, Lifetick is an online system for goal setting and achievement. The system helps users set goals online and reminds users to attain their goals. Users are able to track their own task progress and determine if they achieved the preset goals. The features of the system include defining values, setting task, setting goals, creating tasks, staying motivated, tracking progress, measuring performance, and journal thoughts (Lifetick, 2011). Although most of these systems are for task management, they can still be the references for the development of online goal-setting mechanisms in the present study.

1.3. Research objectives and questions

However, are the online goal-setting mechanisms mentioned above appropriate to be implemented in a WBPAS? Are they enough for a WBPAS? What goal-setting mechanisms should be included in a WBPAS? Are these goal-setting mechanisms helpful to students’ SRL? Do students learning with goal-setting mechanisms in a WBPAS perform better than students learning with paper-based portfolio? What aspects of SRL can be enhanced? AsAzevedo (2005)andKollar and Fischer (2006)argued that digital environment is beneficial to SRL. According to the background above, the purpose of the present study was to construct online goal-setting mechanisms in a WBPAS and to examine its effects on SRL. The research questions are the following:

1) What is the level of student satisfaction about the online goal-setting mechanisms in the WBPAS?

2) Are there any significant differences on SRL between students who are highly satisfied with the goal-setting mechanisms in the WBPAS and students who are not highly satisfied?

3) Are there any significant differences on SRL between students learning with the WBPAS and students learning with paper-based portfolio?

2. Research methods 2.1. Participants

Participants in the present study were two classes of 11th graders taking a“Digital Data Processing” class at a vocational high school in USA. The participants were randomly assigned to either an experimental group learning with a WBPAS (40 participants) or a control group learning with paper-based portfolio (41 participants). There were a total of 81 participants, with 36 males and 45 females. The content of the course was primarily webpage design. It was a hands-on computer course which required students to complete and submit works via computer and to set goals, so it was appropriate to be implemented with the WBPAS. The main similar between WBPAS and paper-based portfolio is that the items in both types of portfolios are the same. The mail differences are that the presentation and organization formats of the items in both types of portfolios are quite different.

2.2. Research framework

The pretest–posttest controlled group design, an approach of quasi-experimental research design, was conducted in the present study, as shown inTable 1. The effect of goal-setting mechanisms in the WBPAS on SRL was examined in the present study.

Table 1

Experimental design.

Group Pretest Treatment Posttest

Experimental Grades in last semester, SRL Set learning goals by WBPAS Satisfaction toward online goal-setting mechanisms, SRL

Control Grades in last semester, SRL Set learning goals by paper-based portfolio SRL ***p< 0.001.

(3)

In order to avoid the Hawthorne effect, students in the experimental group were not informed of the experiment, and the experiment was naturally integrated into the course. In order to avoid the John Henry effect, students in the control group were also not informed of the experiment. Both groups were taught by the same teacher, and received the same curricular content, teaching schedule, requirements for work, and activities for goal setting.

2.3. Variables

The independent variables were satisfaction toward goal-setting mechanisms in the WBPAS and goal-setting method. The dependent variable was SRL.

2.3.1. Satisfaction

Ease of use and usefulness of goal-setting mechanisms, including writing, editing, review, feedback, and assessment, were evaluated. 2.3.2. Goal-setting methods

Goal-setting methods included using WBPAS and paper-based portfolio. 2.3.3. SRL

There were four aspects in SRL, including learning motivation (e.g., self-efficacy, subject value, and learning anxiety), self-observation, self-judgment (e.g., peer model, criteria, self-set goal), and self-reaction (e.g., adaptive and defensive).

1) Learning motivation: Learners’ learning willingness and ambition.

a) Self-efficacy: A learner’s belief about whether his performance satisfies the preset goal. b) Subject value: A learner’s belief about the importance of learning a subject or task.

c) Learning anxiety: A learner’s level of anxiety and pressure toward a subject test.

2) Self-observation: Learners’ records and monitor status about their achievement of preset goals.

3) Self-judgment: Learners’ belief about whether they achieve preset goals based on work of peers, criteria set by teachers, and goals set by themselves.

a) Peer model: Learners beliefs about whether they achieved preset goals based on work of peers.

b) Teacher criteria: Learners beliefs about whether they achieved preset goals based on criteria set by teachers. c) Self-set goals: Learners beliefs about whether they achieved preset goals based on goals set themselves. 4) Self-reaction: Learners’ feelings toward their progress on goal achievement.

a) Adaptive self-reaction: Learners’ positive feelings and acceptance toward their progress on goal achievement. b) Defensive self-reaction: Learners’ negative feelings and resist toward their progress on goal achievement.

2.4. Construction of online goal-setting mechanisms in WBPAS

Online goal-setting mechanisms were embedded in the WBPAS. Functions for the system included: a) a guideline for creating a portfolio; b) an area for creating a portfolio with functions of adding new goals (setting due date for achieving goals, check point for the goal progress, and outline for goal-setting guideline), adding new works, and adding new reflection (searching tools, outline for reflection guideline, and reflection prompts); c) an area for reviewing portfolio with functions of reviewing goals, reviewing reflections and reviewing works; d) an area for review of peer portfolios with functions of observing peer reflections, observing peer goals, and observing peer works; and e) an area for portfolio assessment with functions of checking grades of assessment, self-assessment and peer assessment.

2.4.1. Online goal setting and editing

A goal-setting table with functions of writing and editing goals was designed and developed in the WBPAS in the present study for assisting students to set personal learning goals. Students set their learning goals and due dates based on their own pace, and then planned to achieve goals before the due dates, as shown inFig. 1. The system also provided an online instant revising function, which allowed students to adjust their learning goals based on their learning progress.

In addition, the system also provided an outline for setting learning goals (including the scope and emphasis of learning goals), which arrows students to easily writing learning goals.

2.4.2. Online learning goal review and feedback

Students would be able to improve their learning goals and progress by observing others’ learning goals and progress via online learning goal review mechanism, as shown inFig. 2. Students could exchange ideas toward learning goals with one another and encourage one another through the feedback mechanism, as shown inFig. 3. Peer review and feedback facilitated students’ goal identification, maintained students’ learning motivations, and enhanced students’ continuance to achieve their learning goals.

2.4.3. Online learning goal assessment mechanism

Online learning goal assessment mechanisms included self-assessment and peer assessment, as shown inFigs. 4and5. To stimulate continuance learning, students could judge their status of achieving learning goals based on their learning performance. The learning goal performance scale in the WBPA proposed byChang et al. (2011)was embedded as web pages in the system, which could befilled online by students and be checked instantly for statistics results. The reliability and validity of the goal performance scale were verified by an experiment research and were highly adequate.

(4)

Fig. 2. Online learning goal review mechanism. Fig. 1. Online learning goal writing and editing.

(5)

Fig. 3. Online learning goal feedback mechanism.

(6)

Fig. 5. Online learning goal peer assessment mechanism.

(7)

In order to understand students’ regulated progress of learning goals, the system kept goal setting and editing in record for teachers and students to review their status of goal setting and editing, as shown inFig. 6.

2.5. Experimental procedures

The experiment lasted for ten weeks and there were 2 h per week. Learning activities for each week are described as the following. 1) First week

The teacher explained general ideas of the WBPAS and the meaning of goal setting to experimental group in the class. For the experi-mental group to have a better understanding of goal-setting mechanisms in the WBPAS, the teacher provided the course information, demonstrated the use of goal-setting mechanisms in the system and skills of goal setting, and allowed students to practice the basic techniques of using the WBPAS. For the control group, the teacher provided the course information and explained goal-setting skills of paper-based portfolio to students. The questionnaire on SRL was administered as the pretest to both groups in thefirst week.

2) Second to fourth weeks

For both groups, the teacher gave lectures in the class. Students’ performance on goal setting was assessed by the teacher and teacher assistant each week. For the experimental group, students engaged in each learning activity, such as goal setting, goal review and feedback, goal revision, work uploading, self-assessment on achievement of learning goal in the previous week, and anonymous peer assessment, with the WBPAS. For the control group, students engaged in each learning activity, such as goal setting, work collection, self-assessment on the achievement of learning goal in the previous week, and peer assessment, with paper-based portfolio. The class schedule for both groups was the same.

3) Fifth week

For the experimental group, students engaged in online work review and online portfolio assessment (e.g., teacher assessment, student self-assessment and peer assessment and feedback), and the questionnaire toward the satisfaction of goal-setting mechanisms was administered. For the control group, students engaged in work review and paper-based portfolio assessment (e.g., teacher assessment, student self-assessment and peer assessment and feedback). Those activities are requirement after the class for both groups. Both groups have the same assessment activities but different performing ways (online versus non-online). Moreover, both groups have the same course content and teaching schedule.

4) Sixth to tenth weeks

The course content from the sixth to the tenth weeks was different from the course content from second tofifth week. Both groups were required to repeat the learning tasks they performed from second tofifth week. In the tenth week, both groups engaged in the second time of work review and portfolio assessment, and the questionnaire for SRL was administered as the posttest.

Table 2

Bartlett’s test of sphericity and KMO for SRL questionnaire.

Test Bartlett’s test of sphericity KMO Chi-square p

Pretest 3667.305 0.000 0.729

Posttest 3777.478 0.000 0.756

Table 3

Eigenvalues and explained variance for factor analysis of pretest and posttest on SRL.

Aspect Sub-aspect Pretest Posttest

Eigenvalues Explained variance Eigenvalues Explained variance Learning motivation Self-efficacy 6.026 13.392% 8.402 18.671%

Subject value 4.335 9.634% 3.892 8.649% Learning anxiety 3.337 7.416% 3.536 7.857%

Self-observation 4.492 9.982% 3.364 7.476%

Self-judgment Peer model 3.869 8.597% 4.165 9.255% Teacher criteria 3.100 6.888% 3.901 8.670% Self-set goal 3.098 6.884% 2.782 6.183% Self-reaction Adaptive 3.467 7.704% 3.049 6.777

Defensive 4.178 9.284% 3.047 6.771

(8)

2.6. Instrument 2.6.1. SRL questionnaire

Wu’s (2005)SRL questionnaire was employed in the present study. His questionnaire was developed based on the framework of SRL proposed by Bandura (1986),Pintrich, Smith, Garcia, and McKeachie (1991,1993),Schunk (2005), and Zimmerman (2002). Most SRL questionnaires, such as Motivated Strategies for Learning Questionnaire (MSLQ) of Pintrich et al., included two main aspects, which were learning motivation and learning strategy.Zimmerman’s (2002)self-regulated learning included motivation, self-control, self-observation, self-judgment and self-reaction process.Schunk’s (2005)self-regulated learning included self-observation, self-judgment and self-reaction process. In addition to learning motivation, the questionnaire in the present study included observation, judgment, and self-reaction. The validity and reliability of the questionnaire were analyzed by factor analysis. In Wu’s experimental study, the reliability for each aspect was greater than 0.7, which was acceptable. The questionnaire contained four aspects, which were learning motivation (sub-aspects: self-efficacy, subject value and learning anxiety), self-observation, self-judgment (sub-aspects: peer model, criteria and self-set goal) and self-reaction (sub-aspect: adaptive and defensive). There were a total of 50 items in the questionnaire. The participants were required to rate themselves on a 7-point Likert-type scale with response options from 1 (extremely disagree) to 7 (extremely agree).

1) Factor analysis

As shown inTable 2, the result showed that Bartlett’s test of sphericity was significant, which implied that there was a common factor. Kaiser–Meyer–Olkin (KMO) measure of sampling adequacy for pretest and posttest were greater than 0.7, meaning that it was appropriate for proceeding of a factor analysis (Gravetter & Wallnau, 2008;Kaiser, 1974). Principal components analysis (Harman, 1976) with oblique rotation was conducted for the factor analysis because relationships among factors were assumed to exist (Reise, Waller, & Comrey, 2000). For thefirst factor analysis, the result showed that the factor loading for item 9 in the pretest was smaller than 0.3, which should be deleted (Ford, MacCllum, & Tait, 1986). After item 9 was deleted, the result for the second factor analysis revealed that factor loading for each item was greater than 0.5, therefore no item was deleted (Hair, Black, Babin, & Anderson, 2010), as shown inTable 3. The eigenvalue of each aspect was greater than 1, hence four aspects and sub-aspects could be established (Kaiser, 1974;Tucker, Koopman, & Linn, 1969). The total explained variance was close to 80% implying that the validity of the questionnaire was good enough (Hair et al., 2010).

Table 4

Item analysis and reliability for satisfaction toward goal-setting mechanisms.

Item Discrimination Test of homogeneity

t Correlation coefficients Cronbach’sa

Ease of use 1 4.189** 0.567** 0.832 3 4.531*** 0.648** 5 5.664*** 0.712** 7 5.359*** 0.706** 9 5.664*** 0.796** Usefulness 2 3.073** 0.683** 0.827 4 3.591** 0.559** 6 4.973*** 0.719** 8 5.664*** 0.725** 10 5.132*** 0.801** **p< 0.01, ***p < 0.001. Table 5

Satisfaction on ease of use and usefulness toward goal-setting mechanisms.

em Percentage Mean t Sig.

1 2 3 4 5

Ease of use

1. Writing learning goals is convenient. 0 5.7 28.6 42.9 22.9 3.83 5.720 0.000*** 2. Revising and editing learning goals are convenient. 0 8.6 28.6 42.9 20.0 3.74 4.961 0.000*** 3. Observing classmates’ learning goals is convenient. 2.9 0 17.1 34.3 45.7 4.20 7.609 0.000*** 4. Providing feedback on classmates’ learning goals is convenient. 0 11.4 28.6 40.0 20.0 3.69 4.352 0.000*** 5. Assessing classmates’ learning goals is convenient. 0 8.6 31.4 40.0 20.0 3.71 4.730 0.000***

Total 0.6 6.9 26.9 40.0 25.8 3.8343 7.168 0.000***

Usefulness

6. The system is helpful to writing learning goals. 2.9 11.4 31.4 40.0 14.3 3.51 3.100 0.004** 7. The system is helpful to revising and editing. 0 2.9 40.0 42.9 14.3 3.69 5.351 0.000*** 8. The system is helpful to observing classmates’ learning goals. 2.9 8.6 17.1 28.6 42.9 4.00 5.323 0.000*** 9. The system is helpful to giving feedback on classmates’ learning goals. 0 8.6 40.0 31.4 20.0 3.63 4.085 0.000*** 10. The system is helpful to assessing classmates’ learning goals. 0 2.9 28.6 40.0 28.6 3.94 6.655 0.000***

Total 1.1 6.9 31.4 36.6 24.0 3.75 6.317 0.000***

Overall 0.9 6.9 29.1 38.3 24.9 3.79 6.996 0.000***

(9)

2) Analysis of reliability

The reliability coefficients of the aspects and sub-aspects were higher than 0.86, as measured by Cronbach’s

a

, suggesting that the items had relatively high internal consistency. The reliability coefficients of the pretest and posttest were 0.906 and 0.912, respectively, as measured by Cronbach’s

a

, suggesting that the questionnaire had a relatively high reliability (Nunnally, 1978).

2.6.2. Satisfaction questionnaire

First, the satisfaction questionnaire in the present study was developed based on the writing, editing, review, feedback, and assessment of goal-setting mechanisms and was reviewed by the researcher and experts several times. Consequent, the questionnaire was revised after being subsequently reviewed by the class teacher and responded by three students. There were two parts in the questionnaire. Thefirst part was ease of use of goal-setting mechanisms, withfive items. The higher the score, the more easier the goal-setting mechanisms would be perceived by a student. The second part was usefulness of goal-setting mechanisms, withfive items. The higher the score, the more helpful the goal-setting mechanisms would be perceived by a student.Table 5in theResults and discussionsection shows the detailed content of each item.

Item analysis was performedfirst. The overall score of each student questionnaire was divided into high and low score group (Kelley, 1939). An Independent Samples t-test was conducted to compare the differences in the score of each item between high and low score groups, as shown inTable 4. All the items obtained a significant t value, revealing that each item had an acceptable discrimination index. The relationship between the score of each item and the overall score of the questionnaire were significant, presenting a consistence between each item and the overall questionnaire. The reliability coefficients of ease of use and usefulness were higher than 0.80, as measured by Cronbach’s

a

, suggesting that the items had relatively high internal consistency (Nunnally, 1978).

3. Results and discussion

3.1. Students’ satisfaction toward online goal-setting mechanisms

According toTable 5, for the aspect of ease of use toward goal-setting mechanisms, the greatest percentage of students marked was on rank 4.0 for writing (item 1), editing (item 2), feedback (item 4) and assessment (item 5). For the aspect of usefulness, the greatest per-centage of students marked was on rank 4.0 for writing (item 6), editing (item 7) and assessment (item 10), and the perper-centage of students was al higher than 40%. Besides, the scores that the greatest percentage of students marked for ease of use of review (item 3) and usefulness of review (item 8) were close to 5.0, and the percentage of students was 42%.

Among the mean scores on ease of use, review (item 3) got the highest mean score followed by writing (item 1), and feedback got the lowest mean score. Among the mean scores on usefulness, review (item 3) got the highest mean score followed by assessment (item 10), and writing got the lowest mean score. The result of the Single Samples t-test revealed that the differences in each item between means and the median (¼3.5 points) for both ease of use and usefulness were significant.

According toTable 6, the differences in each aspect between the overall median (¼3.0 points) and means on ease of use, usefulness and the overall online goal-setting mechanisms were significant. Among the mean scores of the two aspects, ease of use had a higher mean score than usefulness, implying that the system’s ease of use were greater than usefulness.

According toTable 7, the review mechanism got the highest satisfaction. About 51.4% of students marked 5.0 on review, and more than 42% of students marked 4.0 points on writing, editing, feedback, and assessment. The order (from great to small) for mean scores on satisfaction of each mechanism was review, assessment, editing, writing, and feedback. The result of the Single Samples t-test revealed that the difference between mean of each mechanism and the median (¼3.0) was significant.

Based on the results about the satisfaction toward goal-setting mechanisms mentioned above, the experimental group held a signi fi-cantly positive attitude toward ease of use and usefulness of goal-setting mechanisms. This implied that the system was appropriate to be

Table 6

Satisfaction toward goal-setting mechanisms based on cross aspects.

Aspects Mean SD t Sig.

Ease of use 3.83 0.69 7.168 0.000***

Usefulness 3.75 0.71 6.317 0.000***

Overall 3.79 0.67 6.996 0.000***

***p< 0.001.

Table 7

Satisfaction toward goal-setting mechanisms.

Goal-setting mechanisms Percentage Mean t Sig.

1 2 3 4 5 Writing (item 1, 6) 0 5.7 22.9 48.6 22.9 3.67 5.058 0.000*** Editing (item 2, 7) 0 2.9 22.9 54.3 20.0 3.71 6.040 0.000*** Review (item 3, 8) 2.9 0 14.3 31.4 51.4 4.10 6.826 0.000*** Feedback (item 4, 9) 0 8.6 20.0 51.4 20.0 3.66 4.456 0.000*** Assessment (item 5, 10) 0 2.9 22.9 42.9 31.4 3.83 6.030 0.000*** Overall 0.6 4.0 20.6 45.7 29.1 3.79 6.996 0.000*** ***p< 0.001.

(10)

applied in the present experimental study. Among thefive mechanisms, the review mechanism got the highest score on satisfaction for ease of use, usefulness and overall.

3.2. Effects of satisfaction toward goal-setting mechanisms in WBPA on SRL

Students in the experimental group were grouped into a high satisfaction group and a low satisfaction group, according to their satisfaction toward the overall goal-setting mechanism. The t-test was employed to compare the differences on SRL posttest between high satisfactions and low satisfactions. As shown inTable 8, the difference on the overall SRL between the two groups was significant (t ¼ 3.12; p< 0.005), meaning that high satisfactions significantly outperformed low satisfactions in overall SRL. This result implied that goal-setting mechanisms in the WBPAS had positive effects on overall SRL. Among aspects, learning motivation and self-judgment were significant. Among sub-aspects, self-efficacy, subject values and self-judgment comparing with self-set goal were also significant. High satisfactions with these aspect and sub-aspects significantly outperformed those of low satisfactions, implying that goal-setting mechanisms in the WBPAS had positive effects on these aspects.

3.3. Effects of goal setting by WBPAS or by paper-based portfolio on SRL

As shown inTable 9, Levene’s test of equality of variance was insignificant (p > 0.05), meaning that the variance of SRL was equal across groups and the homogeneity assumption was sustained. Furthermore, homogeneity of regression slope appeared to be insignificant (p> 0.05), suggesting that the homogeneity assumption was sustained, and analysis of covariance (ANCOVA) could be performed. Addi-tional, there was no significant difference in the SRL pretest between the two groups by using the t-test, implying the two groups were equal in initial SRL.

Table 8

t-Test for effects of satisfaction toward overall goal-setting mechanism on SRL posttest.

Aspect Mean (SD) t Sig. Effect size

High satisfactions Low satisfactions

Learning motivation 82.76 (12.45) 76.15 (10.06) 2.18 0.042* 0.101 Self-efficacy 41.27 (7.12) 35.70 (8.82) 2.061 0.047* 0.111 Subject value 28.61 (5.64) 24.00 (5.71) 2.401 0.022* 0.145 Learning anxiety 19.88 (7.99) 17.7 (7.07) 0.854 0.400 0.021 Self-observation 26.27(4.37) 22.76(5.91) 2.006 0.053 0.106 Self-judgment 57.05 (11.45) 45.29 (10.74) 2.330 0.026* 0.138 Peer model 17.61 (7.29) 14.11 (5.78) 1.564 0.127 0.067 Teacher criteria 19.00 (3.71) 16.52 (3.57) 2.005 0.053 0.106 Self-set goal 20.44 (3.39) 17.64 (3.53) 2.387 0.001** 0.144 Self-reaction 48.66 (8.99) 46.35 (9.44) 0.742 0.463 0.016 Adaptive 20.44 (5.62) 17.82 (6.33) 1.295 0.204 0.047 Defensive 12.77 (5.98) 14.29 (6.73) 0.705 0.486 0.014 Overall SRL 205.33 (26.71) 178.94 (23.05) 3.120 0.004** 0.223 *p< 0.05, **p < 0.01. Table 9

Homogeneity of variance and regression slope for both groups.

Aspect Levene’s test Homogeneity of regression slope

Experimental group (before regulation)

Control group (after regulation) F Sig. F Sig. M SD M SD Learning motivation 0.863 0.361 1.307 0.278 83.77 13.46 75.83 10.13 Self-efficacy 0.429 0.515 0.312 0.733 37.43 4.66 33.79 5.01 Subject value 1.100 0.298 1.753 0.182 25.90 0.88 22.85 0.88 Learning anxiety 0.367 0.547 2.243 0.114 19.87 1.06 19.18 1.08 Self-observation 0.694 0.408 0.220 0.803 24.59 1.06 23.30 1.03 Self-judgment 0.003 0.957 0.462 0.632 53.11 1.44 48.12 1.72 Peer model 0.003 0.973 0.933 0.399 16.21 0.82 15.19 0.91 Teacher criteria 2.677 0.106 0.237 0.790 17.39 0.60 15.43 0.68 Self-set goal 0.566 0.454 0.485 0.618 18.88 0.55 18.67 0.62 Self-reaction 0.042 0.839 0.887 0.417 31.17 1.09 30.01 1.09 Adaptive 1.417 0.238 0.608 0.548 18.48 0.89 17.64 0.94 Defensive 1.462 0.231 0.083 0.921 12.93 1.16 12.71 1.38 Overall SRL 1.290 0.260 0.130 0.878 190.99 4.66 177.51 5.019

(11)

As shown inTables 9and10, both groups were significantly different in learning motivation, self-efficacy and subject value, and the experimental group had significantly higher scores in these three aspects than the control group, meaning that the effect of WBPAS on SRL was greater than the effect of paper-based portfolio.

Table 10shows that there was no significant difference in self-observation between the two groups. This implied that the WBPAS had no significant effect on self-observation. Both groups were significantly different in self-judgment (F ¼ 4.371, p < 0.05), and the experimental group had significantly higher scores in overall self-judgment and self-judgment comparing with teacher criteria than the control group did, meaning that the effect of WBPAS on self-judgment was greater than the effect of paper-based portfolio. There was no significant difference in self-reaction between the two groups, which revealed that the WBPAS had no significant effect on self-reaction.

Both groups were significantly different in overall SRL (F ¼ 7.025, p < 0.05), revealing that students learning with the WBPAS significantly outperformed students learning with paper-based portfolio in SRL.

4. Discussions

According to the study results, students who were highly satisfied with goal-setting mechanisms had a significantly better performance in overall SRL than students who were less satisfied. This revealed that the better the online goal-setting mechanisms in a WBPAS, the better the SRL. This also implied that the online goal-setting mechanisms had a significantly positive effect on SRL. Besides, students using WBPAS

Table 10

The difference in SRL between both groups using ANCOVA.

Aspect Source of variation F Sig. Effect size

Learning motivation Academic grade 0.209 0.649 0.003

Pretest 12.699 0.001 0.161

Group 7.810 0.007** 0.106

Self-efficacy Academic grade 0.411 0.524 0.006

Pretest 8.836 0.004 0.118

Group 4.192 0.045* 0.060

Subject value Academic grade 0.351 0.556 0.005

Pretest 30.463 0.000 0.316

Group 7.837 0.007** 0.106

Learning anxiety Academic grade 1.583 0.213 0.023

Pretest 21.586 0.000 0.246

Group 0.696 0.407 0.010

Self-observation Academic grade 0.782 0.380 0.012

Pretest 15.330 0.000 0.188

Group 1.192 0.279 0.018

Self-judgment Academic grade 0.532 0.468 0.008

Pretest 80.315 0.000 0.549

Group 4.371 0.040* 0.062

Peer model Academic grade 0.124 0.725 0.002

Pretest 67.463 0.000 0.505

Group 0.330 0.567 0.005

Teacher criteria Academic grade 1.711 0.195 0.025

Pretest 21.242 0.000 0.243

Group 5.787 0.019* 0.081

Self-set goal Academic grade 0.255 0.615 0.004

Pretest 38.350 0.000 0.368

Group 0.793 0.377 0.012

Self-reaction Academic grade 6.738 0.012 0.093

Pretest 1.881 0.175 0.028

Group 0.173 0.679 0.003

Adaptive Academic grade 5.291 0.025 0.074

Pretest 4.147 0.046 0.059

Group 0.686 0.411 0.010

Defensive Academic grade 0.373 0.543 0.006

Pretest 9.535 0.003 0.126

Group 0.698 0.406 0.010

Overall SRL Academic grade 0.301 0.585 0.005

Pretest 25.560 0.003 0.279

Group 7.025 0.010* 0.096

(12)

to set goals outperformed students using paper-based portfolio to set goals in overall SRL and several related aspects. This showed that the online goal-setting mechanisms in the WBPAS were more effective. These two study results were consistent, which also double confirmed that online goal-setting mechanisms had a positive effect on SRL.

The study results were consistent with the study results byArsal (2010)andWang (2011). The instruments employed by the present study and the studies of Arsal and Wang belonged to the formative assessment. Arsal asked pre-service science teachers to use a diary for 14 weeks, and the result showed that teachers using a diary outperformed teachers without using a diary in SRL. The study by Wang revealed that students using web-based formative assessment outperformed students using traditional assessment in SRL. However, the educational intervention in Wang’s study was Peer-Driven Assessment Module of the Web-based Assessment and Test Analysis System (PDAM-WATAS), and participants were seventh graders, which were different from the intervention and participants in the present study. AsWinters, Greene, and Costich’s (2008)study revealed that computer-based learning environment facilitated SRL.Azevedo (2005)andKollar and Fischer (2006)argued that interactive digital environment was beneficial to the development of SRL.Carneiro et al. (2011)also pointed out that technology enhanced learning environment (TELE) was helpful to SRL. The present study and these studies significantly have the educational implication of digital technology applications in teaching and learning.

The experimental group held significantly positive attitude toward learning motivation, self-efficacy, and subject values than the control group, which confirmed the study results done byShin and Alexander (2000)andArsal (2010). Both studies done byShin and Alexander (2000)andArsal (2010)showed that WBPAS enhanced students’ learning motivation more than paper-based portfolio did and were helpful to students’ learning self-efficacy and subject value. However, these two studies were different in participants’ educational level and intervention.Shin and Alexander (2000)focused on interventions of goal setting and feedback, but not through the Internet and Web, and participants were children.Arsal (2010)emphasized intervention of diary, but not via the Internet and Web, and participants were pre-service teachers. Neither study utilized the portfolio and did not examine the effects of the interventions on the other three aspects of SRL of the present study. There were also studies confirming that learning motivation and performance could be facilitated and enhanced by SRL with a technology enhanced environment (Greene & Azevedo, 2007;Greene, Costa, Robertson, Pan, & Deekens, 2010;Kramarski & Gutman, 2006;Santhanam, Sasidharan, & Webster, 2008;Wang, 2011). Therefore, technology-assisted goal setting plays an important role in enhancing learning performance.

Winne and Hadwin (2008)stated that change of motivation is resulted by SRL. On the other hand, SRL is also resulted from change of motivation. This showed that the change of motivation and self-learning regulated learning were a cycle of cause and effect. The features of SRL cycle can change learning motivation or other regulated elements, likewise, change of motivation can facilitate students to regulate learning goals and direction. According to the present study results and the theory proposed byWinne and Hadwin (2008), goal setting regulated learning motivation, and regulation of learning motivation changed the direction of learning goals, therefore, learning goal and learning motivation had a mutual causal relationship.Chow (2009)argued that regulation of motivation was probably affected by self-efficacy, interest or learning goals. Motivation and self-efficacy are all the aspects of SRL. Therefore, not only goal setting affected moti-vation, there were also mutual causal relationships among the aspects of SRL, such as the relationship between motivation and self-efficacy. 5. Conclusion and implication

Regarding the revision of the WBPAS, the feedback mechanism for goal setting ranked the lowest in terms of student satisfaction, so further improvement was required. Besides, auto assessment, statistics of feedback and peer assessment mechanisms could be enhanced for a better goal-setting mechanism. Among the goal-setting mechanisms in the system, the peer model mechanism ranked the highest in terms of student satisfaction, revealing that students believed that peer model was beneficial to students in goal setting. Although both groups were significantly different in self-judgment, they were significantly different in self-judgment comparing with teacher criteria and self-set goals but not comparing with peer model. Therefore, the peer model, review and feedback mechanism was required to be enhanced more for facilitating peer model in self-judgment.

The goal-setting mechanisms in the WBPAS were appropriate to curricula that require students to submit computerized works. Hence, in order to avoid a bias generalization, curricular property, instructional situation, participants and students’ background should be taken into consideration, if the study results were applied.Dabbagh and Kitsantas (2011)proposed that a personal learning environment established by social media can facilitate students’ SRL.Carneiro et al. (2011)also mentioned that a blog is probably more helpful to SRL than e-portfolio because a blog not only possesses the features of diary but also provides a chance to students to share and give feedback. With these theories, blog-based portfolio or micro-blog-based portfolio can be employed as an educational intervention in the future studies for facilitating SRL behavior.

References

Abrami, P. C., Wade, C. A., Pillay, V., Aslan, O., Bures, E. M., & Bentley, C. (2008). Encouraging self-regulated learning through electronic portfolios. Canadian Journal of Learning and Technology, 34(3), 93–117.

Arsal, Z. (2010). The effects of diaries on self-regulation strategies of preservice science teachers. International Journal of Environmental & Science Education, 5(1), 85–103.

Azevedo, R. (2005). Using hypermedia as a metacognitive tool for enhancing student learning? The role of self-regulated learning. Educational Psychologist, 40(4), 199–209.

Bandura. (1986). Social foundations of thought and action: A social cognitive theory. Englewood Cliffs, NJ: Prentice-Hall.

Carneiro, B., Lefrere, P., Steffens, K., & Underwood, J. (Eds.), (2011). Self-regulated learning in technology enhanced learning environments: A European perspective. Rotterdam, Netherlands: Sense Publishers.

Chang, C. C., Tseng, K. H., Chou, P. N., & Chan, Y. H. (2011). Reliability and validity of web-based portfolio peer assessment: a case study for a senior high school’s students taking computer course. Computers & Education, 57(1), 1306–1316.

Chow, B. (2009). Literature review: clarifying the constructs of motivation, regulation of motivation, and volition in models of self-regulated learning. Transformative Di-alogues: Teaching & Learning Journal, 3(2), 1–11.

Dabbagh, N., & Kitsantas, A. (2011). Personal learning environments, social media, and self-regulated learning: a natural formula for connecting formal and informal learning. Internet and Higher Education, 15(1), 3–8.

Ford, J. K., MacCllum, R. C., & Tait, M. (1986). The application of exploratory factor analysis in applied psychology: a critical review and analysis. Personnel Psychology, 39, 291–314.

(13)

Greene, J. A., & Azevedo, R. (2007). Adolescents’ use of self-regulatory processes and their relation to qualitative mental model shifts while using hypermedia. Journal of Educational Computing Research, 36(2), 125–148.

Greene, J. A., Costa, L. J., Robertson, j., Pan, Y., & Deekens, V. M. (2010). Exploring relations among college students’ prior knowledge, implicit theories of intelligence, and self-regulated learning in a hypermedia environment. Computers & Education, 55(3), 1027–1043.

Hair, J. F., Black, W. C., Babin, B. J., & Anderson, R. E. (2010). Multivariate data analysis: A global perspective. New Jersey, USA: Pearson Education.

Harman, H. H. (1976). Modern factor analysis (3rd ed.). Chicago, IL: University of Chicago Press.

Heo, H. (2000). Theoretical underpinnings for structuring the classroom as self-regulated learning environment. Educational Technology International, 2(1), 31–50.

Hwang, G. J., Chu, H. C., Chen, T. L., Wang, T. T., Tseng, J. C. R., & Hwang, G. H. (2007). The development of a computer-assisted self-regulation system on the internet. Chinese Journal of Science Education, 15(3), 317–334.

Kaiser, H. F. (1974). Little Jiffy, Mark IV. Educational and Psychological Measurement, 34, 111–117.

Kelley, T. L. (1939). The selection of upper and lower groups for the validation of test items. Journal of Educational Psychology, 30(1), 17–24.

Kollar, I., & Fischer, F. (2006). Supporting self-regulated learners for a while and what computers can contributes. Journal of Educational Computing Research, 35(4), 425–435.

Kramarski, B., & Gutman, M. (2006). How can self-regulated learning be supported in mathematical e-learning environments? Journal of Computer Assisted Learning, 22, 24–33.

Latham, G. P., & Locke, E. A. (1991). Self-regulation through goal setting. Organization Behavior and Human Decision Process, 50, 212–247. Lifetick. (2011). Lifetick: online goal setting. Retrieved 20.07.13, fromhttp://lifetick.com/.

Lougheed, P., Bogyo, B., Brokenshire, D., & Kumar, V. (2005). Formalizing electronic portfolios in the SPARC ePortfolio tool. Paper presented at applications of semantic web technologies for e-learning workshop at the third international conference on knowledge capture. Eindhoven, The Netherlands.

Neber, H., & Schommer-Aikins, M. (2002). Self-regulated science learning with highly gifted students: the role of cognitive, motivational, epistemological, and environmental variables. High Ability Studies, 13(1), 59–74.

Nunnally, J. C. (1978). Psychometric theory. New York: McGraw-Hill.

Pintrich, P. R., Smith, D. A. F., Garcia, T., & McKeachie, W. J. (1991). The motivated strategies for learning questionnaire (MSLQ). Ann Arbor, MI: The University of Michigan.

Pintrich, P. R., Smith, D. A. F., Garcia, T., & McKeachie, W. J. (1993). Reliability and predicative validity of the motivated strategies for learning questionnaire (MSLQ). Educational and Psychological Measurement, 53(3), 801–813.

Reise, S. P., Waller, N. G., & Comery, A. L. (2000). Factor analysis and scale revision. Psychological Assessment, 12(3), 287–297.

Riedinger, B. (2004). Using the ePortfolio for advising,first-year programs, and writing assessment. The Connecticut Distance Learning Consortium. Retrieved 25.08.11, from

http://www.eportfolio.org/references/Advising3.pdf.

Santhanam, R., Sasidharan, S., & Webster, J. (2008). Using self-regulatory learning to enhance e-learning based on information technology training. Information Systems Research, 19(1), 26–47.

Schmitz, B., & Wiese, B. (2006). New perspectives for the evaluation of training session in self-regulated learning: time-series analyses of diary data. Contemporary Educational Psychology, 31(1), 83–88.

Schunk, S. H. (2005). Self-regulated learning: the educational legacy of Paul R. Pintrich. Educational Psychologist, 40, 85–94.

Shin, S. S., & Alexander, J. M. (2000). Interacting effects of goal setting and self or other referenced feedback on children’s development of self-efficacy and cognitive skill within the Taiwanese classroom. Journal of Educational Psychology, 92(3), 536–543.

Tucker, R. F., Koopman, R. F., & Linn, R. L. (1969). Evaluation of factor analytic research procedures by means of simulated correlation matrices. Psychometrika, 34, 421–459.

Wang, T. H. (2011). Developing web-based assessment strategies for facilitating junior high school students to perform self-regulated learning in an e-learning environment. Computers & Education, 57(4), 1801–1812.

Winne, P. H. (2005). Key issues in modeling and applying research on self-regulated learning. Applied Psychology: An International Review, 54(2), 232–238.

Winne, P. H., & Hadwin, A. F. (2008). The weave of motivation and self-regulated learning. In D. H. Schunk, & B. J. Zimmerman (Eds.), Motivation and self-regulated learning: Theory, research, and applications (pp. 297–314). Mahwah, NJ: Lawrence Erlbaum Associates.

Winters, F. I., Greene, J. A., & Costich, C. M. (2008). Self-regulation of learning within computer-based learning environments: a critical analysis. Educational Psychology Review, 20(4), 429–444.

Wu, P. H. (2005). Examining the effects of self-regulated learning on junior high school students’ academic performance using structural equation modeling. Unpublished Master’s thesis. National Taiwan University of Science and Technology: Tapei.

Zimmerman, B. J. (2002). Becoming a self-regulated learner: an overview. Theory into Practice, 41(2), 64–70.

Zimmerman, B. J. (2008). Goal setting: a key proactive source of academic self-regulation. In D. H. Schunk, & B. J. Zimmerman (Eds.), Motivation and self-regulated learning: Theory, research, and applications (pp. 267–295). Mahwah, NJ: Lawrence Erlbaum Associates Publishers.

數據

Fig. 2. Online learning goal review mechanism. Fig. 1. Online learning goal writing and editing.
Fig. 3. Online learning goal feedback mechanism.
Fig. 5. Online learning goal peer assessment mechanism.
Table 10 shows that there was no signi ficant difference in self-observation between the two groups

參考文獻

相關文件

The teacher needs to plant the seed for ideas by describing a conflict before asking the students to start writing the acrostic script.. Once the students have read and understood the

The teacher-to-class (T/C) ratio for public sector primary and secondary schools (including special schools) has been increased by 0.1 across-the-board starting from the

(ii) “The dismissal of any teacher who is employed in the school – (a) to occupy a teacher post in the establishment of staff provided for in the code of aid for primary

(ii) “The dismissal of any teacher who is employed in the school – (a) to occupy a teacher post in the establishment of staff provided for in the code of aid for primary

After teaching the use and importance of rhyme and rhythm in chants, an English teacher designs a choice board for students to create a new verse about transport based on the chant

In addition, based on the information available, to meet the demand for school places in Central Allocation of POA 2022, the provisional number of students allocated to each class

This study aims at evaluating the learning effects on the part of students from the teacher-designed concrete vector classroom-teaching approach in Plane Vector course on the basis

Please liaise with the officer in your school who are responsible for the Class and Subject Details Survey for using of the same class names in both the Class and Subject