The study of self-assessment with prompts, learning journal and referencing
through sharing for regulation of cognition and their effect on web-based
programming learning
Rustam Shadiev, Wu-Yuin Hwang1, Jung-Lung Hsu2, Chia-Ling Chang1, Ray Yueh-Min Huang Department of Engineering Science, National Cheng Kung University, Taiwan
rustamsh@gmail.com, huang@mail.ncku.edu.tw
1Graduate Institute of Network Learning Technology, National Central University, Taiwan wyhwang@cc.ncu.edu.tw, trcy555@gmail.com
2Department of Information Management, Kainan University, Taiwan srl666@gmail.com
Abstract— In this study we proposed self-assessment with
prompts, learning journal and referencing through sharing learning methods for regulation of cognition with one annotation system called VPen. We also aimed to investigate the effectiveness of the learning methods on web-based programming learning performance and achievement. The results of this study revealed students perceived the learning methods are useful for the strategies of regulation of cognition. We also found that self-assessment with prompts, learning journal and referencing through sharing of learning performance and behavior significantly correlate with each other and learning achievement. In addition, the results revealed learning journal brings more effect on learning performance and achievement than self-assessment with prompts.
Keywords- self-assessment with prompts, learning journal, referencing through sharing, perceptions, strategies of regulation of cognition, learning performance and achievement, VPen
I. INTRODUCTION
According to Akyol and Garrison [1], Schraw and Dennison [15], and Sperling et al., [16], regulation of cognition is one important component of metacognition. It was defined as abilities of students to plan, monitor and regulate their learning activities. Reference [15] argued that regulation of cognition includes five skills that facilitate control aspect of learning: a) Planning is for planning, goal setting, and allocating resources prior to learning, b) Information management strategies are skills and strategy sequences used to process information more efficiently, c) Comprehension monitoring is for assessment of one's learning or strategy use, d) Debugging strategies are used to correct comprehension and performance errors, and e) Evaluation is for analysis of performance and strategy effectiveness after a learning episode.
Recent evidence showed significant improvement in learning when students used strategies of regulation of cognition [1]. Reference [15] suggested that regulation of cognition improves performance in a number of ways, including better use of attentional resources, better use of existing strategies, and a greater awareness of
comprehension breakdowns. Reference [16] claimed students with regulation of cognition ability are more strategic and perform better than students without it.
Self-assessment was defined as the process by which a student makes self-directed subjective judgment about his/her ability or confidence in a particular skill or knowledge for the purpose of ongoing self-improvement [4], [11]. Reference [4] argued that self-assessment can increase students' engagement in learning and student achievement. According to Milford and L. Brown [11] and Nicol [13], self-assessment allows students' reflection and has a significant main effect for grade. Moreover, self-assessment may enhance students’ knowledge structure and increase metacognitive skills.
A learning journal is a collection of notes, observations, thoughts related to learning experience [3], [5]. For example, students may think about what did they learn, what concept did not they understand, why did not they understand it, how to overcome this issue and then record thoughts in a learning journal. A learning journal is generally reflective and accumulative as it focuses on ongoing issues over time and there is some intention to learn from either the process of doing it or from the results of it [12]. According to Berthold, Nückles, and Renkl [3] and Cowan [5], a learning journal has great potential to facilitate learning and metacognitive strategies. Reference [12] reported students who have used a learning journal showed more sophisticated conceptions of learning, used more metacognitive strategies during a learning task, and performed significantly better on the final exam.
Sharing notes stimulates informal knowledge related to a concept [6]. Students may effectively collaborate and learn from each other through sharing and exchanging their ideas with peers about the same accessing learning material. Students who reference to work of more knowledgeable and experienced peer have the opportunity to use the reference in making adjustments to own work [8], [9], and [10]. Reference [3] argued students refer to peers' work in order to improve their own. As a result, students who could get assistance perform significantly better comparing to those students who couldn't get it.
programming problem solving [9], [17], which enables practicing programming skills over and over for the purpose of improving or mastering it. References [9] and [10] suggested engaging students who learn programming language online into writing source codes, execution of a program, debugging practice and feedback activities so that students can solve programming problem efficiently. For the same purpose they recommended organizing programming problems or prompt-questions in proper sequence, from simple to complex. In web-based programming learning courses, practice is important for improving students’ learning [9], [10]. Students should be given enough practice opportunities in an environment where they can receive constructive and corrective feedback and if students practice frequently their programming skills may improve [17].
Previous studies on metacognition have acknowledged that the strategies of regulation of cognition are beneficial for learning [1], [15], [16]. However, not much attention was paid on whether these strategies can be facilitated by self-assessment with prompts, learning journal and referencing through sharing learning methods, particularly in web-based programming learning environment. Moreover, the effectiveness of the learning methods on web-based programming learning performance and achievement is still not well explored issue. Therefore, this study aimed to investigate students' perceptions regarding usefulness of self-assessment with prompts, learning journal and referencing through sharing for the strategies of regulation of cognition. Then this study aimed to explore the relationship between self-assessment with prompts, learning journal and referencing through sharing of learning performance and behavior and their effect on learning achievement. Two primarily research questions were addressed in this study:
- What are students' perceptions regarding usefulness of self-assessment with prompts, learning journal and referencing through sharing for information management, comprehension monitoring and debugging strategies?
- Do self-assessment with prompts, learning journal and referencing through sharing of learning performance and behavior have relationship with each other and what is their effect on learning achievement?
II. METHODS
A. Participants and the subject
Sixty eight undergraduate freshmen students majoring in foreign language participated in this study. Age of students ranged from 18 to 19 years, yet most of students were female. Four-month course entitled Visual Basic programming design was administered to improve students' concept about programming design and logic programming. Learning material was uploaded online and contained nine chapters.
B. The VPen system
The web-based annotation system VPen was developed for supporting programming language [9], [10] and math [7] learning through creating and sharing annotations. In this study the VPen system (see Figures 1 and 2) was employed
for supporting learning methods such as self-assessment with prompts, learning journal and referencing through sharing.
Self-assessment with prompts (Figure 1) was available for each chapter of learning material; students used prompt-questions to assess the level of their understanding of a chapter after reading it. Prompt-questions can be accessed by clicking on "Self-assessment time" link on left-upper part of the system (Figure 1). Each chapter contained six questions and each question represented one level of cognition [2]. The level increased in complexity as the learner moved through the questions. For example, the first question in Figure 2 "What kind of program can you write?" represents the lowest level of cognition and the sixth question "Can you write complete program using concept you learn from this chapter?" represents the highest level. Students were invited to create an annotation using the VPen system to answer a question (Figure 1).
Figure 1. Self-assessment with prompts
Figure 2. Learning journal
less attention. One learning journal was formed for one chapter of learning material and it could be accessed by clicking on "Learning journal" link on upper-left corner of the system.
Referencing through sharing allowed students to share their knowledge and experience (i.e., answers to prompt-questions and learning journal) regarding learning material with peers. Students studied peers' shared knowledge to identify missed important concepts and/or expand their own answers.
C. Experiment procedure
Two-hour weekly face-to-face classes were conducted by one instructor in form of lectures in a computer classroom. Students have studied learning material related to the course after class in the computer classroom or at home. We proposed self-assessment with prompts, learning journal and referencing through sharing learning methods with the VPen system for students to facilitate their strategies of regulation of cognition and programming learning. A post-test, a questionnaire survey and interviews with students were administered at the last class.
D. Data collection
A questionnaire survey was designed following general recommendations of [15]. In this study we focused on information management, comprehension monitoring and debugging strategies only. That is we did not include items related to planning as the course content and learning objects were set by an instructor, neither had we included the items related to evaluation as analysis of performance and strategy effectiveness had to take place after a learning episode. Totally fifty-five answer-sheets were obtained from students to the survey. Responses to the questionnaire were scored using a five-point Likert scale, anchored by the end-points “strongly disagree” (1) and “strongly agree” (5).
Students' levels of cognition of self-assessment with prompts and learning journal were assessed using taxonomy for assessment proposed by [2]. This study adopted a concept as a coding unit and seven-point scales for assessment of cognitive level.
We calculated all annotations which represented quantity of self-assessment with prompts and quantity of learning journal respectively. Frequency of referencing represented number of times students referenced to self-assessment with prompts/learning journal of peers. Frequency of being referenced represented number of times a student's self-assessment with prompts/learning journal was referenced to. One-on-one semi-structured interviews were conducted to confirm findings obtained from our data analysis. During the interviews students were asked the open-ended questions about a) usefulness of self-assessment with prompts, learning journal and referencing through sharing for the strategies of regulation of cognition, b) students' experience with self-assessment with prompts, learning journal and referencing through sharing and c) the difference of usefulness of self-assessment with prompts and learning journal for the strategies of regulation of cognition and web-based programming learning.
III. RESULTS AND DISCUSSION
According to the results of the questionnaire survey, almost all items of information management (Mean=3.22; SD=0.83), comprehension monitoring (Mean=3.18; SD=0.86) and debugging strategies (Mean=3.35; SD=0.88) were ranked high. The results revealed that students generally agreed the learning methods were useful for information management, comprehension monitoring and debugging strategies.
During the interviews students confirmed they considered self-assessment with prompts as useful method for strategies of regulation of cognition, e.g. to test their learning progress and promote their understanding of learning material. Students agreed they could review answers to prompt-questions and know what did they learn and what need to be learnt regarding learning material. Students concurred that using self-assessment with prompts facilitated their strategies of regulation of cognition as well as their cognition. The followings are from the interviews:
- I was aware of what did I learn from my answers to prompt-questions.
- Self-assessment with prompts is useful to test my understanding and it helps promoting my cognition.
References [1], [4], [11], and [16] in their studies also reported that self-assessment is useful for metacognitive skills and increasing learning performance which confirm our finding.
Students mentioned in the interviews that learning journal is one important method for strategies of regulation of cognition. Mainly, because learning journal have given students a clear picture of what they have learned [8], which parts of a chapter they understand and which parts they do not. Students usually have reviewed their own learning journal again in case they have identified any parts of a chapter they still do not understand. Then students would create annotations to these parts of a chapter and complete learning journal. Students admitted that following the strategies of regulation of cognition with learning journal positively effected on their cognition. The followings are from the interviews:
- Reviewing learning journal helps me to know where important key concepts of learning material are.
- I can discover what parts of learning material I didn't understand from reviewing learning journal.
Similar finding were reported elsewhere, e.g., regarding benefit of creating, reviewing and modifying notes on learning performance and behavior in [7] and [8], and regarding learning journal for facilitating metacognitive strategies and learning achievement in [3] and [12].
cognition as well as their cognition. The following is from the interview:
- I prefer referring to peers' self-assessment or learning journal if I don't understand learning material in order to get better understanding of it.
- After reviewing peers' work I could understand key concepts very well!
Similar results were reported in [3], [6], and [10]. They argued that reviewing peers' work is beneficial for strategies of regulation of cognition and learning as students are able to exchange ideas, learn from each other, and make adjustments to own work.
However, a few items of the questionnaire survey were ranked as the lowest with mean value less than 2.90. The interviews with students who ranked the items low revealed these students had no time to use the learning methods; even they thought the methods are useful. Moreover, the experimental course did not relate to students major in foreign language and therefore they had low motivation to study programming and use proposed learning methods. The followings are from the interviews:
- I forgot to use the methods since this course did not relate to my major.
- Learning journals and self-assessment are useful; however, I had no time for using them.
The Pearson correlation analysis was administered to investigate the relationship between variables of self-assessment with prompts, learning journal and referencing through sharing of learning performance and behavior and what is their effect on learning achievement. According to the results (Table 1), all variables of self-assessment with prompts, learning journal and referencing through sharing of learning performance and behavior have significant correlation with each other (p<0.05). This finding may suggest that the better students perform in one independent variable the better they will perform in another. For example, if students have higher level of cognition in self-assessment with prompts than they will probably have higher level of cognition in learning journal.
TABLE I. CORRELATION STATISTICS
Variables Cogni-tive level1 Quan-tity1 Cogni-tive level2 Quan-tity2 Freq. of referenc.3 Freq. of being referenc.3 Quantity1 0.927** Cognitive level2 0.733** 0.664** Quantity2 0.652** 0.590** 0.732** Freq. of referenc.3 0.555** 0.576** 0.429** 0.332* Freq. of being referenc.3 0.501 ** 0.469** 0.470** 0.600** 0.232 Learning achie-vement 0.707** 0.581** 0.719** 0.576** 0.325* 0.418**
1 Self-assessment with prompts, 2Learning journal, 3Referencing through sharing, *p<.05, **p<.01
Moreover, the results showed that all variables of self-assessment with prompts, learning journal and referencing through sharing of learning performance and behavior have significant correlation with learning achievement (p<0.05).
This finding may suggest that the better students perform in one of the independent variables, the better is learning achievement. For example, if cognitive level of learning journal is high than probably learning achievement will also be high and so on. Similar finding were reported in [7].
The results of Pearson correlation analysis showed cognitive level of learning journal has the most significant correlation with learning achievement (0.719**). Therefore we administered the paired t-test analysis to investigate the difference in cognitive levels of self-assessment with prompts and of learning journal. The results showed that cognitive level of learning journal was significantly higher comparing to cognitive level of self-assessment with prompts (t=5.787, p<.001). Then multiple regression analysis was administered to find the predictive degree of variables of self-assessment with prompts, learning journal and referencing through sharing for learning achievement. The results are presented in Table 2.
TABLE II. MULTIPLE REGRESSION ANALYSIS
Model Predicting variables Dependent variable R2 Adjusted R2 t 1 Cognitive level of learning journal Learning achie-vement 0.517 0.508 0.719 7.537 *** 2 Cognitive level of learning journal Learning achie-vement 0.587 0.571 0.435 3.318 ** Cognitive level of self-assessment with prompts 0.388 2.961** **p<.01, ***<.001
achievement than self-assessment with prompts does. The reason is that learning journal was actively created and conducted by students on voluntarily bases, although prompt-questions of self-assessment with prompts were designed and assigned to students by an instructor. Similar findings were reported in [7].
However, the results of the interviews did not support this finding as students have given their preference to self-assessment with prompts over learning journal. According to students' opinions, prompt-questions of self-assessment were as guidelines and could scaffold learning, yet self-assessment with prompts helped them test understanding of learning material. Therefore, students perceived self-assessment with prompts could promote cognition better. References [3] and [11] also suggested a prompt is very effective learning tool for facilitating cognition and its regulation.
Based on the results of this study we recommend using self-assessment with prompts, learning journal and referencing through sharing for web-based programming learning as they facilitate the strategies of regulation of cognition and positively effect on learning performance and achievement. In addition, we suggest a few modifications for improving proposed learning methods and their applications. We suggest providing a set of guidelines in form of prompts for learning journal which may guide and scaffold learning. Prompts of self-assessment and learning journal need to be designed in a way that foster students' reflection on understanding of learning material and cover all levels of cognition so that they can be used by students of different cognitive level. Students need to be engaged into sharing and reviewing own and peers' self-assessment with prompts and learning journal actively in order to improve their own work [3], [10].
IV. CONCLUSIONS
In this study we proposed self-assessment with prompts, learning journal and referencing through sharing for facilitating the strategies of regulation of cognition and web-based programming learning with the VPen system. Self-assessment with prompts enabled students assesses their understanding of learning material. Students used learning journal to collect key concepts and reflections on learning material. Referencing through sharing allowed students share self-assessment with prompts and learning journal with peers for peer-learning purpose.
The results of this study demonstrated that students generally perceived that self-assessment with prompts, learning journal and referencing through sharing are useful learning methods for the strategies of regulation of cognition. We found that self-assessment with prompts, learning journal and referencing through sharing of learning performance and behavior are significantly correlate with each other and also with learning achievement. We also found that learning journal brings more effect on learning performance and learning achievement than self-assessment with prompts does. Students reported prompts of self-assessment are helpful for learning; therefore, we will
provide prompts for learning journal in the future to guide and scaffold learning. In the future we will also involve students with major related to the experimental course, modify learning methods and further investigate students’ perceptions regarding usefulness of learning methods for the strategies of regulation of cognition.
REFERENCES
[1] Z. Akyol and D.R. Garrison, “Assessing metacognition in an online community of inquiry,” Internet and Higher Education, vol. 14, 2011, pp. 183–190.
[2] W. Anderson and D. R. Krathwohl, Eds., A taxonomy for learning, teaching, and Assessment: A revision of Blooms’ Educational Objectives. New York, NY: Longman, 2001. [3] K. Berthold, M. Nückles, and A. Renkl, “Do learning
protocols support learning strategies and outcomes? The role of cognitive and metacognitive prompts,” Learning and Instruction, vol. 17, 2007, pp. 564-577.
[4] D. Boud, Enhancing learning through self assessment. Kogan Page, London, 1995.
[5] J. Cowan, “Learning journals: what are they for?,” British Journal of Educational Technology, vol. 23, no. 2, 1992, pp. 139-140.
[6] D. Hicks, “Supporting personalization and customization in a collaborative setting,” Computers in Industry, vol. 52, no. 1, 2003, pp. 71–79.
[7] W. Y. Hwang, N. S. Chen, R. Shadiev, and J.S. Li, “Effects of reviewing annotations and homework solutions on math learning achievement,” British Journal of Educational Technology, vol. 42, no. 6, 2011, pp. 1016-1028.
[8] W.Y. Hwang, R. Shadiev, and S.M. Huang, “A study of a multimedia web annotation system and its effect on the EFL writing and speaking performance of junior high school students,” ReCALL, vol. 23, no. 2, 2011, pp. 160–180. [9] W.Y. Hwang, R. Shadiev, C.Y. Wang, and Z.H. Huang, “A
Pilot Study of Cooperative Programming Learning Behavior and its Relationship with Students' Learning Performance,” Computers & Education, in press.
[10] W.Y. Hwang, C.Y. Wang, G.J. Hwang, Y.M. Huang, and S. Huang, “A web-based programming learning environment to support cognitive development,” Interacting with Computers, vol. 20, 2008, pp. 524–534.
[11] T. Milford and L. Brown, “The Role of Self-Assessment in the Preparation of Pre-Service Teachers,” Journal of Educational Practices, vol. 8, no. 1, 2010, pp. 47-58.
[12] J. Moon, “Learning Journals and Logs,” Reflective Diaries for UCD Teaching & Learning, 2010, 1-23.
[13] D. Nicol, “Laying a foundation for lifelong learning: Case studies of e-assessment in large 1st-year classes,” British Journal of Educational Technology, vol. 38, no. 4, 2007, pp. 668-678.
[14] L.B. Nilson, Teaching at Its Best: A Research-Based Resource for College Instructors. San-Francisco, CA: John Wiley and Sons Inc., 2010.
[15] G. Schraw and R.S. Dennison, “Assessing Metacognitive Awareness,” Contemporary Educational Pshychology, vol. 19, 1994, pp. 460-475.
[16] R.A. Sperling, B.C. Howard, L.A. Miller, and Murphy, “Measures of Children’s Knowledge and Regulation of Cognition,” Contemporary Educational Psychology, vol. 27, 2002, pp. 51–79.
