Chapter 3 Self-organizing Behavior Modeling
3.2 The idea
With our observations, students could have meaningful learning actions when the purposes of the actions in specific learning context are obtained. To model this property in the learning platform, the knowledge-based programming learning platform is proposed to provide learning guidance based on the high level knowledge structure as shown in Figure 3.3. The learning content ontology is used to construct the metadata and assessment rule for the learning status of students. From the obtained portfolio, the guidance rules of learning platform can be extended by the ontology evolving process called ontology crystallization.
Figure 3.3 The knowledge-based programming learning platform
With the knowledge-based learning platform, the knowledge engineering approach is applied to obtain the self-organizing behavior model as shown in Figure 3.4. There are four steps in the self-organizing model which are the knowledge representation to define the extensible ontology structure, the knowledge acquisition
to extend the knowledge from folksonomy and maintain the stability using the proposed Folksonomy-based Delphi method, the knowledge reasoning to provide learning guidance using the assessment rules generated from constructed Ontology and the knowledge retaining to discover new behaviors and maintain the stability by proposed cascading data mining.
Figure 3.4 The knowledge engineering for self-organizing behavior modeling
Accordingly, to build the students‟ behavior model, the Purpose-based Ontology is built to model the purpose of the actions. The definition of Purpose-based Ontology is as followings.
Definition 3.2 The Purpose-based Ontology (O) is defined as O = (P, C, A, R), where
P={p1, p2, …, pn} is a finite set of purpose nodes in the Purpose Layer to represent predefined purposes in the domain.
15
For each purpose pi, the concepts Ci={pi.c1, pi.c2, …}is a finite set of topic nodes in the Concept Layer to represent different topics discussed in the forum. Topic nodes are linked to the corresponding purpose node by the “A Part OF” relations.
For each purpose pi and concept cj, the action Aijk={ pi.cj.a1, pi.cj.a2, …} is a finite set of nodes in the Action Layer to represent the linkages associated to the original action log. The action nodes are linked to the corresponding concept node by the
“Instance Of” relations.
The Purpose-based Ontology can be instantiated to different concepts definition in different learning contexts. Next, the ontology-based learner behavior modeling approach is proposed to analyze the frequent action patterns and organize the obtained patterns with the structure of ontology as learning behaviors. Therefore, the learning guidance issues under different contexts can be resolved by the following behavior modeling approaches to provide the adaptive learning guidance based on the built behavior models.
Under the context of intelligent tutoring system, the Generalized Model Tracing approach is proposed to organize the diagnosis results of different program model tracing with Problem Solving Strategy Ontology. The diagnosis result can be used to provide the learning guidance for problem solving strategy. The learning guidance applications of adaptive programming misconception diagnosis and game rule tuning learning activity are investigated to evaluate the self-assessment behavior model.
Under the context of learning forum, the Cascading Topic Clustering Algorithm and Self-organized Ontology Maintenance Scheme are proposed to organize the forum experts‟ inquiry activities with the Purpose-based Ontology. Building the forum experts‟ behavior model can provide trustworthy expert finding service for
inquiry-based learning. The learning guidance application of trustworthy expert finding service for inquiry learning on the programming learning forum is investigated to evaluate the inquiry behavior model.
Under the context of collaborative constructed sharable content repository, the IRT-Based Metadata Reengineering Scheme is proposed to evaluate the effectiveness of folksonomy tags and resolve the synonym, redundancy and incompleteness problem of metadata by the domain taxonomy. Accordingly, the tag effectiveness value can detect the conflict and provide the tagging guidance to resolve the consensus building problem to obtain the well-tagged metadata. The learning guidance applications of metadata reengineering and intelligent solution retrieval system are investigated to evaluate the knowledge sharing behavior model.
17
Chapter 4
Behavior modeling of self-assessment activity on online testing system
Comprehending the program examples is important for students in the course of Introductory to Programming to develop their problem solving capability. With our observation, well-trained programmers usually have the capability to explain the problem solving strategic concept of program examples. For example, the linear recurrence strategy is used in the Selection Sort Program to reduce the original sorting problem of n elements into the sub-problem of sorting n-1 elements. The recurrence strategy can be used in a series of programming designs such as quick sort, binary search, printing multiplication table, etc. In this chapter, the behavior modeling to provide adaptive learning guidance for problem analysis is discussed.
4.1 Programming behavior modeling to provide adaptive testing and learning guidance
To model the programming learning behavior, the Bloom‟s taxonomy [10] can be used to classify the programming learning achievement into different capability levels.
The first is knowledge level in which the students can recognize the syntax and statement of program. The second is comprehension level in which the students can understand the semantic flow chart of program. The third is application level in which the students can understand the problem solving strategy used in the algorithm [21].
The fourth is problem analysis level in which the learner can analyze the properties of problem and select suitable problem solving strategy. However, most students can
only understand the knowledge level of the program examples. Thus, students are usually frustrated by the high barrier of program designing in the advanced programming course.
In the researches of behavior modeling on programming testing, the model tracing approach [3][40][60] is the most well-known. The traditional model tracing approach is based on a sample program of specific problem. Before testing, the student‟s possible coding actions should be modeled already. Thus, the testing process would interactively ask student to write down the code step by step. During the model tracing process, once the student encounters troubles in some step, the corresponding remedial instructions can be triggered to assist the student to finish the program coding. However, traditional model tracing approach can only support the intra-program behavior assessment e.g., it can only provide the syntax and semantic level instructions of single program statement.
To solve the problem, this chapter applied the pedagogical theory of Vygotsky‟s Zone of Proximal Development to support students learning the problem solving strategy with the metacognitive scaffolding. The Problem Solving Strategy Ontology is constructed to connect the relationship between low level programming statements and high level problem solving strategic concept.
Definition 4.1 Problem Solving Strategy Ontology (PSSO) There are three layers in PSSO:
Problem Strategic Layer: the algorithms of specific problem for students want to learn.
Semantic Layer: the pragmatic semantic concepts that may be the misconceptions in comprehension level for students.
19
Figure 4.1 Problem Solving Strategy Ontology (PSSO)
Lev Vygotsky‟s theory of Zone of Proximal Development (ZPD) is a core of the theory of cognitive development which explores the development of the psychological function by analyzing the relationship between actual development level learning and developing. That is to say, we want to know the students' current status and their potential capabilities in the learning.
Students often have the primitive semantic knowledge, but they mostly do not have the pragmatic semantic knowledge. To refer to the ZPD theory, we thought that the students have the misconceptions in program understanding, because there is the knowledge boundary in the comprehension level of programming learning, as shown in Figure 4.2.
Knowledge
Figure 4.2 Knowledge Boundary in programming learning domain
Therefore, based on the Problem Solving Strategy Ontology, the classical program examples such as selection sort, bubble sort, binary search, etc are collected as learning scaffolding for training the recurrence strategy. In addition, the primitive program examples of the textbook are collected to provide the diagnosis of students‟
actual development level. The collected program examples are tagged with concepts of the Problem Solving Strategy Ontology.
According to the tagged program examples, the students‟ programming behaviors including frequent used pattern and error patterns can be built from mining the programming testing log. Once the behavior is built, the adaptive learning guidance service can be provided to assist students during the training of problem solving.
4.2 The behavior mining to discover error patterns
In this section, the debugging testing activity is proposed to assess the students‟
debugging capability. There are predefined buggy patterns as followings.
21
1). The detection of syntax error that violates the grammar of programming language.
Missing token: student can detect the token omitted in the statement such as the missing token “;” omitted in the end of the statement “printf("hello world!")”.
Extra token: student can detect the redundant token which should be removed such as extra token “;” in statement “for(….); printf(“*”);”.
Illegal delimiter: student can understand that delimiter is a sequence of one or more characters used to specify the start and end boundary between separate regions of program expressions such as the curly brackets “{“ and
“}”.
2). The detection of misuse token error that happened in the mapping of pseudo code to program.
Misuse operator: student can detect the wrong usage of comparison operator “==” with assignment expression “=” in the statement “if(a=1)”.
Misuse operand data type: student can detect the incorrect operand usage with wrong data type or constraint such as integer variable “a” is mismatched with the wrong data type in the statement “int a = „c‟; ”.
3). The detection of logical error that happened in the design of pseudo code for solving a given problem.
Unawareness of operator precedence: student can detect the computational priority of operators such as the programming for formula “(a+b)*h/2” may be written as “Area_Trapzium = a+b*h/2;” where the unawareness of operator precedence may cause the wrong result.
Absence of boundary condition of some variable: student can detect the boundary condition of variables such as the “divided by zero” error in statement “b=0; a=1000/b;”.
Unawareness of infinite loop: student can detect if the stopping criterion of
“for/while” or “if” the branching condition of expression is unreachable such as the infinite loop in statement “for(i=1, i<10, i--)” or “i=1;
while(i>1){ i++; }”.
Accordingly, the buggy patterns database can be constructed and indexed by the
taxonomy of the Problem Solving Strategy Ontology. Buggy patterns can be embedded in the selected programs and the students are asked to detect those buggy patterns by tracing the program to facilitate the error detection assessment. Two examples of “selection sort” and “factorial number generation” programs with embedded buggy patterns are shown in Example 4.1.
Example 4.1 Examples of program with embedded buggy patterns
With the constructed buggy patterns database, the buggy programs needed for error detection assessment can be provided. As shown in Figure 2, the example of selection sort with Program No. Q001 is embedded with buggy patterns “misuse token” in line 5, the “infinite loop” in line 8, and the “missing token” in line 15; the example of factorial number generation with Program No. Q002 is embedded with buggy patterns “Misuse operand data type” in line 7, the “Misuse operator” in line 10 and line 12.
Figure 4.3. Buggy programs of “selection sort” and “factorial number generation”
Definition 4.2 The student’s portfolio
23
The portfolio of a student is composed of a set of testing results: P = {s1, s2, s3, …}
where symptom si=(Program_no, line_no, position_no, symptom_id) represents the novice student‟s mistake on program detection. The Program_no, line_no and position_no denote the position information of the occurred symptoms. The symptom_id denotes the identification of buggy pattern.
Example 4.2 The portfolio
Assume that the student failed to detect the “misused operator” or “infinite loop”
buggy patterns in program Q001, then the portfolio is {(Q001, 5, 10, Bug_misused_token), (Q001, 8, 1, Bug_infinite_loop)}.
Thus, the assessments of different symptoms are as follows.
Assessment based on symptoms of syntax error: the symptoms of syntax error are including “missing token”, “extra token”, and “illegal delimiter”. The root causes for these symptoms can be most likely identified as misconception in
“program structure” if the student only has symptoms in illegal delimiter;
“unfamiliar with the statement” if the student fails in most of the symptoms; and
“incautious” if the student only has symptom in partial program.
Assessment based on symptoms of misuse token: the symptoms of misuse token are including “misuse operator” and “misuse operand data type”. The root cause can be mostly identified as misconception in the “operator” if the student only has symptom in program operator; “the operand data type” if the student only has the corresponding symptoms; “fail in the mapping of pseudo code to program” if the learner has both symptoms.
Assessment based on symptoms of logical error: the symptoms of logical error
are including “unawareness of operator precedence”, “absence of boundary condition of variable”, and “unawareness of infinite loop”. The root cause can be mostly identified as the corresponding misconception only when learner passes the syntax assessment but has the logical error symptoms.
Therefore, if the major symptoms in novice‟s portfolio matched the symptoms of repertory grid, it can be identified as major root cause misconception. Besides, cooperating with other minor symptoms, the possible misconceptions can also be provided to the learner for further remedial learning.
To build the students‟ behaviors, 20 students who participated in “Introduction to programming language” in the Asia University of Taiwan are involved. In the assessment, 11 programs are collected including 37 buggy patterns.
Next, the association rule mining is applied to generate the Concept effect relationship (CER) to demonstrate how the learning status of certain error patterns can be influenced by the other error patterns.
With the students‟ portfolio, we use the software WEKA 3.4.10 to mine the relation between the misconceptions of the students by Apriori Association Algorithm.
This algorithm is used to mine the relation between the misconceptions of the student.
There are fifty records and fifty-five attributes proceed in the WEKA. The result we have found is shown in tables below.
Table 4.1 Large 1 Itemsets of buggy pattern
Large Itemsets L(1):
Minimum support: 0.6 Minimum metric <confidence>: 0.9
Number of cycles performed: 8
The Buggy Pattern Support(%)
printf1 60%
printf6 65%
25
Table 4.2 Large 2 Itemsets of buggy pattern
Large Itemsets L(2):
Table 4.3 Large 3 Itemsets of buggy pattern
Table 4.4 Best rules found from large itemsets
Best rules found:
27
if1→doWhile1→if4 92%
scanf5→if4→doWhile1 92%
With the discovered behavior patterns, the relations of behavior patterns are shown in Figure 4.3. If the students made the mistake with misuse token of the statement “if”, then the students most likely made the mistake with misuse declaration to the statement “printf” as well. Furthermore, we can point out the misconception of the student who made several mistakes at the same time. For example, if the students misuse token in the statement “if”, then the students may have logical error in the statement “do-while” and misuse token in the statement
“while “ at the same time.
Figure 4.4 The relations of behavior patterns
After analyzing students‟ symptom portfolio, there are different error patterns discovered as followings.
Assessment for statement “printf”: There are 55% students with symptom on the detection of “printf”. With the consideration of more symptoms, 15% students failed to detect almost all bugs. Therefore, their misconception can be judged as unfamiliar with the statement “printf”. The other 40% students only failed in the
misuse operator buggy pattern of “printf”. Therefore, their misconception can be judged as misuse operator with the statement “printf”.
Assessment for statement “scanf”: There are 80% students with symptom on the detection of “scanf”. With the consideration of more symptoms, 25% students failed to detect almost all bugs. Therefore, their misconception can be judged as unfamiliar with the statement “scanf”. The other 55% students only failed in the misuse operand data type of “scanf”. Therefore, their misconception can be judged as misuse operand data type with the statement “scanf”.
Assessment for the statement “if”: There are 85% students with symptom on the misuse operator of “if”. With the consideration of more symptoms, 45% students failed to detect almost all bugs of the misuse operator. Therefore, their misconception can be judged as misuse operator with the “if”. The other 40%
students also failed in the bugs of operator precedence, therefore, their misconception can be judged as “unawareness of operator precedence”.
Assessment for the statement “for”: There are 65% students with symptom on the missing token of “for”. Since they only failed in this bug; therefore their misconception can be judged as unfamiliar with the “for” statement .
29
4.3 Evaluation
With the discovered programming error patterns, scaffolding-based assessment which is the adaptive learning guidance service can be provided. Our learning activity is implemented by using cloze testing to obtain student‟s knowledge boundary and guide students to comprehend the problem solving strategy.
As shown in Figure 4.4, there are four processes for the scaffolding-based learning by example process. The first is to collect classical sample programs as the example database. The second, the programming comprehension starts from providing a target program for the learner to learn. While learner has misconception on the target program, instead of give the answer directly, the third process provides more sample programs for learner to practice by decomposition the program structure of original sample program. Thus, for each concept of program structure, it retrieves several programs from the program database which are similar to the provided sub-structure of the target program as the remedial sample program. Since the remedial sample programs are easier than the original one, the learner may easier to comprehend. Next in the forth process, once the provided remedial sample programs are learned, the scaffolding-based learning by example process applies the remedial learning using these programs as the scaffolding examples to hint the learner. Thus, with the sample programs as learning scaffolding, the learner can easier comprehend the target program.
During the scaffolding-based learning by example process, when a program is provided for leaner to learn, we still want to know if she can really understand the program or not. Therefore, our idea is to apply the cloze test on program structure to facilitate the assessment during the programming comprehension learning. Since the assessment using cloze test is open-ended for the learner to provide her answer, it
motivates the learner to comprehend context and structure of the sample program in order to answer the cloze test item. Moreover, since the cloze box can be set in the program structure level, block statement level and statement element level, for one sample program, it can further detect the learner‟s misconception is happened in what kind of comprehension level. Thus, it can provide more informative feedbacks.
Programming
Figure 4.5. The scaffolding-based learning by example process
Scaffolding-based cloze item generation
With the retrieved sample programs, the learning of programming comprehension is based on the structured programming model. The cloze test is provided for the learner level by level based on the program structure. As shown in Figure 4.5, the cloze test item of the sample program is generated by replacing the program
With the retrieved sample programs, the learning of programming comprehension is based on the structured programming model. The cloze test is provided for the learner level by level based on the program structure. As shown in Figure 4.5, the cloze test item of the sample program is generated by replacing the program