The behavior mining to discover error patterns

In this section, the debugging testing activity is proposed to assess the students‟

debugging capability. There are predefined buggy patterns as followings.

21

1). The detection of syntax error that violates the grammar of programming language.

 Missing token: student can detect the token omitted in the statement such as the missing token “;” omitted in the end of the statement “printf("hello world!")”.

 Extra token: student can detect the redundant token which should be removed such as extra token “;” in statement “for(….); printf(“*”);”.

 Illegal delimiter: student can understand that delimiter is a sequence of one or more characters used to specify the start and end boundary between separate regions of program expressions such as the curly brackets “{“ and

“}”.

2). The detection of misuse token error that happened in the mapping of pseudo code to program.

 Misuse operator: student can detect the wrong usage of comparison operator “==” with assignment expression “=” in the statement “if(a=1)”.

 Misuse operand data type: student can detect the incorrect operand usage with wrong data type or constraint such as integer variable “a” is mismatched with the wrong data type in the statement “int a = „c‟; ”.

3). The detection of logical error that happened in the design of pseudo code for solving a given problem.

 Unawareness of operator precedence: student can detect the computational priority of operators such as the programming for formula “(a+b)*h/2” may be written as “Area_Trapzium = a+b*h/2;” where the unawareness of operator precedence may cause the wrong result.

 Absence of boundary condition of some variable: student can detect the boundary condition of variables such as the “divided by zero” error in statement “b=0; a=1000/b;”.

 Unawareness of infinite loop: student can detect if the stopping criterion of

“for/while” or “if” the branching condition of expression is unreachable such as the infinite loop in statement “for(i=1, i<10, i--)” or “i=1;

while(i>1){ i++; }”.

Accordingly, the buggy patterns database can be constructed and indexed by the

taxonomy of the Problem Solving Strategy Ontology. Buggy patterns can be embedded in the selected programs and the students are asked to detect those buggy patterns by tracing the program to facilitate the error detection assessment. Two examples of “selection sort” and “factorial number generation” programs with embedded buggy patterns are shown in Example 4.1.

Example 4.1 Examples of program with embedded buggy patterns

With the constructed buggy patterns database, the buggy programs needed for error detection assessment can be provided. As shown in Figure 2, the example of selection sort with Program No. Q001 is embedded with buggy patterns “misuse token” in line 5, the “infinite loop” in line 8, and the “missing token” in line 15; the example of factorial number generation with Program No. Q002 is embedded with buggy patterns “Misuse operand data type” in line 7, the “Misuse operator” in line 10 and line 12.

Figure 4.3. Buggy programs of “selection sort” and “factorial number generation”

Definition 4.2 The student’s portfolio

23

The portfolio of a student is composed of a set of testing results: P = {s₁, s₂, s₃, …}

where symptom si=(Program_no, line_no, position_no, symptom_id) represents the novice student‟s mistake on program detection. The Program_no, line_no and position_no denote the position information of the occurred symptoms. The symptom_id denotes the identification of buggy pattern.

Example 4.2 The portfolio

Assume that the student failed to detect the “misused operator” or “infinite loop”

buggy patterns in program Q001, then the portfolio is {(Q001, 5, 10, Bug_misused_token), (Q001, 8, 1, Bug_infinite_loop)}.

Thus, the assessments of different symptoms are as follows.

 Assessment based on symptoms of syntax error: the symptoms of syntax error are including “missing token”, “extra token”, and “illegal delimiter”. The root causes for these symptoms can be most likely identified as misconception in

“program structure” if the student only has symptoms in illegal delimiter;

“unfamiliar with the statement” if the student fails in most of the symptoms; and

“incautious” if the student only has symptom in partial program.

 Assessment based on symptoms of misuse token: the symptoms of misuse token are including “misuse operator” and “misuse operand data type”. The root cause can be mostly identified as misconception in the “operator” if the student only has symptom in program operator; “the operand data type” if the student only has the corresponding symptoms; “fail in the mapping of pseudo code to program” if the learner has both symptoms.

 Assessment based on symptoms of logical error: the symptoms of logical error

are including “unawareness of operator precedence”, “absence of boundary condition of variable”, and “unawareness of infinite loop”. The root cause can be mostly identified as the corresponding misconception only when learner passes the syntax assessment but has the logical error symptoms.

Therefore, if the major symptoms in novice‟s portfolio matched the symptoms of repertory grid, it can be identified as major root cause misconception. Besides, cooperating with other minor symptoms, the possible misconceptions can also be provided to the learner for further remedial learning.

To build the students‟ behaviors, 20 students who participated in “Introduction to programming language” in the Asia University of Taiwan are involved. In the assessment, 11 programs are collected including 37 buggy patterns.

Next, the association rule mining is applied to generate the Concept effect relationship (CER) to demonstrate how the learning status of certain error patterns can be influenced by the other error patterns.

With the students‟ portfolio, we use the software WEKA 3.4.10 to mine the relation between the misconceptions of the students by Apriori Association Algorithm.

This algorithm is used to mine the relation between the misconceptions of the student.

There are fifty records and fifty-five attributes proceed in the WEKA. The result we have found is shown in tables below.

Table 4.1 Large 1 Itemsets of buggy pattern

Large Itemsets L(1):

Minimum support: 0.6 Minimum metric <confidence>: 0.9

Number of cycles performed: 8

The Buggy Pattern Support(%)

printf₁ 60%

printf6 65%

25

Table 4.2 Large 2 Itemsets of buggy pattern

Large Itemsets L(2):

Table 4.3 Large 3 Itemsets of buggy pattern

Table 4.4 Best rules found from large itemsets

Best rules found:

27

if1→doWhile1→if4 92%

scanf5→if4→doWhile1 92%

With the discovered behavior patterns, the relations of behavior patterns are shown in Figure 4.3. If the students made the mistake with misuse token of the statement “if”, then the students most likely made the mistake with misuse declaration to the statement “printf” as well. Furthermore, we can point out the misconception of the student who made several mistakes at the same time. For example, if the students misuse token in the statement “if”, then the students may have logical error in the statement “do-while” and misuse token in the statement

“while “ at the same time.

Figure 4.4 The relations of behavior patterns

After analyzing students‟ symptom portfolio, there are different error patterns discovered as followings.

 Assessment for statement “printf”: There are 55% students with symptom on the detection of “printf”. With the consideration of more symptoms, 15% students failed to detect almost all bugs. Therefore, their misconception can be judged as unfamiliar with the statement “printf”. The other 40% students only failed in the

misuse operator buggy pattern of “printf”. Therefore, their misconception can be judged as misuse operator with the statement “printf”.

 Assessment for statement “scanf”: There are 80% students with symptom on the detection of “scanf”. With the consideration of more symptoms, 25% students failed to detect almost all bugs. Therefore, their misconception can be judged as unfamiliar with the statement “scanf”. The other 55% students only failed in the misuse operand data type of “scanf”. Therefore, their misconception can be judged as misuse operand data type with the statement “scanf”.

 Assessment for the statement “if”: There are 85% students with symptom on the misuse operator of “if”. With the consideration of more symptoms, 45% students failed to detect almost all bugs of the misuse operator. Therefore, their misconception can be judged as misuse operator with the “if”. The other 40%

students also failed in the bugs of operator precedence, therefore, their misconception can be judged as “unawareness of operator precedence”.

 Assessment for the statement “for”: There are 65% students with symptom on the missing token of “for”. Since they only failed in this bug; therefore their misconception can be judged as unfamiliar with the “for” statement .

29