Chapter 3 Methodology
3.4 Proposing to Apply RaschGSP IRT Theory for Evaluating the Difficulty of
Students
In this section, the study would like to improve the proposed method above about the ability to differentiate ability score of students who are placed on the same line in Person-Map PM. With that thought, the research proposed the application of RaschGSP IRT theory to construct a new assessment method in order to evaluate the difficulty of questions and ability of students in the test, it could be applied for both small and large samples in order to solve the faced problem. In that way, the proposed method would in turn be applied to handle small and large sample, the obtained results are then compared to the results processed by the previous methods to test the validity and reliability of the proposed method (Sheu, Nguyen, Pham, Nguyen, et al., 2014; Sheu, Nguyen, Pham, Tsai, et al., 2014).
The flowchart for applying the proposed method is shown in Figure 3-13.
The purpose of this section is to evaluate the difficulty of each question in the test, and then estimate the ability of each student through test result. For each task, the process of processing data is carried out independently, so their processes are presented respectively as follows:
72
The process of processing data to evaluate the difficulty of questions in the test:
Step 1: Calculate and arrange the data with GSP chart
The original data are tested and normalized, then processed and arranged according to the rule of GSP chart, the results are presented according to specifications as in GSP chart (in the basic theory section).
Step 2: Arrange the data according to Gamma value
The questions are arranged in the order of their Gamma values (GP) from small to large.
Fig. 3-13 Flowchart for applying RaschGSP IRT to evaluate difficulty of questions and ability of students
Step 3: Rearrange the order according to the caution index for problem (CP)
The questions, which have the same Gamma value (GP), are rearranged in order of CP value from large to small.
Determine the last Gamma values STOP
Rearrange the data by CS, CP START
Determine raw data set
Test and normalize the data Analyze the data by GSP chart
Apply the RaschGSP IRT theory
73
Step 4: Apply the RaschGSP IRT theory for Gamma values arranged according to the above steps
The RaschGSP IRT theory is applied, RaschGSP IRT curve is plotted for the entire data set.
Step 5: Match the new Gamma values for all the questions
Each question corresponds to a point on the RaschGSP IRT curve, the value corresponding to each question on the y axis is its new Gamma value that needs to be determined.
The difficulties of questions are calculated according to their Gamma values.
The process of processing data to evaluate the ability of examinee:
Step 1: Calculate and arrange the data with GSP chart
The original data are tested and normalized, then processed and arranged according to the rule of GSP chart, the results are presented according to specification as in GSP chart.
Step 2: Arrange the order according to Gamma value
The examinees are arranged in the order of their Gamma value (GS) from large to small.
Step 3: Rearrange the order according to caution index for student (CS)
The examinees who have the same Gamma value (GS), are rearranged in order of CS value from small to large.
Step 4: Apply the RaschGSP IRT theory for Gamma values arranged according to the above steps
The RaschGSP IRT theory is applied, RaschGSP IRT curve is plotted for all data.
Step 5: Match the new Gamma values for all the examinees
Each examinee corresponds to a point on the RaschGSP IRT curve, the value corresponding to each examinee on the y axis is its new Gamma value that needs to be determined. The abilities of examinees are calculated according to their Gamma values, the larger the Gamma value an examinee obtains the higher his (her) ability presents.
74
Example 3.2:
There was an examination with a 20-question Math test for a class including 15 junior high school students. The obtained data were the dichotomous S-P chart (See appendix 2). The requirement is applying the proposed method above to evaluate difficulty of question and ability of student.
Firstly, the data are tested for reliability, and the Cronbach’s Alpha reaches 0.88.
After being tested for normalization, the data satisfy the condition for analysis (data verifying).
Secondly, using the program written in MATLAB (Program for RaschGSP IRT) to analyze the data, the proposed method is applied (data processing).
Finally, the results of RaschGSP IRT analysis are RaschGSP IRT curve for problems (Fig. 3-14) and RaschGSP IRT curve for students (Fig. 3-15) mentioned in step 4 (output data).
Fig. 3-14 RaschGSP IRT curve for problems in example 3.2
75
Fig. 3-15 RaschGSP IRT curve for students in example 3.2
To show the result of evaluating difficulty of each problem, a coordinate method was applied. The difficulty of questions were calculated and exported from Program for RaschGSP IRT and presented in Table 3-2, they were arranged in order from small to large value.
Table 3-2
Difficulty of questions (example 3.2)
Question-number 15 8 5 12 16 20 19 4 14 3
Difficulty 0.115 0.120 0.127 0.136 0.148 0.164 0.184 0.209 0.241 0.279
Question-number 6 17 18 9 2 7 1 10 13 11
Difficulty 0.325 0.377 0.436 0.499 0.565 0.629 0.691 0.747 0.796 0.838
The difficulties of questions are distinguished obviously even they have the same correct answer rates, so they are arranged in logical order. Similarly, the student abilities estimated by the proposed method were presented in Table 3-3, students are sorted according to their ability value in order from small to large.
76
Table 3-3
Ability of students (example 3.2)
Student-number 10204 10212 10205 10211 10215 10213 10208 10203 Ability 0.134 0.135 0.136 0.138 0.152 0.152 0.152 0.200 Student-number 10201 10209 10214 10210 10206 10207 10202
Ability 0.256 0.345 0.543 0.543 0.751 0.890 0.890
The students’ ability are evaluated and differentiated by the proposed method clearly, so the students are arranged in order according to their ability reasonably even their correct answer rates are similar.