Chapter 3. Methods
3.6. Research Tools
The research tools used in this study included midterm math quiz, invisible cube enumeration test, cube enumeration test, VIB-based CAT, VIB-based CBT and a learning system.
3.6.1. Math Midterm Quiz
In this study, we used the examinees’ grades of midterm math quiz of the last year as the external criterion to analyze whether the research tool of this study could measure their spatial ability effectively or not. This exam paper was the fifth graders’
exam paper of square measure of cuboid and rectangle edited by the teachers who were from Cheng Kung elementary school and Chung Yi elementary school. The content of this quiz was showed as appendix A. The descriptive statistics of it was bellowed.
Table 3-3
Descriptive statistics of the midterm math quiz (N = 267).
M SD
Midterm math quiz score 84.14 9.00
Hence, the researcher regarded that as the students calculated volume by layers so they could calculate square measure correctly and therefore they should also do well when calculating cube number by layers. In addition, for volume, most items in this exam asked them to infer the volume of these cubes using the concepts of stacked cubes. As a result, this study also reasoned that if they could answer them correctly then they would also do well in the cube enumeration test. Finally, some items were the traditional volume calculation questions: providing length, width and height and then asking them to calculate the volume. According to the above literature review, it was known that the major purpose of the cube enumeration test was to measure the ability of spatial orientation and spatial visualization and volume concepts, and therefore we used this midterm exam score to be the external criterion in order to
54
understand whether the research tool developed by this study could measure their spatial ability effectively or not.
3.6.2. Invisible cube enumeration test
Invisible cube enumeration test was edited by three teachers with 10 years of teaching experience. Its contents reference volume of teaching materials for fifth-grade students. The purpose of this test was aim of measuring the spatial ability and mathematical volume computing ability. And by analyzing the invisible cube number and rotation functions, in order to understand how these two factors affect the difficulty of the cube enumeration test.
3.6.2.1. Pilot test
Before the formal test was conducted, the researcher first had a pretest on these 30 sixth graders; then the reliability of the results was analyzed and its value was .78 showing that they had a high reliability. The test editors used the CTT to calculate the item difficulty and discrimination of the test. Difficulty and discrimination of each item were shown in the following:
55 Table 3-4
Examinees’ answering status of the pilot-test (Number of items =25, N=30) Item
The descriptive statistics were shown in the following table:
56 Table 3-5
The descriptive statistics of item difficulty and discrimination of pilot test (Number of items =25, N=30)
Minimum Maximum M SD
Difficulty .56 1.00 .87 .15
Discrimination .00 .86 .31 .35
For item selection, the researcher removed items with difficulty equal 1 and discrimination equal 0; besides, items with difficulty close to 1 were rotatable, the study thus kept them and would discuss them in order to analyze whether rotation could reduce the difficulty for the examinees to do spatial orientation and visualization.
After removing items with difficulty equal 1and discrimination equal 0, the descriptive statistics were shown in the following table, and the items were shown in Appendix B:
Table 3-6
The descriptive statistics of item difficulty and discrimination of the invisible cube enumeration test (number of items = 20, N=30)
Minimum Maximum M SD
Difficulty .56 .99 .85 .13
Discrimination .00 .86 .41 .33
Regarding to the results of the allowance of item rotation, the average item difficulty showed in Table 3-7. The average item difficulty was .97 and the average item discrimination was .12.
Table 3-7
The descriptive statistics of item difficulty and discrimination of the rotatable invisible cube enumeration test (number of items = 10, N=30)
Minimum Maximum M SD
Difficulty .87 .99 .97 .12
Discrimination .00 .22 .12 .11
57
Therefore, it seems that the test items couldn’t calculate examinees’ ability precisely in the allowance of item rotation situation. Hence, this study will consider using the items which could be rotated in the following studies to do the further analysis of the influence of the allowance of item rotation.
In addition, from the literature review we found out that the cube enumeration test majorly measured spatial orientation and spatial visualization. Most students had difficulty in doing spatial orientation and spatial visualization; nevertheless, when items could be rotated, they found them easier. This meant that when items of a cube enumeration test could be rotated on a 2D screen, it would assist them in doing spatial orientation and spatial visualization. Therefore, this study will keep on investigating whether “rotation” function is appropriate in teaching or not in last sections.
3.6.2.2. The contents of invisible cube enumeration test
The invisible cube enumeration test contained items of different setting regarding to the allowance of item rotation and the number of invisible cubes. As showed in Table 3-8, there were 20 items in this test.
58 Table 3-8
The contents of invisible cube enumeration test Item
Number
Number of Invisible Cubes
Allowance of Rotation Number of Total Cubes
3.6.2.3. Flowchart of taking the invisible cube enumeration test
Figure 3-11 described how examinees took the invisible cube enumeration test.
After examinees logged in the system, an example item would be assigned to them as an exercise. If the answer was answered correctly, the formal test would begin;
otherwise, they had to take the exercise again. The major purpose of this test was to
59
ask the examinees to count the total number of cubes, and some items could be rotated.
When the formal test began, the system would assign items randomly. And after the test finished, it would record the results and show them to the examinees. In this test, the system will record the subjects’ examination time for future research.
Figure 3-11. The flowchart of taking the invisible cube enumeration test Examinees login into the system
Explain the concept of invisible cube enumeration and introduce the test interface and example
Generate an example item
Show the correct answer Is Correct?
The system display cube enumeration test items
Finish the test?
Show the total number of items correctly answered
End
N Y
N
Y
60
3.6.2.4. Analysis of the results of the invisible cube enumeration test
In the invisible cube enumeration test, the total cube number ranged from 8 to 16 while the number of invisible cubes was from 3 to 7. The main purpose was to investigate the factors influencing invisible cube number and the allowance of rotation.
The item difficulty and discrimination of the invisible cube enumeration test were showed in Table 3-9. The mean of item difficulty was .80 and the mean of item discrimination was .40, which meant that the overall difficulty was too easy for the sixth graders.
Table 3-9
Statistics properties of test items of the invisible cube enumeration test (number of items = 20, N = 267)
Minimum Maximum M SD
Difficulty .58 .99 .80 .16
Discrimination .00 .87 .40 .32
The item difficulty and discrimination of each item and the examinees’ answering status of the invisible cube enumeration test were showed in Table 3-10. In the test, the item 17 and item 18 were the most difficult one, and corresponding item difficulty was .58. The test item 3, item 4 and item 7 which can be rotated was the easiest one, and its item difficulty was .99. For the details of each test item, please refer to research tools section. According to Table 3-10, it was found that when an item had more invisible cubes, it was more difficult, the difficult parameter value is small.
61 Table 3-10
The examinees’ answering status of the invisible cube enumeration test Number of invisible average item difficulty was show in Table 3-11 the average item difficulty was .94 and the average item discrimination was .13.
62 Table 3-11
The descriptive statistics of item difficulty and discrimination of the invisible cube enumeration test (number of items = 10, N = 267)
Minimum Maximum M SD
Difficulty .83 .99 .94 .06
Discrimination .00 .35 .13 .11
In addition, after interviewing with several students the researcher understood that rotatable cubes did a great help when they took the test. Parts of the interview results showed that when items could be rotated, they could calculate cube amount right away and the accuracy rate also greatly improved. Furthermore, after they had the experience of rotatable items they would calculate the answer easily due to the prior experience. Besides, some students could know that some cubes were covered by visible cubes from those rotatable cubes, and this represented that rotatable items could help their spatial orientation and spatial visualization.
Finally, some of them understood the concepts of volume better and some understood that volume was calculated by multiplying square measure by height. This also indicated that the teaching of cube enumeration test did help their concepts of volume and square measure. In light of it, this study would use the cube enumeration test as a teaching tool to teach the students cube enumeration and volume calculation hoping that this could improve their ability of spatial orientation, spatial visualization and volume calculation.
3.6.3. Cube enumeration test
The cube enumeration test was constructed by 3 experienced teachers with 10 years of experience who referred to the fifth graders’ course contents related to volume in the first semester. Its purposes were to measure the examinees’ abilities of spatial orientation, spatial visualization and volume calculation and also produce an item difficulty formula of the test through analyzing invisible cubes and item integrity.
63
3.6.3.1. Pilot test
Before the formal test was conducted, the researcher first had a pretest on these 30 sixth graders; then the reliability of the results was analyzed and its value was .81 showing that they had a high reliability and consistency. The item difficulty and discrimination of the pretest were calculated by classic difficulty for selecting items.
Difficulty and discrimination of each item were shown in the following:
Table 3-12
Item difficulty and discrimination of the cube enumeration test (N=30) Item
Number Number of invisible cube Integrity of cubes Difficulty Discrimination
1 4 7 .49 .86
64
The descriptive statistics were shown in the following table:
Table 3-13
The descriptive statistics of item difficulty and discrimination of the cube enumeration test (number of items = 36, N=30)
Minimum Maximum M SD
Difficulty .22 .95 .59 .16
Discrimination .17 .90 .60 .24
For item selection, an item with discrimination lower than .2 was removed. After the selection, the descriptive statistics of the items were shown in Table 3-14. The difficulty level was medium and the items were shown in the Appendix C:
65 Table 3-14
The descriptive statistics of item difficulty and discrimination of the cube enumeration test (number of items = 30, N=30)
Minimum Maximum M SD
Difficulty .38 .89 .54 .09
Discrimination .37 .90 .69 .16
3.6.3.2. The contents of cube enumeration test
The cube enumeration test was developed to present different settings regarding to the invisible cube number and the integrity of cubes for the examinees to take. This test contained 30 items showed in Table 3-15.
Table 3-15
The contents of cube enumeration test Item
Number
Number of Invisible Cubes Integrity of Cubes Number of Total Cubes
66
15 6 9 20
16 6 7 20
17 6 8 18
19 6 7 17
19 7 7 20
20 7 9 20
21 7 6 20
22 7 7 20
23 7 8 19
24 7 7 18
25 8 8 20
26 8 9 20
27 8 7 20
29 8 6 24
29 8 10 21
30 8 6 20
3.6.3.3. Flowchart of taking the cube enumeration test
The process as examinees took the cube enumeration test was demonstrated in Figure 3-12. After examinees logged in the system, the system would explain how to take the test, and an example item would be then presented to them as an exercise. If the answer was answered correctly, the formal test would begin; otherwise, they had to take the exercise again. The major purpose of this test was to ask the examinees to count the number of total cubes. When the formal test began, the system would assign items randomly. After the test finished, it would recorded the results and showed them to the examinees.
67
Figure 3-12. Flowchart of taking the cube enumeration test
3.6.4. Analysis of IRT Rasch Model
The contents of the cube enumeration test used in this study were the stacked cube images generated by the system. In this test, the examinees had to calculate the amount of the cubes. The total cube number ranged from 17 to 24 while invisible cube number ranged from 4 to 8, and the integrity of cubes ranged from 6 to 10. The overall 30 items’ difficulty parameters of Rasch model were listed in Table 3-16. The descriptive statistics of items were showed in Table 3-17 . The overall difficulty was a little easy for the sixth graders.
Examinees login into the system
Explain the concept of cube enumeration and introduce the test interface and example item Generate an example item
Show the correct answer Is Correct?
The system display cube enumeration test items
Finish test?
Show the total number of items correctly answered
End Y N
68 Table 3-16
The overall 30 items’ difficulty parameters of Rasch model
Item Number Number of invisible cube Integrity of cubes b
1 4 7 -3.59
69 Table 3-17
Statistical properties of the test items of the cube enumeration test (number of items = 30, N = 267)
IRT Parameter Minimum Maximum M SD
B -3.59 1.51 -0.32 1.39
The test characteristic curve of the test is as bellow:
Figure 3-13. The test characteristic curve
After using IRT to analyze the results, it was found out that the difficulty level was low to medium. Besides, the study planned to use the results to analyze IRT difficulty formula and utilize invisible cubes and item integrity to do the algorithm expecting that the results could assist the construction of VIB and VIB-based CAT.
0 5 10 15 20 25 30 35
-3.873 -2.873 -1.873 -0.873 0.127 1.127 2.127 3.127
Score
Z TCC
70
3.6.5. Analysis of CTT
The contents of the cube enumeration test used in this study were the stacked cube images generated by the system. In this test, the examinees had to calculate the amount of the cubes. The total cube number ranged from 17 to 24 while invisible cube number ranged from 4 to 8, and the integrity of cubes ranged from 6 to 10. There were totally 30 items in the test. The overall item difficulty and discrimination of the cube enumeration test were listed in Table 3-18. In Table 3-19 the average item difficulty was .65 and the average item discrimination was .70. The overall difficulty was a little easy for the sixth graders and item discrimination was quite well.
Table 3-18
The CTT item difficulty and discrimination of the cube enumeration test Item
Number Number of invisible cube Integrity of cubes Difficulty Discrimination 1
71
Statistical properties of the test items of the cube enumeration test (number of items = 30, N = 267).
Minimum Maximum M SD
Difficulty .55 .87 .65 .07
Discrimination .26 .89 .70 .14
According to Table 3-18, the most difficult item with difficulty value of .55 consisted of 6 invisible cubes and its integrity of cubes is 7. After averaging the groups within each invisible cube number (Table 3-20), the results showed that the average value of item difficulty of group with 4 invisible cubes was .66, group with 5 invisible cubes was .69, group with 6 invisible cubes was .64, group with 7 invisible cubes was .64, and group with 8 invisible cubes was .62. This was consistent with the difficulty tendency of the invisible cube enumeration test. In other words, the number of invisible cubes would affect item difficulty, and when there were more invisible cubes, the item would be more difficult. The line chart of Table 3-20 was showed in Figure 3-14.
72 Table 3-20
Mean item difficulty and discrimination of groups of invisible cubes numbers of the cube enumeration test
Figure 3-14. The line chart of mean item difficulty and discrimination of groups of invisible cubes numbers of the cube enumeration test
0
Mean difficulty of the group with lower score Mean difficulty of the group with higher score Difficulty
Discrimination
73
After grouping integrity of cubes of the cube enumeration test (Table 3-21), the results showed that the average item difficulty of group with integrity degree 6 was .75 and the average item difficulty of group with integrity degree 7 was .62, and the average item difficulty of group with integrity degree 8 was .62, the average item difficulty of group with integrity degree 9 was .61, and the average item difficulty of group with integrity degree 10 was .58. This meant that integrity degree numbers would affect item difficulty, and the item was more difficult when there were more integrity degree. The line chart of Table 3-21 was showed as Figure 3-15.
Table 3-21
Average item difficulty and discrimination of groups within different integrity of cubes in the cube enumeration test
Integrity of cubes
Average item difficulty of the group with lower score
Average item difficulty of the group with higher score
Difficulty Discrimination
6 .29 1.00 .75 .5
7 .31 .93 .62 .75
8 .11 .85 .62 .77
9 .17 .79 .61 .77
10 .02 .71 .58 .85
74
Figure 3-15. The line chart of mean item difficulty and discrimination of groups within different integrity of cubes in the cube enumeration test
The test response statistics of the invisible cube enumeration test by gender is showed in Table 3-22.There were totally 146 male examinees and their average number of correct items was 17.12 with the standard deviation of 3.21. As for the 121 female examinees, their average number of correct items was 17.91 and the standard deviation was 3.98. The t-test of their test results shown in Table 3-23 indicated that there was no significant difference. Previous studies pointed out that male and female didn’t have much difference in spatial ability before puberty (Linn & Petersen, 1985) and this was consistent with the results of the invisible cube enumeration test.
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
6 7 8 9 10
Average item difficulty of the group with lower score Average item difficulty of the group with higher score Difficulty
Discrimination
75 Table 3-22
Descriptive statistics of male and female examinees and correct item of the invisible cube enumeration test
N M SD Minimum Maximum
Male 146 15.66 12.37 0 30
Female 121 16.76 11.98 0 30
Total 267 17.28 12.18
Table 3-23
t-test results of correct item of males and females in the invisible cube enumeration test
N M SD df t
Male 146 15.66 12.37 265 -.89
Female 121 16.76 11.98
3.6.6. Cube enumeration learning system
This system was designed by the “rotation” function of the computerized cube enumeration test; teachers could use this system to do instruction. In this system, the cubes on the screen could be rotated by multi viewpoint so that students could easily understand the relative position of these cubes and hence it could assist them to build up spatial orientation and spatial visualization in their minds. By using this system, not only could teachers perform the teaching of computerized cube enumeration test, but students could use it to learn and do exercises of computerized cube enumeration test.
Besides, it could either allow teachers to build up exercise items for teaching or test students. Its functions included:
76
1. It uses a computer screen to display 3D cubes on a 2D screen and allows students to rotate cubes by a mouse. It will assist teachers in teaching.
Figure 3-16. To use cube enumeration learning system help learning.
2. Students can log in the system, and it can display the process that cubes are stacked and allow them to operate by a mouse so that they can understand the relative position of cubes. This will assist them to build up spatial orientation and spatial visualization.
77
Figure 3-17. An item stack example by using cube enumeration learning system.
3. Students can use the system to do exercises, and when they have wrong answers it will give them clues and allow them to rotate cubes so that it will assist them to learn.
3.6.7. Virtual Item Bank of Cube Enumeration System
Assisted by “invisible cube enumeration test” and “cube enumeration test”, this study used the test results to build up a virtual item bank which only had objects (cubes) and difficulty formula. The generation of items and options were done by these two systems, and related sub-systems were described in the following:
3.6.7.1. Item rule definition subsystem:
This subsystem assisted test editors to define “object” and “rule”. Editors could describe “object” and the “rules” of how these “objects” would be generated. From the pictures of this subsystem, we could know that when the subsystem generated “figure
78
test” it could assign the location of an “object”. In addition, it could choose the
test” it could assign the location of an “object”. In addition, it could choose the