The first thing we would like to investigate is the proportion of negative main effect estimates in all cases. Because only the non-negativity of the main effects are particularly addressed in the G-DINA model, therefore the counterintuitive phenomenon occurring at the main effects is easy to detect.
Table 4 underlines the proportions of main effects estimated to be negative being less than 10%, over 20%, even 50% in most case. Interestingly, when test 1 is all composed of the DINA items, the proportion of obtaining negative main effect estimates is relatively high, and in which there are three-quarters of these cases having more than 50% negative estimates. On the other hand, for test 2 all composed of the A-CDM items, the proportion declines obviously and most of these cases only have below 10% of negative estimates. In other words, if a test is all composed of A-CDM items, or allow student to use multiple ways or several key concepts for problem solving, the estimates of main effects are less likely to be negative. In summary, negative estimates are obtained with certain probability when the item parameters are all positive.
Thus, it is useful to develop an estimation algorithm for the G-DINA model with order restriction and hopefully the new method will produce estimates with better property.
The second thing that we are concerned about is the accuracy performance of the proposed estimation methods. We consider three manipulated factors:
percentage of DINA model items, distribution of cognitive pattern, and sample size. According to our design of the Q-matrix, items 1 to10 involve only one attribute, items 11 to 20 involve exactly two attributes, and items 21 to 30 involve exactly three attributes. The three item levels defined in the begin-ning of this section are: the items involving one or more attributes, that is, all of items, the items involving two or more attributes, and the items involving three attributes. We compare the values of MAD and RMSEA of three kinds of method, including the unrestricted method and two restricted method, namely the upward and downward methods.
MAD
Tables 5 to 7 can be broadly inspected in two different ways to serve differ-ent purposes. In order to examine which estimation method has the smallest
Table 4: Proportions of Negative Main Effect in Tests 1 to 3 for Each Distri-bution of Cognitive Patterns and Sample Size
Test 1
Higher-Order MVN 1 MVN 2 Uniform
200 0.53 0.5054 0.5364 0.4974
500 0.5338 0.4966 0.5508 0.5144
1000 0.5232 0.5196 0.5292 0.4994
Test 2
Higher-Order MVN 1 MVN 2 Uniform
200 0.1376 0.064 0.1898 0.0434
500 0.0806 0.0158 0.0964 0.0062
1000 0.0518 0.0038 0.07 0.0024
Test 3
Higher-Order MVN 1 MVN 2 Uniform
200 0.3212 0.285 0.3584 0.2658
500 0.2816 0.2544 0.324 0.256
1000 0.2748 0.2526 0.2846 0.2566
MAD value, we compare the average of MADs under a variety of designs.
Next, the MAD values are compared item by item to examine whether the average of MADs are effected by extreme value or not. Table 5 is for the first item level with items involving one or more attributes (all items), Table 6 is for the second item level containing the items involving two or more attributes, and Table 7 is for the third item level containing items measuring all three attributes exactly. According to Table 5, we can see on average for each test containing various percentage of DINA items, distribution of cognitive pat-tern, or sample size, the upward method has the smallest average of MAD among the three methods, with the except of only one case (test 3, multivari-ate normal with higher correlation, sample size of 200), denoted with a shaded gray representation. If we compare the MAD values item by item, the upward method in most cases produces the largest number of items with the smallest MAD. However, there are some of cases (10 in a total of 36 cases), estimates with no order restriction having the largest number of items with the smallest MAD.
Table 6 and Table 5 are different because Table 6 is made for the second item level and therefore only accounted for the items involving two or more attributes. That is, MAD for items 11 to 30, not all. On average, the upward method has the smallest MAD in each case. Moreover, when we compare the MAD values item by item, the number of items having the smallest MAD by the unrestricted method is substantially reduced. That is, in Table 5, the items have the smallest MAD of estimate by the unrestricted method are mostly the items involving only one attribute. Table 7 is for the third item level and there-fore considers only the items involving three attributes. That is, only items 21
Table5:PerformanceofMAD:OverallMeanMADofAllItemsandtheNumberofItemswiththeSmallestValue MADTest1Test2Test3 HOMVN1MVN2UnifHOMVN1MVN2UnifHOMVN1MVN2Unif 200 meanovernorestriction0.09330.07150.10480.06450.14600.11550.15940.09710.11540.09250.13210.0789 MADofitemsupwardmethod0.06650.04610.09790.04140.12550.10070.14300.08930.11370.07500.13620.0630 1-30downwardmethod0.10880.08150.11360.06610.14200.11020.15850.09490.12210.09760.13790.0777 numberofnorestriction87113148163187176 itemshasminupwardmethod2223102711141221919719 MADdownwardmethod009058263465 500 meanovernorestriction0.06280.04590.07510.04040.09960.06700.11280.05580.08030.05630.09530.0475 MADofitemsupwardmethod0.04270.02900.05850.02520.08680.06390.10570.05490.06840.04640.09370.0397 1-30downwardmethod0.07910.05430.09090.04150.10280.06650.12300.05550.09590.06210.11150.0483 numberofnorestriction751051311182125147 itemshasminupwardmethod21251324151511191422917 MADdownwardmethod207124194376 1000 meanovernorestriction0.04340.03110.05480.02810.07320.04520.08730.03860.05770.03790.06960.0329 MADofitemsupwardmethod0.02620.01920.03510.01820.06500.04420.08110.03840.04550.03130.06310.0276 1-30downwardmethod0.06100.03670.07470.02860.07460.04500.09730.07460.07310.04100.09340.0333 numberofnorestriction55851761610119146 itemshasminupwardmethod252320221317141817181320 MADdownwardmethod022307022334
Table6:PerformanceofMAD:MeanMADofItems11to30andtheNumberofItemswiththeSmallestValue MADTest1Test2Test3 HOMVN1MVN2UnifHOMVN1MVN2UnifHOMVN1MVN2Unif 200 meanovernorestriction0.11820.08730.13580.07660.19640.14930.21730.12300.15230.11780.17800.0975 MADofitemsupwardmethod0.07650.04890.12040.04210.15860.12600.18040.11140.14290.09030.17370.0738 11-30downwardmethod0.13340.09870.14200.07800.18720.14240.21090.11990.15790.12350.18180.0955 numberofnorestriction001054618172 itemshasminupwardmethod2020102011131218916716 MADdownwardmethod009043213362 500 meanovernorestriction0.08110.05560.09950.04860.13630.08680.15650.07040.10780.07140.13030.0588 MADofitemsupwardmethod0.05000.03020.07240.02570.11350.08180.13900.06910.08800.05660.12210.0471 11-30downwardmethod0.10160.06720.11850.05000.13890.08600.16860.06990.12800.07950.15060.0600 numberofnorestriction000057915262 itemshasminupwardmethod18201320141111131317912 MADdownwardmethod207012062156 1000 meanovernorestriction0.05600.03760.07300.03340.10070.05820.12270.04890.07750.04780.09560.0406 MADofitemsupwardmethod0.03010.01980.04290.01850.08700.05680.10880.04860.05860.03800.08330.0327 11-30downwardmethod0.07980.04570.09950.03410.10190.05800.13620.10190.09890.05230.12850.0412 numberofnorestriction000083743343 itemshasminupwardmethod202018201213131615151316 MADdownwardmethod002004002231
Table7:PerformanceofMAD:MeanMADofItems21to30andtheNumberofItemswiththeSmallestValue MADTest1Test2Test3 HOMVN1MVN2UnifHOMVN1MVN2UnifHOMVN1MVN2Unif 200 meanovernorestriction0.15460.10430.17600.08990.26730.20230.29040.16020.20350.15130.23100.1231 MADofitemsupwardmethodd0.09020.04850.13390.03950.18500.15610.20330.13880.16250.10510.18840.0860 21-30downwardmethod0.19160.13410.20170.10160.24550.18970.26700.15250.21760.16730.23980.1243 numberofnorestriction000000003040 itemshasminupwardmethod1010101010101010710610 MADdownwardmethod000000000000 500 meanovernorestriction0.11090.06610.12960.05770.19260.11750.22070.09060.15380.09170.17680.0729 MADofitemsupwardmethod0.05870.03000.08120.02470.14390.10680.16500.08800.10970.06870.14020.0554 21-30downwardmethod0.15750.09570.18650.06710.18830.11570.22140.08960.18810.10980.21080.0788 numberofnorestriction000000000031 itemshasminupwardmethod101010101010109101077 MADdownwardmethod000000010002 1000 meanovernorestriction0.07590.04520.09790.03990.14960.07750.17530.06260.11270.06120.13310.0510 MADofitemsupwardmethod0.03380.01940.05060.01790.11890.07460.13660.06200.07780.04600.09930.0391 21-30downwardmethod0.12720.06600.16340.04540.14740.07700.18700.14740.14990.07230.18420.0540 numberofnorestriction000000020001 itemshasminupwardmethod10101010108108109108 MADdownwardmethod000002000101
to 30 are used to compute MAD. In this case, comparing either the average of MADs or MAD values item by item, the upward method has the best perfor-mance. That is, upward method produces the smallest average of MADs and the largest number of items having the smallest MAD. Furthermore, when the items we considered that involve attributes increase, the difference of average of MAD between no order restriction and upward method is more significant.
Next, we examine the effects of each factor on parameter estimates of the three methods using Table 5.
Type of Items. Recall that the items in test 1 are all DINA items, the items in test 2 are all A-CDM items, and test 3 contains both types of items.
We can see that when the percentage of DINA items in a test increases, the average of MAD decreases.
Distribution of cognitive pattern. Among the four kinds of distribution of cognitive pattern, the discrete uniform distribution mostly has the smallest average of MAD, the next is the multivariate normal distribution with lower correlation, followed by higher-order distribution, and lastly the multivariate normal distribution with higher correlation has the largest average of MAD.
With Figure 1, we found that if the cognitive pattern is very concentrated in a few classes, the average of MAD is also relatively large.
Sample size. In the case of holding the other factors constant, the av-erage of MAD become smaller as sample size increases. That is, as expected the larger the sample size, the more accurate the estimates are.
In addition, comparing the MAD values of the same position of in Ta-ble 5 (take into account all items), TaTa-ble 6 (take into account items 11-30), and Table 7 (take into account items 21-30) under various item level, the re-sults show that that the first item level has the smallest average of MAD, and the third item level has the largest average of MAD. For example, in the case of test 1, cognitive pattern resulting from the higher-order distribution, sample size of 200, and using the unrestricted method, the averages of MAD are respectively 0.0933 in Table 5, 0.1182 in Table 6, and 0.1546 in Table 7.
In other words, when the number of attributes the items involve increases, the average of MAD also becomes larger.
RMSEA
The construction of Table 8 to 10 is the same as Table 5 to 7, but con-sidering the index RMSEA in this part. In is shown in Table 8 for the first item level that no matter what the design, the upward method almost always has the smallest average of RMSEA, except only two case (test 1, multivariate normal with higher correlation, sample size of 200 and test 3, multivariate nor-mal with higher correlation, sample size of 200). If we compare the RMSEA
values item by item, the method that has the largest number of RMSEA is not the upward method only in a small part of designs (8 of all 36 designs).
In the majority of cases, the upward method has the largest number of items with the smallest RMSEA. According to Table 9 for the second item level, the upward method almost always yields the largest number of items having the smallest RMSEA, with the except of only two case (test 1, multivariate normal with higher correlation, sample size of 200 and test 3, multivariate nor-mal with higher correlation, sample size of 500) that the unrestricted method having the smallest RMSEA. Table 10 presents the results for the third item level and they are almost identical with Table 10 that regardless of the design, the upward method has both the smallest average RMSEA and the largest number of items with smallest RMSEA.
In addition, the effects of each factor on RMSEA of the estimates are also examined. When the percentage of DINA items in a test increases, the average of RMSEA decreases. The discrete uniform distribution of cognitive patterns has the smallest average of RMSEA, followed by the multivariate normal with lower correlation, the higher-order distribution, and lastly the multivariate normal distribution with higher correlation having the larger average of RM-SEA. With increasing sample size, the average of RMSEA becomes smaller.
On the other hand, comparing RMSEA values on the same position of Tables 8 to 10, we also found the value in Table 8 the smallest and that in Table 10 the largest. In other words, the average of RMSEA becomes large when the num-ber of attributes the items involving increases. The results on RMSEA display consistent findings with those obtained from MAD. Another point worth not-ing is that, with respect to either the average or largest number of items on MAD or RMSEA, the estimates with order restrictions always perform better than the unrestricted ones when we only examine items involving two or more attributes.
Classification Accuracy P
caFinally, we compare the classification accuracy of the three methods in this study. From Table 11, classification accuracy index values of the upward and downward methods are shown to be higher than the estimation without order restrictions. We also find that the upward method has the highest classifica-tion accuracy among the three methods in test 1, and the downward method has the highest classification accuracy when the sample size is less than 500 in test 2, except when the distribution of cognitive patterns is discrete uniform.
When the sample size is large enough (more than 1000), the upward method has the highest classification accuracy in most cases, except for only one case (test 2, multivariate normal distribution with higher correlation). In addition, the impacts of manipulated factors for classification accuracy are less salient.
Table8:PerformanceofRMSEA:OverallMeanRMSEAofAllItemsandtheNumberofItemswiththeSmallestValue RMSEATest1Test2Test3 HOMVN1MVN2UnifHOMVN1MVN2UnifHOMVN1MVN2Unif 200 meanovernorestriction0.10470.08050.12540.07410.16970.13320.19220.11490.13080.10570.15630.0918 RMSEAofitemsupwardmethod0.09390.05980.14160.05240.12580.11010.14250.10420.12820.08610.15760.0752 1-30downwardmethod0.12910.09490.14050.07810.17120.13010.20120.11320.14450.11370.16870.0918 numberofnorestriction106191126141167186 itemshasminupwardmethod20242291720152210231022 RMSEAdownwardmethod009014174022 500 meanovernorestriction0.06940.05140.08820.04640.11290.07750.13580.06620.08870.06430.11330.0556 RMSEAofitemsupwardmethod0.06150.03720.08590.03190.08970.07250.10700.06490.07880.05460.10930.0479 1-30downwardmethod0.09140.06240.10900.04840.11790.07710.15380.06590.10950.07140.13400.0569 numberofnorestriction84134128183145197 itemshasminupwardmethod21261026181911191422621 RMSEAdownwardmethod107003182352 1000 meanovernorestriction0.04770.03500.06340.03240.08120.05240.10500.04610.06340.04300.08180.0385 RMSEAofitemsupwardmethod0.03700.02460.05320.02300.06780.05100.08370.04580.05320.03710.07470.0334 1-30downwardmethod0.06970.04170.08780.03330.08310.05230.11950.08310.08120.04690.11050.0390 numberofnorestriction35861741791010186 itemshasminupwardmethod27232021131813191917822 RMSEAdownwardmethod022308021342
Table9:PerformanceofRMSEA:MeanRMSEAofItems11to30andtheNumberofItemswiththeSmallestValue RMSEATest1Test2Test3 HOMVN1MVN2UnifHOMVN1MVN2UnifHOMVN1MVN2Unif 200 meanovernorestriction0.13400.09950.16550.08930.23090.17450.26500.14790.17430.13650.21320.1153 RMSEAofitemsupwardmethod0.11640.06830.18460.05720.15730.13850.17690.13200.16290.10550.20400.0907 11-30downwardmethod0.16050.11670.17890.09410.22870.17060.27190.14540.18880.14580.22510.1149 numberofnorestriction209032416081 itemshasminupwardmethod18202201718151810201019 RMSEAdownwardmethod009000114020 500 meanovernorestriction0.09030.06310.11840.05660.15550.10170.19030.08480.11950.08260.15650.0701 RMSEAofitemsupwardmethod0.07740.04180.11280.03490.11660.09390.13910.08300.10260.06800.14410.0584 11-30downwardmethod0.11840.07820.14360.05930.16010.10100.21320.08440.14670.09260.18270.0718 numberofnorestriction2030349061112 itemshasminupwardmethod17201020171511141318616 RMSEAdownwardmethod107001061132 1000 meanovernorestriction0.06200.04300.08550.03910.11220.06860.14870.05940.08550.05500.11340.0483 RMSEAofitemsupwardmethod0.04570.02740.06950.02500.09040.06640.11130.05890.06950.04610.09970.0407 11-30downwardmethod0.09180.05270.11790.04040.11400.06830.16880.11400.11010.06060.15310.0492 numberofnorestriction000083744381 itemshasminupwardmethod20201820121413161515817 RMSEAdownwardmethod002003001242
Table10:PerformanceofRMSEA:MeanRMSEAofItems21to30andtheNumberofItemswiththeSmallestValue RMSEATest1Test2Test3 HOMVN1MVN2UnifHOMVN1MVN2UnifHOMVN1MVN2Unif 200 meanovernorestriction0.17830.12140.22410.10660.32250.24190.36530.19660.23890.17930.28590.1483 RMSEAofitemsupwardmethod0.15550.07630.24280.05820.18310.17320.19890.16720.20090.12670.24250.1083 21-30downwardmethod0.22660.15650.25400.12130.30850.23350.35760.18920.26260.19870.30120.1505 numberofnorestriction205000004050 itemshasminupwardmethod81021010101010610510 RMSEAdownwardmethod003000000000 500 meanovernorestriction0.12480.07620.15950.06820.22320.14050.27690.11150.17250.10770.21940.0886 RMSEAofitemsupwardmethod0.10440.04560.14980.03580.14750.12440.16340.10780.13400.08450.17770.0706 21-30downwardmethod0.17990.10890.22380.07840.21830.13880.28910.11050.21430.12790.25720.0946 numberofnorestriction003000002050 itemshasminupwardmethod10107101010109810510 RMSEAdownwardmethod000000010000 1000 meanovernorestriction0.08470.05240.11720.04730.16780.09320.21890.07790.12520.07190.16190.0617 RMSEAofitemsupwardmethod0.05920.02980.09530.02600.12170.08900.13950.07690.09450.05750.12520.0498 21-30downwardmethod0.14340.07470.19120.05310.16490.09270.23770.16490.16520.08400.21980.0647 numberofnorestriction000000020031 itemshasminupwardmethod1010101010910810978 RMSEAdownwardmethod000001000101
Table11:PerformanceofClassificationAccuracyoftheThreeMethodsforEachDistributionofCognitivePatterns,Typeof Items,andSampleSize Test1Test2Test3 HOMVN1MVN2UnifHOMVN1MVN2UnifHOMVN1MVN2Unif 200 norestriction0.85060.84040.86280.83220.80360.76290.82690.73040.86020.83400.87140.8049 upwardmethod0.97210.96540.96050.95420.88730.91430.86630.93170.90880.94080.89250.9555 downwardmethod0.94640.94120.94010.93940.92340.93470.88840.92910.92580.94290.91510.9471 500 norestriction0.87140.85200.88670.85300.84090.79870.86210.77680.88190.85190.89410.8308 upwardmethod0.97850.97240.97490.96030.92950.94740.89360.94370.95770.96700.91540.9601 downwardmethod0.96480.96380.95480.95700.96530.95060.93320.94350.96000.96190.93750.9585 1000 norectriction0.87670.86450.89180.85860.85570.81220.87140.78720.88970.85810.90190.8383 upwardmethod0.98870.97460.98200.96220.94980.95470.90740.94570.97700.96900.94770.9616 downwardmethod0.98350.97200.96250.96140.92340.93470.95830.92910.92580.94290.91510.9471
5. Analysis of Empirical Data
In addition to the simulation study, we also apply and illustrate the pro-posed method in analyzing empirical data. The data are the mathematical stage test from a senior high school in Taipei and the percentile rank of the students in this school is higher than 96. The test responses of 309 students in seven classes were collected. As shown in Table 12, the Q-matrix of this test contains 11 items and 9 attributes. Some items in the test belong to the DINA model items and some are A-CDM items, so these test data are rela-tively similar to the test 3 in our simulation.
Table 12: Q-matrix for the Empirical Data
item K1 K2 K3 K4 K5 K6 K7 K8 K9
Tables 13 and 14 respectively report the probability of getting correct an-swers for each corresponding cognitive pattern and the parameter estimates of the G-DINA model obtained without order restrictions. We can easily see that find some counterintuitive phenomenon in Table 13. For the example of item 3, the probability of getting a correct answer for respondents with the cognitive pattern of mastering none of the attribute item 3 involving is 0.52, whereas the probability reduces to 0.33 for respondents with the cognitive patterns of mastering only the first attribute of item 3. In other words, we have for item 3, (1, 0, 0, 0) ≻ (0, 0, 0, 0) but P1000 < P0000. Take another example of item 7, the probability of getting a correct answer for cognitive pattern (1, 0, 0) is 1.00, but the the probability is only 0.94 for the cognitive pattern (1, 1, 1). In other words, we have for item 7, (1, 1, 1) ≻ (1, 0, 0) but P111 < P100. All the counterintuitive phenomenon in Table 13 are marked in shaded gray. Use the relationship between δj and Pj mentioned in section 3, we present the results of δj in Table 14, for j = 1, 2,· · · , 11 to help us directly and conveniently observe the main effects. From Table 14, we find that items 3, 8, and 10 have negative main effect estimates, and the proportion of negative main effects is slightly less than test 3 in Table 4. In addition, even for respondents who have mastered all the attributes that items 4 and 5 involve, the probabilities
of answering these items correctly are less than satisfactory. This could prob-ably result from the fact that these items are so-called application questions and respondents might have mastered all the attributes but do not know how to apply them in solving a problem, therefore, the probability of answering correctly appears to be lower than expected for those who have mastered all the attributes.
According to the simulation results in section 4, the upward method per-forms better than the downward method. Therefore, we compare only the results obtained from the method without order restrictions and the upward method while analyzing the empirical data. The resulting probability of an-swering correctly Pj are reported in Table 15 and the estimates of δj are reported in Table 16, for j = 1, 2, . . . , 11. We can see that the counterintuitive phenomenon in Table 13 has been solved here. In Table 15, the probabilities fully meet the order restrictions, Pj(αl(j)) ≥ Pjαl′(j) when αl(j) ≻ αl′(j), for j = 1, 2, ..., 11. We note that there exists no negative main effect estimates in Table 16.
Table13:EstimatedProbabilityofAnsweringCorrectlyforEachCognitivePatternoftheG-DINAModelwithoutOrder RestrictionsfortheEmpiricalData Attributepattern P0P1 P00P10P01P11 P000P100P010P001P110P101P011P111 itemP0000P1000P0100P0010P0001P1100P1010P1001P0110P0101P0011P1110P1101P1011P0111P1111 10.140.750.580.141.000.800.751.000.580.880.510.800.080.000.660.95 20.000.210.000.000.611.000.211.000.000.610.611.000.000.950.611.00 30.520.330.520.500.000.330.850.660.500.000.871.000.660.810.871.00 40.050.130.050.45 50.000.000.000.62 60.050.440.050.550.751.000.870.960.550.750.000.940.921.000.001.00 70.171.000.170.171.000.820.170.94 80.050.640.000.350.000.001.001.00 90.400.940.400.400.280.000.401.00 100.270.001.001.00 110.001.000.000.001.000.470.051.00
Table14:ParameterEstimatesoftheG-DINAModelwithoutOrderRestrictionsfortheEmpiricalData Attributepattern δ0δ1 δ00δ10δ01δ11 δ000δ100δ010δ001δ110δ101δ011δ111 itemδ0000δ1000δ0100δ0010δ0001δ1100δ1010δ1001δ0110δ0101δ0011δ1110δ1101δ1011δ0111δ1111 10.140.610.440.000.86-0.390.00-0.610.00-0.55-0.490.00-0.42-0.510.271.60 20.000.210.000.000.610.790.000.170.000.000.000.00-1.79-0.050.001.05 30.52-0.190.00-0.02-0.520.000.540.850.000.000.900.150.00-1.270.000.04 40.050.080.000.33 50.000.000.000.62 60.050.390.000.500.700.56-0.07-0.180.000.00-1.25-0.50-0.600.860.000.53 70.170.830.000.000.00-0.180.000.13 80.050.58-0.050.29-0.58-0.930.710.93 90.400.540.000.00-0.66-0.940.001.66 100.27-0.270.730.27 110.001.000.000.000.00-0.530.050.48
Table15:EstimatedProbabilityofAnsweringCorrectlyforEachCognitivePatternoftheG-DINAModelusingUpwardMethod fortheEmpiricalData Attributepattern P0P1 P00P10P01P11 P000P100P010P001P110P101P011P111 itemP0000P1000P0100P0010P0001P1100P1010P1001P0110P0101P0011P1110P1101P1011P0111P1111 10.041.000.630.040.041.001.001.000.630.690.041.001.001.000.691.00 20.000.440.000.001.000.480.441.000.001.001.000.481.001.001.001.00 30.560.610.560.560.560.611.000.800.560.560.661.000.801.000.661.00 40.090.140.090.54 50.020.080.020.70 60.090.950.090.570.251.000.951.000.570.250.571.001.001.000.571.00 70.350.800.350.351.000.840.351.00 80.060.540.210.500.540.540.501.00 90.330.670.330.330.670.670.331.00 100.280.281.001.00 110.001.000.000.011.001.000.391.00
Table16:ParameterEstimatesoftheG-DINAModelusingUpwardMethodfortheEmpiricalData Attributepattern δ0δ1 δ00δ10δ01δ11 δ000δ100δ010δ001δ110δ101δ011δ111 itemδ0000δ1000δ0100δ0010δ0001δ1100δ1010δ1001δ0110δ0101δ0011δ1110δ1101δ1011δ0111δ1111 10.040.960.600.000.00-0.590.000.000.000.050.000.00-0.050.000.000.00 20.000.440.000.001.000.030.00-0.440.000.000.000.00-0.030.000.000.00 30.560.050.000.000.000.000.390.180.000.000.100.000.00-0.280.000.00 40.090.050.000.40 50.020.050.000.63 60.090.850.000.470.160.05-0.47-0.110.000.00-0.160.00-0.050.160.000.00 70.350.450.000.000.200.040.00-0.04 80.060.480.150.44-0.15-0.44-0.150.61 90.330.350.000.000.000.000.000.33 100.280.000.720.00 110.001.000.000.010.00-0.010.39-0.39
6. Discussion and Conclusion
The purpose of this study is to propose an improved algorithm for the estimation of the G-DINA model to avoid the counter-intuitive phenomena that a lower probability of getting a correct answer is obtained for the students who have mastered more attributes than those who know less. We evaluate the performance of our proposed algorithms through a simulation study of which a number of factors were manipulated, including item type, distribution of the cognitive patterns, and sample size.
First of all, we can almost affirm that such a counter-intuitive phenomenon would not occur on the items which intuitively involve only one attribute.
That is, for items involving only one attribute, there is no need to consider the order restrictions. Because the Q-matrix describes the relationship between the items and the attributes, the above counter-intuitive phenomenon occurring on an item involving only one attribute indicates that mastering this sole attribute in fact decreases the probability of answering the item correctly. This would imply that the attribute is very likely to be mistakenly identified as associated with the item in the Q-matrix. That is, there are factors other than the attribute affecting the probability of getting a correct answer for that item.
Our results show that if we focus on the items involving two or more at-tributes, the estimates obtained from the upward method perform the best for both MAD and RMSEA regardless of the factorial conditions. The estimates from the upward method not only on average have smaller MAD and RMSEA than the other two methods, but also have the smallest MAD and RMSEA in most of the items involving three attributes. In other words, the greater the number of attributes an item involves, the more salient the advantages of using the upward method for estimation is. As for classification accuracy, although the classification accuracy of the downward method is higher than the upward method in some cases, the upward method is still better than estimation with-out restrictions in all cases. If the researcher accepts the classification accuracy of the existent estimation method without restrictions, they should find the classification accuracy of the upward method satisfactory. Moreover, the con-vergence of the upward method does not take relatively longer time compared to that with no order restrictions. Especially when all the estimates naturally fall in the permissible parameter space of the order restrictions, the same max-imum likelihood estimates will be obtained and adding the order restrictions in the estimation algorithm has no effect on the estimates. In summary, if the Q-matrix of a test contains items involving two or more attributes, it is recommended to use the G-DINA model with order restrictions and estimate the parameters with the upward method.
In fact, the problem we have raised and discussed in this study is not just for the G-DINA model, it should be applicable to some other CDMs such as the log linear cognitive diagnosis model (Henson, Templin, & Willse, 2009; Rupp &
Templin, 2008; von Davier, 2005). Although the “monotonicity” property that respondents with a superior attribute pattern should have a larger probability
of answering correctly is often mentioned as necessary constraints, it is unclear whether and how they are incorporated into the algorithms or procedures for parameter estimation (de la Torre, 2011). Therefore, our algorithm provides a possible modification on the existent algorithms for the CDMs model when-ever some order restrictions should be employed upon estimating the model parameters.
It is noteworthy that If the items of a test belong to the category of the DINA model items, the proportion of main effects estimated negatively will be relatively high. Because the parameter settings in test 1 is met by the DINA model in the sense that only the respondents who have mastered all the attributes that item involves will have a high probability to answer correctly.
Therefore, the estimates of the main effect parameters would be close to 0 for the DINA model items. Thus, obtaining a small negative estimate for main effect would give some information and suggest fitting the DINA model instead or implies that the Q-matrix in use might be doubtful. With our proposed algorithm, the main effects are forced to be non-negative under order restrictions, either the model with some equality constraints of zeros or possible misspecification of Q-matrix will become harder to detect. Therefore, it is
Therefore, the estimates of the main effect parameters would be close to 0 for the DINA model items. Thus, obtaining a small negative estimate for main effect would give some information and suggest fitting the DINA model instead or implies that the Q-matrix in use might be doubtful. With our proposed algorithm, the main effects are forced to be non-negative under order restrictions, either the model with some equality constraints of zeros or possible misspecification of Q-matrix will become harder to detect. Therefore, it is