Overspecification

In the overspecification setting, we consider the situation that although only at-tribute 4, not 5, is associated with the mastery of an item, atat-tribute 5 is mistakenly included whenever attribute 4 is present. As a result, the misspecified items in our Q-matrix are item 4, 9, 13, 16, 18, 22, and 27. In the true Q-matrix, items 4 and 9 measure one attribute (only attribute 4), items 13 and 16 measure two attributes, and items 18 and 22 measure three attributes.

Table 4: Overspecification on the Q-matrix

item A1 A2 A3 A4 A5 item A1 A2 A3 A4 A5

It is important to point out that although these two different conditions led to different number of times some items being assessed in the test, the average number of items measured by an attribute and the average number of attributes measured by an item both remained approximately the same in the misspecified Q-matrix com-pared to the true Q-matrix. This is called a balanced design, which is expected to offer more robust results across various simulation conditions. Balanced design have been considered in researches relating to Q-matrix misspecification (Rupp & Templin, 2008; Kunina-Habenicht, Rupp, & Wilhelm, 2012). Furthermore, enough one-attribute item(more than three) is crucial for maintaining the stability of parameter estimates and classification accuracy, this is also considered in our condition setting.

4 Results

In this section, we firstly report the overall impact of Q-matrix misspecification on parameter estimates. Since the results for different sample sizes are similar, we report and interpret the results for sample size of 1000 under the four distributions of attribute patterns.

Tables 5 to 8 report the results for ˆδ_j and Tables 9 to 12 are for ˆP_j. Furthermore, to understand how the manipulated factors influence the effect of Q-matrix misspec-ification, we examine the interaction effect of the type of misspecification and other factors on parameter estimates. More specifically, two-way interaction plots of the type of misspecification and sample size, and the type of misspecification and distribution of the attribute patterns are depicted in Figures 2 and 3, respectively.

To better understand how the type of Q-matrix misspecification interacts with the manipulated factors on attribute-specific classification accuracy, we depict Figure 4 to show the attribute-specific classification accuracy with respect to different types of Q-matrix misspecification for four distributions of cognitive attribute patterns. Again, we only report the cases with sample size of 1000 because the similar results are obtained for cases with different sample sizes.

Tables 13 and 14 present the results for the overall classification accuracy, and we also give the interaction plot between the type of misspecification and the manipulated factors on the overall classification accuracy, as shown in Figure 5.

4.1 Effect on parameter estimates

Firstly, we focus on the effect of misspecification on the quality of item parameter estimates. Tables 5 to 8 present the MAD’s of each ˆδ_j in the G-DINA model within the underspecification condition with sample size of 1000. A great impact of underspecified Q-matrix on parameter recovery or estimation bias are shown for those misspecified items (items 20, 26, 29, and 30). For these items, MAD values of some parameter estimates related to the fourth attribute are greatly magnified. For instance, as shown in Table 5 for simulated data with sample size of 1000 and the distribution of attribute patterns is discrete uniform, the MAD values for ˆδ_20,4of item 20 with q₂₀= (00011) → (00010) is 0.409, for ˆδ_26,14 of item 26 with q₂₆ = (10011) → (10010) is 0.401, for ˆδ_29,24

of item 29 with q₂₉ = (01011) → (01010) is 0.445, and ˆδ_30,34 of item 30 with q₃₀ = (00111) → (00110) is 0.299. The common characteristic of these items is that all of these magnified estimates are for the highest interaction parameter for the misspecified item. On the other hand, there are no MAD values greater than 0.1 for those estimates of non-misspecified item parameter.

In contrast, recovery of item parameter estimates shows little impact on the item parameter estimates under the overspecification condition. The MAD values under this condition are all smaller than 0.1, even for those misspecified items. In other words, parameter estimates appear to be unaffected by overspecification of the Q-matrix.

To investigate the possible interaction effects between the type of Q-matrix mis-specification and other factors on parameter estimates, we first look at the interaction between the type of misspecification and sample size. Figure 2 shows an interaction plot for the type of Q-matrix misspecification and sample size on the mean MAD val-ues of the parameter estimates of those misspecified items. More specifically, the mean MAD value of items 20, 26, 29, and 30 are computed for the comparison between the underspecification and true Q-matrix specification conditions. For the effect of over-specification, the mean MAD value of items 4, 9, 13, 16, 18, 22, and 27 are computed for comparison.

We find that the mean MAD values become smaller as the sample size increases within the underspecification condition, whereas those of the three levels of sample size under the overspecification condition are nearly the same. That is, the factor of sample size only has an impact on the effect of Q-matrix underspecification. In fact, similar interaction patterns are also present for the other three distributions of attribute patterns and therefore they are omitted for brevity.

Secondly, Figure 3 presents the interaction plot for the type of Q-matrix misspec-ification and the distribution of cognitive patterns on the mean MAD values of the parameter estimates of those misspecified items. The results show that the MAD values are the largest for higher-order distribution, followed by multivariate normal distribution with ρ = 0.3 and ρ = 0.8, and lastly the uniform distribution. In other words, the distribution of discrete uniform for the attribute patterns results in the best parameter recovery. This phenomenon remains the same for three levels of sample size.

By conducting an three-factor analysis of variance for type of misspecification, sample size and distribution of cognitive patterns, no significant three-way interaction effect is

Figure 2: The interaction plots of the type of Q-matrix misspecification and sample size on the mean MAD values of the parameter estimates of those misspecified items

found (p = 0.88).

We also look at Tables 5 to 8 to investigate whether MADs differ for items with different numbers of attribute and its parameter type. The results show that the MAD values are the highest for those items measuring three attributes, and the lowest for those measuring only one attribute. Moreover, the MADs of the three types of item parameters i.e. the intercept, the main effect, and the two-way and the three-way interaction parameters are also examined. We can see that the MADs are much greater for the estimates of higher-order interaction parameters of the misspecified items 26, 29 and 30. Similarly, the MADs for the main effects of item 20 are higher than expected with the underspecification that its true number of measuring attributes of two are misspecified to be only one.

Figure 3: The interaction plots of the type of Q-matrix misspecification and the underlying distribution of attribute patterns on the mean MAD values of the parameter estimates of those misspecified items

Table 5: The MADs of ˆδj for Q-matrix underspecification, discrete uniform distribution of attribute patterns, and sample size of 1000

item δ₀ δ₁^∗ q_j 21 0.002 0.004 0.003 0.003 0.006 0.005 0.005 0.007 (11100) 22 0.002 0.002 0.003 0.005 0.006 0.006 0.006 0.010 (11010) 23 0.005 0.006 0.008 0.009 0.012 0.011 0.014 0.016 (11001) 24 0.003 0.004 0.004 0.006 0.006 0.007 0.007 0.009 (10110) 25 0.005 0.006 0.006 0.009 0.009 0.010 0.012 0.015 (10101) 26 0.018 0.023 0.026 - 0.401 - - - (10011)

27 0.003 0.004 0.003 0.005 0.005 0.008 0.007 0.010 (01110) 28 0.005 0.007 0.007 0.009 0.011 0.012 0.010 0.016 (01101) 29 0.031 0.030 0.061 - 0.445 - - - (01011)

30 0.034 0.049 0.061 - 0.299 - - - (00111) δ₀: The intercept parameter;

δ^∗_k: The main effect parameter of the kth attribute of that item;

δ^∗

kk⁰: The interaction parameter of the kth and k⁰th attributes of that item.

Table 6: The MADs of ˆδj for Q-matrix underspecification, underlying multivariate normal distribution with ρ = 0.3 of attribute patterns, and sample size of 1000

item δ₀ δ₁^∗ q_j 21 0.002 0.005 0.004 0.002 0.008 0.007 0.005 0.010 (11100) 22 0.003 0.007 0.005 0.006 0.018 0.011 0.010 0.023 (11010) 23 0.004 0.012 0.011 0.006 0.023 0.014 0.014 0.025 (11001) 24 0.003 0.004 0.004 0.006 0.006 0.007 0.007 0.009 (10110) 25 0.005 0.008 0.005 0.005 0.012 0.013 0.007 0.016 (10101) 26 0.018 0.046 0.039 - 0.577 - - - (10011)

27 0.003 0.008 0.006 0.005 0.012 0.012 0.009 0.016 (01110) 28 0.005 0.015 0.009 0.008 0.022 0.018 0.016 0.027 (01101) 29 0.039 0.048 0.074 - 0.524 - - - (01011)

30 0.040 0.060 0.083 - 0.595 - - - (00111) δ₀: The intercept parameter;

δ^∗_k: The main effect parameter of the kth attribute of that item;

δ^∗

kk⁰: The interaction parameter of the kth and k⁰th attributes of that item.

Table 7: The MADs of ˆδj for Q-matrix underspecification, underlying multivariate normal distribution with ρ = 0.8 of attribute patterns, and sample size of 1000

item δ₀ δ₁^∗ q_j 21 0.002 0.004 0.003 0.003 0.006 0.005 0.005 0.007 (11100) 22 0.002 0.002 0.003 0.005 0.006 0.006 0.006 0.010 (11010) 23 0.005 0.006 0.008 0.009 0.012 0.011 0.014 0.016 (11001) 24 0.003 0.004 0.004 0.006 0.006 0.007 0.007 0.009 (10110) 25 0.005 0.006 0.006 0.009 0.009 0.010 0.012 0.015 (10101) 26 0.018 0.023 0.026 - 0.593 - - - (10011)

27 0.003 0.004 0.003 0.005 0.005 0.008 0.007 0.010 (01110) 28 0.005 0.007 0.007 0.009 0.011 0.012 0.010 0.016 (01101) 29 0.031 0.030 0.061 - 0.547 - - - (01011)

30 0.034 0.049 0.061 - 0.563 - - - (00111) δ₀: The intercept parameter;

δ^∗_k: The main effect parameter of the kth attribute of that item;

δ^∗

kk⁰: The interaction parameter of the kth and k⁰th attributes of that item.

Table 8: The MADs of ˆδj for Q-matrix underspecification, underlying higher-order multi-variate normal distribution of attribute patterns, and sample size of 1000

item δ₀ δ₁^∗ q_j 21 0.001 0.020 0.007 0.003 0.028 0.023 0.009 0.030 (11100) 22 0.002 0.038 0.008 0.004 0.053 0.042 0.012 0.055 (11010) 23 0.003 0.035 0.021 0.005 0.048 0.036 0.024 0.050 (11001) 24 0.004 0.029 0.004 0.005 0.038 0.033 0.007 0.044 (10110) 25 0.003 0.056 0.011 0.005 0.066 0.058 0.011 0.068 (10101) 26 0.015 0.021 0.030 - 0.747 - - - (10011)

27 0.002 0.023 0.007 0.005 0.030 0.028 0.011 0.037 (01110) 28 0.005 0.030 0.011 0.007 0.031 0.033 0.014 0.036 (01101) 29 0.039 0.100 0.071 - 0.787 - - - (01011)

30 0.036 0.064 0.075 - 0.723 - - - (00111) δ₀: The intercept parameter;

δ^∗_k: The main effect parameter of the kth attribute of that item;

δ^∗

kk⁰: The interaction parameter of the kth and k⁰th attributes of that item.

Since the parameter estimates ˆδ_j can be one-to-one transformed to or from the probability for correctly answering an item, namely ˆP_j, we also analyze ˆP_j to under-stand how the probabilities of correctly answering an item are influenced by the two types of Q-matrix misspecification for respondents with the attribute patterns related to the misspecified item. Firstly, under the Q-matrix underspecification, the discrete uniform distribution of attribute patterns, and a sample size of 1000, the empirical bias of the probability of correctly answering for respondents with attribute pattern (00010), denoted by EB_P20(00010), was 0.371 for item 20, EB_P26(10010) was 0.380 for item 26, EB_P29(01010) was 0.384 for item 29, and EB_P30(00110) was 0.339 for item 30.

On the other hand, we also found that Q-matrix underspecification caused con-siderable decrease in the probabilities of answering correctly for those attribute pat-terns mastering both the fourth and the fifth attributes, such as the empirical bias for P_20(00011) for item 20 was -0.401, EB_P26(10011) for item 26 was -0.446, EB_P29(01011) for item 29 was -0.381, and EB_P30(00111) for item 30 was -0.363. In Tables 9 to 12, we list all the affected EB_Pj(m) values in accordance with the corresponding attribute pattern due to Q-matrix underspecification. The symbol EB₀ represents the empirical bias of parameter estimates under the true Q-matrix averaged over 100 replications, and the symbol EB_q represents the empirical bias of parameter estimates under Q-matrix misspecification averaged over 100 replications.

When Q-matrix overspecification occurs, the EBs for all attribute patterns among all the items ranged from -0.1 to 0.1. That is, overspecification does not show apparent impact on the probability of answering correctly for each attribute pattern. This is consistent with the results of MAD_δj’s.

Table 9: The EBs of ˆP_j(m) with and without Q-matrix underspecification, discrete uniform distribution of attribute patterns, and sample size of 1000

item (misspecification) sample size Attribute pattern of EBP_j(m)

00010 00011

20 (00011)→ (00010) EB₀ 200 0.074 -0.074

500 -0.003 -0.035

26 (10011)→ (10010) EB0 200 -0.079 0.010

500 -0.041 -0.015

29 (00011)→ (00010) EB₀ 200 0.068 -0.039

500 -0.047 -0.034

30 (00011)→ (00010) EB₀ 200 0.034 -0.065

500 0.035 -0.026

EB₀: Empirical bias of estimates under the correct Q-matrix;

EB_q: Empirical bias of estimates under Q-matrix misspecification.

Table 10: The EBs of ˆP_j(m)with and without Q-matrix underspecification, underlying mul-tivariate normal distribution(ρ = 0.3) of attribute patterns, and sample size of 1000

item (misspecification) sample size Attribute pattern of EB_Pj(m)

00010 00011

20 (00011)→ (00010) EB₀ 200 -0.032 -0.014

500 -0.014 0.009

26 (10011)→ (10010) EB₀ 200 0.273 0.033

500 -0.110 -0.017

29 (00011)→ (00010) EB₀ 200 0.024 -0.001

500 -0.021 -0.074

30 (00011)→ (00010) EB₀ 200 0.046 -0.038

500 0.025 -0.017

EB₀: Empirical bias of estimates under the correct Q-matrix;

EB_q: Empirical bias of estimates under Q-matrix misspecification.

Table 11: The EBs of ˆP_j(m)with and without Q-matrix underspecification, underlying mul-tivariate normal distribution(ρ = 0.8) of attribute patterns, and sample size of 1000

item (misspecification) sample size Attribute pattern of EB_Pj(m)

00010 00011

20 (00011)→ (00010) EB0 200 0.074 -0.074

500 -0.003 -0.035

26 (10011)→ (10010) EB₀ 200 -0.079 0.010

500 -0.041 -0.015

29 (00011)→ (00010) EB₀ 200 0.068 -0.039

500 -0.047 -0.034

30 (00011)→ (00010) EB₀ 200 0.034 -0.065

500 0.035 -0.026

EB₀: Empirical bias of estimates under the correct Q-matrix;

EB_q: Empirical bias of estimates under Q-matrix misspecification.

Table 12: The EBs of ˆP_j(m) with and without Q-matrix underspecification, underlying higher-order distribution of attribute patterns, and sample size of 1000

item (misspecification) sample size Attribute pattern of EBP_j(m)

00010 00011

20 (00011)→ (00010) EB₀ 200 0.064 -0.057

500 -0.045 -0.028

26 (10011)→ (10010) EB0 200 -0.089 0.034

500 -0.052 0.006

29 (00011)→ (00010) EB₀ 200 0.119 0.055

500 0.087 0.001

30 (00011)→ (00010) EB₀ 200 0.014 -0.008

500 -0.138 0.023

EB₀: Empirical bias of estimates under the correct Q-matrix;

EB_q: Empirical bias of estimates under Q-matrix misspecification.