G-DINA模型考慮順序限制下之參數估計

全文

(1)國立臺灣師範大學數學系碩士班碩士論文. 指導教授：蔡蓉青博士. Estimation of Generalized DINA Model with Order Restrictions. 研究生：洪辰諭. 中華民國一 ○ 二年七月.

(2) Estimation of Generalized DINA Model with Order Restrictions Chen-Yu Hong July 15, 2013.

(3) 致謝流光瞬息，碩士班的生活即將步入尾聲。這篇論文的完成，首先要感謝我的指導教授蔡蓉青老師。不論是做事的態度，還是論文的撰寫、編排，在老師悉心、耐心的指導下，使我對認知診斷模型這個領域有了深刻的認識。此外，我也要感謝郭伯臣老師和蔡碧紋老師，感謝兩位老師特別抽空擔任我的口試委員，點出問題的癥結，並提供諸多寶貴的意見，使我獲益良多，也讓論文可以更加的完整。再者，我要感謝張育瑋學長和簡鴻宇學長，在程式語言上提供了我許多的幫助。還有其它一起奮鬥，一起努力的同學們。最後我要感謝我的家人和朋友們，在這段時間的支持與陪伴是我前進的動力。還有許多要感謝的人、事、物，我會把這份感謝放在心裡。這些日子看似單調，卻非常充實，我很高興能有這段美好的經驗。僅以這篇論文獻給所有關心我、幫助過我的人。願這份快樂能與你們共享。. i.

(4) 摘要測驗的目的在於評估學生的學習狀況，但是不論是百分制還是量尺分數都沒有辦法對於學生掌握了哪些概念提供足夠的訊息。為了讓老師和學生能從測驗的結果中得到更完整的資訊，認知診斷模型在這方面提供了很大的幫助。其中，DINA 模型概念簡單，也容易解釋，它的一般化形式 G-DINA 模型則提供了更彈性的參數估計。然而，G-DINA 原則上允許即使掌握的概念較多，也可能在試題的答對率上不如掌握概念較少的學生。這篇論文對於參數空間受限制的 G-DINA 模型，提供了上移和下移兩種參數估計的算法，藉此避免在實際應用認知診斷模型時會發生上述被認為是違反直覺的現象。透過模擬研究，在參數空間確實在受限制之範圍時，我們比較這兩種參數估計方法與原本的估計方法在參數估計上的準確度以及受試者認知組型的辨識率。結果顯示我們所提出的上移法表現較其它兩個方法好。因此不論受試者的認知組型、題目的類型或樣本大小為何，本文都建議使用這個方法。關鍵字: 認知診斷模型, G-DINA 模型. ii.

(5) Abstract The purpose of a test is to assess student learning, but the percentile or the total score of the test does not seem to provide enough information as for whether the students master all or some attributes the test intends to evaluate. In order to obtain a better understanding of the test results for both the teachers and the students, cognitive diagnostic models can provide more information in this regard. Among them, the DINA model is very straightforward and its generalization, the G-DINA model, offers a more flexible extension. However, the unrestricted parameter space of G-DINA model allows for the possibility of the lower correct rates for the students who master more attributes than those who know less. This paper considers the use of order-constrained parameter space of the G-DINA model to avoid such counter-intuitive phenomena and proposes two algorithms, the upward and downward methods, for its parameter estimation. Through simulation studies, we compare both the accuracy in parameter estimates and in classification of attribute patterns obtained from the proposed two algorithms and the existent one when the restrictive parameter space is true. Our results show that the upward method performs the best among the three and therefore it is recommended for estimation, regardless of the distributions of respondents’ cognitive patterns, feature types of the test items, and sample sizes of the data. Keywords: cognitive diagnostic model, G-DINA model. iii.

(6) Contents 1 Introduction. 1. 2 The G-DINA Model. 4. 3 Estimation Procedures with Order Restrictions. 6. 4 Simulation Studies 4.1 Q-matrix . . . . . . . . 4.2 Distribution of Cognitive 4.3 Type of Items . . . . . . 4.4 Sample Size . . . . . . . 4.5 Results . . . . . . . . . .. . . . . . . . . . . . . . . Patterns of Respondents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . .. . . . . .. . . . . .. . . . . .. . . . . .. . . . . .. . . . . .. 12 12 13 16 17 17. 5 Analysis of Empirical Data. 28. 6 Discussion and Conclusion. 34. 7 References. 37. 8 Appendix. 39. iv.

(7) 1. Introduction The purpose of a test is to assess student learning and provide feedback for the teachers to make necessary instructional adjustments or change teaching tactics. In order to meet the needs of more students, we hope the analysis of assessment data or a test will enable us to assist students in making progress or achieving their goals in an individualized way. In practice, the majority of test results only provide the total score for each student. Similarly, the traditional unidimensional item response theory models (IRTs) commonly use a single variable to represent the student’s latent capacity or ability the test intends to measure. However, students with the same total scores or equal latent ability might not have necessarily answered the same questions wrongly. That is, the imperfection of their answers to the test might result from different misconceptions or the non-mastery of different attributes. If we wish to understand student learning better, simply obtaining a total score of the test is definitely not enough. For the purpose of identifying whether the students master all or some attributes involved for solving the test items, cognitive diagnosis models (CDMs) have been developed. In general, CDMs represent the class of model-based classification approaches which aim to assign students to different attribute profile groups. That is, each student’s capacity is often represented by a cognitive attribute profile or attribute mastery pattern consisting of a vector of binary indicator variables with 1 and 0 indicating respectively the mastery and non-mastery status of each attribute. As a result, students are no longer distinguished by their test scores, but classified by their cognitive patterns indicating both mastery and non-mastery attributes instead. With the addition of CDMs for analyzing test results, knowing the respondents’ cognitive patterns can help the teachers make some instructional adjustment or prepare remedial teaching and the students strengthen their understanding of certain attributes and concepts, respectively. Over the years, many different types of CDMs have been developed, such as the deterministic input, noisy ”and” gate model (DINA; Junker & Sijtsma, 2001), the noisy input; deterministic ”and” gate model (NIDA; Junker & Sijtsma, 2001), the general diagnostic model (GDM; von Davier, 2005), the deterministic input; noisy ”or” gate model (DINO; Templin & Henson, 2006), and so on (DiBello, Roussos, & Stout, 2007; Rupp, Templin, & Henson, 2010). These core CDMs are said to mainly differ in their utilized condensation rule of being conjunctive/non-compensatory or disjunctive/compensatory and accordingly necessary restrictions are imposed either on certain parameters or on the model structure. However, von Davier (2013) goes beyond such a distinction by showing that some compensatory model is mathematically equivalent to the (conjunctive) DINA model. That is, a conjunctive diagnostic classification model can be expressed and regarded as a constrained special case of a general compensatory diagnostic modeling framework. With only one slipping (sj ) and one guessing (gj ) parameters for each item j, the DINA model is frequently applied in analyzing real test data because of 1.

(8) its simple structure and ease of interpretation. Its generalization, the G-DINA (generalized deterministic inputs, noisy ”and” gate; de la Torre, 2011) model relaxes the assumption that, regardless the respondent’s attribute mastery pattern, there exists only two possible probabilities of getting each item correct. Allowing the probability of getting each item correct to differ for students with different attribute mastery patterns, G-DINA is more flexible and might be more in line with real data in practice and therefore we focus on the G-DINA model in the current study. Upon estimating the DINA model, its item probabilities are subject to an order constraint that (1 − sj ) must be greater than gj to ensure that a respondent who has mastered all measured attributes has a higher probability of getting an item correct than all other respondents who are lacking at least one of the measured attributes. Such an order restriction seems intuitive and is considered analogous to the monotonicity assumption in other latent variable models such as in the well-known Rasch model (Rasch, 1960). However, estimation of the G-DINA model with the unrestricted parameter space permits the possibility of having lower correct rates for the respondents who have mastered more attributes than those who have mastered less (Wei, 2012). In fact, de la Torre (2011) acknowledges such a phenomenon and suggests a possible scenario that a person who has mastered more attributes that an item requires might have a higher chance of falling into the trap of an intentionally designed distractor option in a multiple-choice item. In this case, the probability of answering this item correctly for a respondent who has mastered some but not all the attributes is likely to be lower than those who just guess the answer. Consequently, it might be necessary to systematically account for such a diversion by incorporating a distractor effect into the G-DINA model for items of such a nature. However, in practical applications of CDMs, it is still considered counter-intuitive that a respondent who knows more has a lower probability of answering an item correctly than a respondent who knows less when no obvious characteristics are observed or no deviation is expected for that particular item. Thus, it is intuitively preferable to impose constraints on the parameter space of the G-DINA model. In line with putting an order restriction on the DINA model, it is natural to impose the non-negativity constraints on the intercept parameter as well as the sum of the rest of the parameters of the G-DINA model. However, it becomes relatively complicated when quite a few parameters are involved in the latter term. That is, it is less than obvious what the signs and the order of the various interaction parameters in the G-DINA model should be. To overcome such a problem, we instead consider putting constraints on the order of the probabilities of answering each item correctly being consistent with the superiority of the attribute mastery patterns. In other words, the order restrictions ensure that a respondent who has mastered some attributes will have a higher probability of answering an item correctly than another respondent who has mastered at least one attribute short of him or her. According to van Eeden (1996), the development of the restricted parame2.

(9) ter space estimation problems could be traced back to the 1950s. Schoenberg (1997) uses the sequential quadratic programming method to obtain the constrained maximum likelihood estimates, that is, maximum likelihood estimates with general parametric constraints (linear or nonlinear, equality or inequality). With respect to order constraints, Peddada, Dunson and Tan (2005) propose a simple iterative algorithm for the estimation of order-restricted means for multivariate normal variants. We adopt a similar idea to propose a minor modification in the existent iterative algorithm for estimating G-DINA model to impose the necessary order restrictions on the resultant probabilities of correct. In the present study, two iterative algorithms for estimating G-DINA model with order restrictions are proposed. To investigate the validity of the proposed algorithms, simulation studies are conducted to evaluate their performance in terms of the mean absolute difference (Henson, Roussos, Douglas, & He, 2008), the root mean square error of approximation (von Davier, 2006), and the classification accuracy index (Cui, Gierl & Chang, 2012). The next section briefly presents the G-DINA models. Section 3 gives details on the two proposed iterative algorithms. The remaining of the thesis focuses on the design and the results of the simulation studies to investigate the validity of the proposed restricted estimation method, followed by some concluding remarks.. 3.

(10) 2. The G-DINA Model The CDMs try to integrate cognitive science with psychological measurement to gain understanding of the test taker’s underlying cognitive status. Owing to the U.S. government’s policy “No Child Left Behind Act 2001” (U.S. Department of Education, 2001), the CDMs have rapidly developed and grown in recent years. Unlike the IRTs which attempt to obtain the respondents’ latent ability, their cognitive attribute patterns of mastery or non-mastery status are represented in a vector form. The cognitive states of respondents in the K ∗ ∗ ∗ ), where l is the index used , . . . , αlK , αl2 attributes are denoted by α∗l = (αl1 to indicate the lth pattern among all. For each attribute, each respondent has two states of mastery or non-mastery. That is, the number of possible cognitive patterns may be up to 2K . If some respondent has the cognitive pattern α∗l = (0, 1, 1, 0, 1), it means that the respondent has mastered the second, the third and the fifth, but not the first and the fourth attributes. Since cognitive diagnosis is our main propose, it is crucial to clearly define or identify the relationship between the items and their associated attributes in the so-called Q-matrix. The Q-matrix is of size J × K, where J is the number of items, and K is the total number of attributes in the test. The (j, k) entry of the Q-matrix qjk = 1 indicating that item j is associated to attribute k, and otherwise qjk = 0. For example, consider the Q-matrix which describes the relationship between the four items and the five attributes in a test such that   1 0 1 0 0  0 1 1 1 0   Q=  1 1 0 0 1 . 0 0 0 1 1 According to the Q-matrix, answering item 1 correctly involves the first and the third attributes, whereas the fourth and the fifth attributes are needed for answering item 4 correctly, and so on. In this study, we focus on the G-DINA model (de la Torre, 2011) which is a generalization of DINA model originated from Junker and Sijtsma (2001). The DINA model classifies all respondents into two latent groups with different probability of correct based on whether or not the respondent is fully equipped with all the attributes measured by that item. Ideally, a person who has mastered all the attributes will surely answer the item correctly whereas the absence of one or more necessary attributes could result in a wrong answer to the item. However, mistakes may still occur for a mastery-all student due to careless. On the contrary, it is also possible that a person lacking one or more attributes might have guessed the item correctly. Let X i = (Xi1 , Xi2 , . . . , XiJ ) be the response vector of person i on the test and Xij is his or her response on item j. When respondent i is ∏ thought to ∗ ∗ belong to the latent cognitive pattern αl , we further define ηj = k:qjk =1 αlk to indicate whether respondent i has mastered all the attributes that item j intends to measure. In other words, ηj = 1 when respondent i has mastered 4.

(11) all those attributes and ηj = 0 otherwise. Because each item is associated with different attributes, we define the reduced cognitive pattern associated with item j such that αl(j) is the sub-vector of α∗l consisting of only those αlk ’s with qjk = 1. For example, if item j requires only the first and the third attributes, we retain only the two corresponding elements from the cognitive pattern α∗l the respondent belonging to. Thus, α∗l can be reduced to ∗ ∗ αl(j) = (αl(j)1 , αl(j)2 ) = (αl1 , αl3 ), we call this the reduced cognitive pattern of item j for a respondent in the cognitive pattern α∗l . Denote the number of attributes associated with item j as Kj , the reduced cognitive pattern αl(j) = (αl(j)1 , . . . , αl(j)Kj ) would be a vector with Kj elements. Moreover, ∏ k ∏ ∗ = K ηj = k:qjk =1 αlk k=1 αl(j)k . In the DINA model, the probability of getting a correct answer for respondent i on item j is formulated as follows: (1−ηj ). P (Xj = 1|ηj ) = (1 − sj )ηj gj. (1). ,. where sj and gj are respectively the slipping and the guessing parameters of item j. As shown in equation (1), only two possible probabilities of 1 − sj and gj are assumed for those who have mastered all attributes associated with item j and those are lacking at least one attribute, respectively. That is, the DINA model simply classifies all the respondents into two latent cognitive groups. For example, for an item associated with only two attributes, the respondents are categorized into two cognitive groups with reduced cognitive patterns {(1, 1)} and {(0, 0), (1, 0), (0, 1)} with respect to their probabilities of getting a correct answer for item j. Despite of its simplicity, the assumption of the DINA model that all the respondents should be divided into only two categories in terms of their probability of correct would not always be appropriate. With a more flexible extension, G-DINA model allows respondents in each reduced cognitive pattern, not only the two latent cognitive groups, to have its own probability of correct (de la Torre, 2011). In other words, the respondents were divided into 2Kj categories for each item j with respect their probability of answering the item correctly. More specifically, the G-DINA model states that P (Xj = 1|αl(j) ) = δj0 +. Kj ∑. δjk αl(j)k +. Kj −1 Kj ∑ ∑ k′ =k+1. k=1. δjkk′ αl(j)k αl(j)k′. k=1. +... + δj12...Kj. Kj ∏. (2) αl(j)k ,. k=1. where δj0 is the intercept for item j, indicating the probability of a correct answer of someone who has equipped with none of the required attribute of item j, δjk is the main effect representing the change in probability of responding correctly once someone has mastered the kth attribute of item j, δjkk′ up to δj12...Kj are the interaction effects associated with mastering more than one attributes. For brevity, the notation Pj (αl(j) ) instead of P (Xj = 1|αl(j) ) is used in the rest of sections. 5.

(12) 3. Estimation Procedures with Order Restrictions As mentioned above, the estimation of the DINA model is subject to an order constraint that (1 − sj ) must be greater than gj to ensure that a respondent who has mastered all measured attributes has a higher probability of getting an item correct than all other respondents who are lacking at least one of the measured attributes. However, estimation of the G-DINA model with the unrestricted parameter space permits the possibility of having lower probability of correct for the respondents who have mastered more attributes than those who have mastered less. In this section, we discuss how to impose necessary constraints to obtain the permissible parameter space which ensures that that a respondent who has mastered some attributes will have a higher probability of answering an item correctly than another respondent who has mastered at least one attribute short of him or her. First of all, we define the order relation between two reduced cognitive patterns for item j such that αl(j) ≽ αl′ (j) if αl(j)k ≥ αl′ (j)k for all k = 1, 2, ..., Kj and αl(j) ≻ αl′ (j) if αl(j)k > αl′ (j)k at least one k. When αl(j)k > αl′ (j)k , it means that a respondent with the cognitive pattern αl(j) has mastered the kth attribute of item j whereas another person with αl′ (j) has not. That is, αl(j) is considered superior to αl′ (j) . In other words, a person with αl(j) not only has mastered at least all the attributes another person with αl′ (j) has mastered for item j, but could also have equipped with some other attributes the latter has not. To ensure the monotonicity relation between the order of the cognitive mastery pattern αl(j) ’s and their corresponding probability of answering an item correctly, we aim to have Pj (αl(j) ) ≥ Pj (αl′ (j) ) when αl(j) ≻ αl′ (j) . With a minor modification of the iterative algorithm used for estimating the G-DINA model by de la Torre (2011), we propose two algorithms to obtain parameter estimates satisfying the order restrictions on the probabilities of correct. de la Torre (2011) considers using marginalized maximum likelihood estimation (MMLE) for the G-DINA model. The log-marginalized likelihood for the test data X = (X 1 , . . . , X I ) of I respondents is written as l(X) = log[L(X)] = log. I ∑ L ∏. L(Xi |α∗l )p(α∗l ),. (3). i=1 l=1. ∏ where L(X i |α∗l ) = Jj=1 Pj (αl(j) )Xij [1 − Pj (αl(j) )](1−Xij ) is the likelihood of the responses of respondent i with cognitive pattern α∗l , and p(α∗l ) is the prior probability of the cognitive pattern α∗l . Taking the derivative of l(X) with respect to Pj (αl(j) ), we could obtain the marginal maximum likelihood estimate of Pj (αl(j) ) such that Pˆj (αl(j) ) = where Iαl(j) =. ∑I i=1. Rαl(j) Iαl(j). (4). p(αl(j) |X i ) represents the expected number of respondents 6.

(13) ∑ to be in the reduced cognitive pattern αl(j) , Rαl(j) = Ii=1 p(αl(j) |X i )Xij is the expected number of respondents in those who have answered item j correctly to be in the reduced cognitive pattern αl(j) , and p(αl(j) |X i ) is the posterior probability of respondent i belonging to the cognitive pattern αl(j) . To obtain the parameter estimates, we running the following algorithms: Step 1. Starting with a set of initial values for all parameters we want to estimate, Pj (αl(j) ) for j = 1, 2, . . . , J. Since. Iαl(j) =. I ∑. Rαl(j) =. I ∑. p(αl(j) |X i ) =. i=1 I ∑. i=1. p(αl(j) |X i )Xij =. i=1. and. p(X i |α∗l )p(α∗l ) ∑2 K ∗ ∗ l=1 p(X i |αl )p(αl ) I ∑ i=1. p(X i |α∗l ) = L(X i |α∗l ) =. J ∏. p(X i |α∗l )p(α∗l ). ∑2 K. ∗ ∗ l=1 p(X i |αl )p(αl ). Xij. Pj (αl(j) )Xij [1 − Pj (αl(j) )](1−Xij ). j=1. Step 2. For each j, calculate Iαl(j) and Rαl(j) using the initial values of Pj (αl(j) ). Step 3. For each j, compute Pˆj (αl(j) ) by substituting Iαl(j) and Rαl(j) into (4). After step 3, we can obtain a set of estimates of Pj (αl(j) ) for j = 1, 2, . . . , J. Of which if the situation that Pˆj (αl(j) ) < Pˆj (αl′ (j) ) when αl(j) ≻ αl′ j occurs, we would need to make some adjustments so that the order restriction is satisfied. We can approach this problem from two different directions, and the respective adjustments are explained as the following: • Method 1: Upward The first aspect is considering that more comprehensive cognitive patterns should not have a lower probability of getting a correct answer for item j. Thus, if Pˆj (αl(j) ) < Pˆj (αl′ (j) ) when αl(j) ≻ αl′ (j) , let Pj (αl(j) ) = max{Pj (αl′ (j) )|αl(j) ≻ αl′ (j) }.. (5). As a result, we can make sure that all cognitive patterns inferior to αl(j) do not have a higher probability of getting a correct answer for item j. This method is referred to as the “upward” method hereinafter. • Method 2: Downward The second aspect is that inferior cognitive patterns should not have a higher probability of answering item j correctly. Thus, if Pˆj (αl(j) ) > Pˆj (αl′ (j) ) when αl(j) ≺ αl′ (j) , let Pj (αl(j) ) = min{Pj (αl′ (j) )|αl′ (j) ≻ αl(j) }.. (6). Similarly, we guarantee that all cognitive patterns superior to αl(j) will have a higher probability of answering item j correctly. This method is referred to as the “downward” method hereinafter. 7.

(14) Step 4. Use the above method to make adjustment on the results obtained in step 3 if necessary. Otherwise, skip step 4 if the unnatural phenomenon does not occur. This step is where we impose order restrictions on the estimates to ensure monotonicity in the probabilities of correct and the superiority of attribute mastery patterns. Step 5. Repeat steps 2 to 4 until convergence is achieved. For illustration, suppose that there is an item with Kj = 3, and its estimates on step 3 at some iteration is shown on the first row of Table 1. Table 1: An Illustrative Example for the Proposed Algorithms P000. P100. P010. no restriction 0.1 0.06 0.27 upward method 0.1 0.1 0.27 downward method 0.06 0.06 0.20 P000 = Pj (αl(j) = (000)) and others are. P001. P110. P101. P011. P111. 0.13 0.25 0.19 0.20 0.89 0.13 0.27 0.19 0.27 0.89 0.13 0.25 0.19 0.20 0.89 similarly defined.. A closer inspection of the estimates in step 3 on the first row, we found that order relations of three pairs do not satisfy the order restriction, including P100 < P000 , P110 < P010 , and P011 < P010 . With the order restrictions, it is intuitively more plausible that P000 ≤ P100 , P010 ≤ P110 , and P010 ≤ P011 . After the adjustments in step 4, the following estimate updates are obtained such that P100 = P000 , P110 = max{P000 , P100 , P010 } and P011 = max{P000 , P010 , P001 } for upward method and P000 = P100 , P010 = min{P110 , P011 , P111 } for downward method, respectively. These adjusted values which can be seen to be in line with the order restrictions are also reported in Table (1) for ease in comparison. Furthermore, after the convergence is reached and the parameter estimates obtained, we can obtain the approximate information matrix, I(Pˆj ), for item j using Pˆj (αl(j) ) and X using the following equation: ∑ ∂ 2 l(X) ∂L(X i ) ∂L(X i ) = − [L−2 (X i ) ] ′ (j) ) ∂Pj (αl(j) )∂Pj (αl′ (j) ) ∂P (α ) ∂P (α j j l(j) l i=1 I. = −. I ∑. {p(αlj |X i ). i=1. {p(αl′ (j) |X i ). Xij − Pj (αl(j) ) }· Pj (αl(j) )[1 − Pj (αl(j) )]. Xij − Pj (αl′ (j) ) } Pj (αl′ (j) )[1 − Pj (αl′ (j) )]. (7). The square-root of the diagonal elements of I −1 (Pˆj ) is used as the estimated 8.

(15) standard error of Pˆj , denoted by SE(Pˆj ). The two proposed algorithms not only deal with the order restrictions on the probabilities of correct, but also ensure the non-negativity of the main effects δjk ’s in the G-DINA model. In fact, the parameter estimates of δˆ j = (δˆj0 , δˆj1 , . . . , δˆj12...Kj ) of the G-DINA model in (2) can be directly obtained from ˆ j through their one-to-one correspondence of P (s) −1 ˆ δˆ j = M j P j,. (8). (s). where M j is called the saturated design matrix, and its structure is described below. Let Aj = {αlk }, a 2Kj × Kj matrix defined by   0 0 0 ··· 0 0  1 0 0 ··· 0 0     0 1 0 ··· 0 0     .. .. .. . . .. ..   . . . . . .     0 0 0 ··· 1 0     0 0 0 ··· 0 1     1 1 0 ··· 0 0      Aj =  1 0 1 · · · 0 0  .  .. .. .. . . . .. ..  . .   . . .    1 0 0 ··· 0 1   . . . .   .. .. .. . . ... ...     0 0 0 ··· 1 1     1 1 1 ··· 0 0     .. .. .. . . .. ..   . . . . . .  1 1 1 ··· 1 1 (s). The lth row of M j can be generated using the lth row of Aj . The first (s) element of M j is 1, and the next Kj elements are the lth row of Aj , followed by αlk ×αlk′ for k = 1, 2, ..., Kj −1 and k ′ = k+1, ..., Kj , and the last element of ∏Kj (s) (s) M j is k=1 αlk . For example, the saturated design matrix M j for Kj = 3 is   1 0 0 0 0 0 0 0  1 1 0 0 0 0 0 0     1 0 1 0 0 0 0 0     1 0 0 1 0 0 0 0  (s) .  Mj =   1 1 1 0 1 0 0 0    1 1 0 1 0 1 0 0     1 0 1 1 0 0 1 0  1 1 1 1 1 1 1 1 The standard error of δˆ j can then be obtained using the fact that (s) −1. Cov(δˆ j ) = M j. (s) −1 ′. ˆ j )(M Cov(P j. 9. ),. (9).

(16) (s) −1. (s) −1. where (M j )′ is the transpose of M j . After the adjustment in step 4, because αl(j) ≻ (0, 0, 0) for all patterns Kj with Σk=1 αl(j)k = 1, we have Pj (αl(j) ) ≥ P000 or Pj (αl(j) ) − P000 ≥ 0. Suppose Kj Σk=1 αl(j)k = αl(j)k′ = 1 for some k ′ th attribute of item j, the main effect δjk′ = Pj (αl(j) )−P000 in the G-DINA model must be non-negative. In addition, since the order relation between αl(j) = (0, 1, 0) and αl′ (j) = (1, 0, 1) is undefined, we can not compare which one of the two is superior for item j. Hence, the order relation between the magnitudes of P010 and P101 is not of our concern. For estimation, we convert the original codes for G-DINA model written in Ox (Doornik, 2002) by de la Torre (2011) to R (R Core Team, 2013) and write the above modified algorithms in R as well. In this study, we consider two indices for evaluating of the performance of the proposed algorithms in estimation accuracy, namely the mean absolute difference (MAD) (Henson, Roussos, Douglas, & He, 2008) and the root mean square error of approximation (RMSEA; von Davier, 2006). • MAD This index measures the mean of absolute value of difference between observed and expected value of correct response probability. It is defined as ∑2K ∗ ∗ l=1 |Pobserved (Xj = 1|αl ) − Pexpected (Xj = 1|αl )| MADj = , (10) 2K where K is the number of attributes we are interested in this test. α∗l = (αl1 , αl2 , ..., αlK ) characterize the latent cognitive group of the respondents. The value of MAD falls between 0 and 1, with a value of MAD close to 0 indicating ”excellent” performance. Conversely, a value of MAD close to 1 represents ”very poor” performance. • RMSEA This index used the square of difference between observed and expected value of correct response probability and weighted those by the latent class distribution as follows: v u 2K u∑ RMSEAj = t p(α∗l )[Pobserved (Xj = 1|α∗l ) − Pexpected (Xj = 1|α∗l )]2 l=1. (11) where is the resultant marginal proportion of latent class based on the most likely latent class group of the respondents. Similarly, performance is considered ”very good” with a value of RMSEA close to 0. p(α∗l ). The above two indices are used to assess the performance of our estimates with order restrictions. More specifically, we would investigate whether the 10.

(17) values of MAD and RMSEA are shown smaller than those obtained without any restrictions. In addition, since the main goal of CDMs is to understand the cognitive patterns of respondents from analyzing their responses. Through simulations, we can evaluate how effectively the proposed algorithms can correctly classify the respondents when their true cognitive patterns are known. Several classification accuracy indices have been proposed in (Cui, Gierl & Chang, 2012). However, we simply use the resultant classification of respondents reported on Ox code for G-DINA models to compute the classification accuracy index as ∑ Pca = P (X α∗l ∈ πα∗l |α∗l )p(α∗l ), (12) l. where X α∗l is the the collection of all the responses of each respondent with true cognitive pattern α∗l and πα∗l is the set of responses that are classified into the cognitive pattern α∗l . It provides a proportion of the respondents correctly classified into their own cognitive pattern, and therefore it is called classification accuracy index.. 11.

(18) 4. Simulation Studies In this simulation study, we will compare the performance of the three estimations at different settings. The factors under consideration are the distribution of cognitive pattern of respondents, percentage of key attributes items, and sample size. In addition, the indices we use to evaluate the performance of the estimates include the MAD, RMSEA, and classification accuracy. And the indices will be examined under three item levels individually. In the following we give more details. 4.1. Q-matrix Using the G-DINA model requires Q-matrix, and the Q-matrix we used is Table 1 in de la Torre & Douglas (2004), with J = 30 and K = 5. The first ten items measure only one attribute, and each attribute are measured twice, for the middle ten (items, items 11 to 20, each measures two attributes ) 5 and therefore include all 2 = 10 possible pairs of the attributes, and for the last ten items, (5) items 21 to 30, each measures three attributes each and those contain all 3 = 10 possible triples of attributes. Therefore, each attribute is measured 12 times in this test, that is, the Q-matrix satisfies the constant attribute information. The Q-matrix structure is shown in Table 2: Table 2: Q-matrix for Simulations Item K1 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15. 1 0 0 0 0 1 0 0 0 0 1 1 1 1 0. K2. K3. K4. K5. Item. K1. K2. K3. K4. K5. 0 1 0 0 0 0 1 0 0 0 1 0 0 0 1. 0 0 1 0 0 0 0 1 0 0 0 1 0 0 1. 0 0 0 1 0 0 0 0 1 0 0 0 1 0 0. 0 0 0 0 1 0 0 0 0 1 0 0 0 1 0. 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30. 0 0 0 0 0 1 1 1 1 1 1 0 0 0 0. 1 1 0 0 0 1 1 1 0 0 0 1 1 1 0. 0 0 1 1 0 1 0 0 1 1 0 1 1 0 1. 1 0 1 0 1 0 1 0 1 0 1 1 0 1 1. 0 1 0 1 1 0 0 1 0 1 1 0 1 1 1. Q-matrix describes the relationship between items and the attributes we want to measure. For example, the qjk = 1 means that item j involves attribute 12.

(19) k to answer correctly, otherwise, qjk = 0 means that the item j does not involves the kth attribute. 4.2. Distribution of Cognitive Patterns of Respondents In this study, we consider three kinds of distribution of cognitive patterns of respondents, consisting of discrete uniform, multivariate normal and higherorder distribution. • Discrete Uniform Distribution In this case, any respondent has equal chance of belonging to any one of the possible cognitive pattern. For instance, if the number of attributes of concerned is five and each of these attributes may be mastered or not, there are a total of 25 = 32 possible cognitive patterns and therefore each respondent has a probability of 0.03125 to belong to any cognitive pattern. Different from the above ideas, the ability of respondent be regarded as a continuum to result in the following two distributions of cognitive pattern, and we say that the attribute is equipped when the continuum reaches a certain level. In the following case, we denote the cognitive pattern and the ability variable of respondent i by C i = (Ci1 , Ci2 , ..., CiK ) and θ i = (θi1 , θi2 , ..., θiK ), respectively. If the kth element in θ i is greater than 0, then we say that respondent i has mastered the kth attribute. In other words, Cik = 1 if θik ≥ 0, and Cik = 0 if θik < 0 for all k = 1, 2, . . . , K. • Multivariate Normal Distribution These five attributes may correspond to the five ability variables with a multivariate normal distribution structure as below: θ i ∼ N (µ, Σ) where.    µ=  . −0.5 −0.25 0 0.25 0.5. . .     ,Σ =     . 1 0.3 0.3 0.3 0.3. 0.3 1 0.3 0.3 0.3. (13). 0.3 0.3 1 0.3 0.3. 0.3 0.3 0.3 1 0.3. 0.3 0.3 0.3 0.3 1.    ,  . θ i is used to represent the ability of respondent i corresponding to each attribute, and µ describes the difficulty of attributes with a large value of µ indicating a lower difficulty, i.e., there are more respondents who have mastered this attribute. Σ is the covariance matrix of latent abilities. Hence, the probability density function of θ i is: fθ i = √. 1 exp(− (θ i − µ)T Σ−1 (θ i − µ)) 2 (2π)K |Σ| 1. 13. (14).

(20) Kunina-Habenicht, Rupp and Wilhelm (2012) mentioned that the correlations between attributes may exceeded 0.8 in some cases. Hence, this study also consider the situation that correlations are equal to 0.8. In this paper, we denote the lower correlation condition by MVN1, and denote the higher correlation condition by MVN2. • Higher-Order Distribution Another idea, there is some common ability variable affecting the attributes that we are interested. That is, whether someone masters each attribute or not has a degree of correlation with his or her ability. Its structure is as follows: K i = β0 + β1 · θi + εi. (15). where    θi ∼ N (0, 1), β 0 =   . −0.5 −0.25 0 0.25 0.5. . .      , β1 =     . 0.9 0.8 0.7 0.6 0.5.    2  , εik ∼ N (0, 1 − β1k )  . where θ is the ability of respondent i behind the attributes we are interested, β 0 describes the difficulty of the attributes, with a large value indicating a lower difficulty, and β 1 represents the relation between attributes and the latent ability. The higher-order distribution mentioned above can also be formulated as a multivariate normal distribution as below: µ = β 0 , Σ = β 1 β T1 + Diag(I − β 1 β T1 ),. (16). where Diag(A) denotes the diagonal matrix with the diagonal elements of A. We particularly use εi to adjust the variance of attribute distributions to 1 for consistency with the design of Multivariate Normal Distribution. A large value of θi indicates that respondent i possess higher ability, and conversely a small value implies that respondent i with lower ability. Again, large values in β 0 indicates a lower difficulty. We consider that more difficult attributes had a higher correlation with the ability the test intends to measure because that mastery of a difficult attribute needs higher capacity. The reason for having variance adjusted to 1 is to be consistent with MVN. The probability mass functions of each distribution of cognitive patterns is shown in figure 1, and the label in x-axis is explained in Table 3.. 14.

(21) Figure 1: The probability mass function of distributions of cognitive patterns.. 15.

(22) Table 3: Index of Cognitive Patterns index pattern 1 00000 2 10000 3 01000 4 00100 5 00010 6 00001 7 11000 8 10100. index 9 10 11 12 13 14 15 16. pattern 10010 10001 01100 01010 01001 00110 00101 00011. index 17 18 19 20 21 22 23 24. pattern 11100 11010 11001 10110 10101 10011 01110 01101. index 25 26 27 28 29 30 31 32. pattern 01011 00111 11110 11101 11011 10111 01111 11111. 4.3. Type of Items In this study, items from two different types of CDMs are considered. • DINA items In the first category, respondents need to equip all the attributes of an item to have a high probability of answering correctly. In other words, for this type of items we set all the parameters, except the intercept and the highest-order interaction effect, to zeros in the G-DINA model. δj0 and δj12...Kj P (Xj = 1|αl(j) ) = δj0 + δj12...Kj. Kj ∏. αl(j)k. k=1. • A-CDM items (de la Torre, 2011) Secondly, the probability of answering item j correctly will increase when respondents have mastered more and more attributes. In other words, for this type of items we set all of their interaction effects, but not the main effects, to zeros in the G-DINA model. P (Xj = 1|αl(j) ) = δj0 +. Kj ∑. δjk αl(j)k. k=1. We consider three kinds of test: items all generated by the DINA model, items all generated by the A-CDM model, and a mixture of both types of items. The examination with only DINA model items is called test 1. Test 2, on the other hand, contains only A-CDM items and 0% DINA model items. Test 3 contains both types of items with 50% of each type. The parameter values and the probability of answering each item correctly for all the attribute patterns in each test are attached as Tables 17 to 22 in the Appendix.. 16.

(23) 4.4. Sample Size Chen (2011) used the sample size of 100, 500 and 1000. But the number of attributes we considered in this study is 5, that is, there are a total of 25 = 32 possible cognitive patterns. On average, each pattern only contains three respondents when the sample size is 100, the estimation results are likely to be unstable. Therefore, we consider three kinds of sample size be considered in this study: 200, 500, and 1000, each respectively representing a small, a medium, and a large sample size. 4.5. Results The first thing we would like to investigate is the proportion of negative main effect estimates in all cases. Because only the non-negativity of the main effects are particularly addressed in the G-DINA model, therefore the counterintuitive phenomenon occurring at the main effects is easy to detect. Table 4 underlines the proportions of main effects estimated to be negative being less than 10%, over 20%, even 50% in most case. Interestingly, when test 1 is all composed of the DINA items, the proportion of obtaining negative main effect estimates is relatively high, and in which there are three-quarters of these cases having more than 50% negative estimates. On the other hand, for test 2 all composed of the A-CDM items, the proportion declines obviously and most of these cases only have below 10% of negative estimates. In other words, if a test is all composed of A-CDM items, or allow student to use multiple ways or several key concepts for problem solving, the estimates of main effects are less likely to be negative. In summary, negative estimates are obtained with certain probability when the item parameters are all positive. Thus, it is useful to develop an estimation algorithm for the G-DINA model with order restriction and hopefully the new method will produce estimates with better property. The second thing that we are concerned about is the accuracy performance of the proposed estimation methods. We consider three manipulated factors: percentage of DINA model items, distribution of cognitive pattern, and sample size. According to our design of the Q-matrix, items 1 to10 involve only one attribute, items 11 to 20 involve exactly two attributes, and items 21 to 30 involve exactly three attributes. The three item levels defined in the beginning of this section are: the items involving one or more attributes, that is, all of items, the items involving two or more attributes, and the items involving three attributes. We compare the values of MAD and RMSEA of three kinds of method, including the unrestricted method and two restricted method, namely the upward and downward methods.. MAD Tables 5 to 7 can be broadly inspected in two different ways to serve different purposes. In order to examine which estimation method has the smallest 17.

(24) Table 4: Proportions of Negative Main Effect in Tests 1 to 3 for Each Distribution of Cognitive Patterns and Sample Size. Test 1 200 500 1000. Higher-Order MVN 1 0.53 0.5054 0.5338 0.4966 0.5232 0.5196. 200 500 1000. Higher-Order MVN 1 0.1376 0.064 0.0806 0.0158 0.0518 0.0038. MVN 2 0.5364 0.5508 0.5292. Uniform 0.4974 0.5144 0.4994. MVN 2 0.1898 0.0964 0.07. Uniform 0.0434 0.0062 0.0024. MVN 2 0.3584 0.324 0.2846. Uniform 0.2658 0.256 0.2566. Test 2. Test 3 200 500 1000. Higher-Order MVN 1 0.3212 0.285 0.2816 0.2544 0.2748 0.2526. MAD value, we compare the average of MADs under a variety of designs. Next, the MAD values are compared item by item to examine whether the average of MADs are effected by extreme value or not. Table 5 is for the first item level with items involving one or more attributes (all items), Table 6 is for the second item level containing the items involving two or more attributes, and Table 7 is for the third item level containing items measuring all three attributes exactly. According to Table 5, we can see on average for each test containing various percentage of DINA items, distribution of cognitive pattern, or sample size, the upward method has the smallest average of MAD among the three methods, with the except of only one case (test 3, multivariate normal with higher correlation, sample size of 200), denoted with a shaded gray representation. If we compare the MAD values item by item, the upward method in most cases produces the largest number of items with the smallest MAD. However, there are some of cases (10 in a total of 36 cases), estimates with no order restriction having the largest number of items with the smallest MAD. Table 6 and Table 5 are different because Table 6 is made for the second item level and therefore only accounted for the items involving two or more attributes. That is, MAD for items 11 to 30, not all. On average, the upward method has the smallest MAD in each case. Moreover, when we compare the MAD values item by item, the number of items having the smallest MAD by the unrestricted method is substantially reduced. That is, in Table 5, the items have the smallest MAD of estimate by the unrestricted method are mostly the items involving only one attribute. Table 7 is for the third item level and therefore considers only the items involving three attributes. That is, only items 21. 18.

(25) 19. no restriction upward method downward method no restriction upward method downward method. no restriction upward method downward method no restriction upward method downward method. mean over MAD of items 1-30 number of items has min MAD. no restriction upward method downward method no restriction upward method downward method. mean over MAD of items 1-30 number of items has min MAD. mean over MAD of items 1-30 number of items has min MAD. M AD. 0.0434 0.0262 0.0610 5 25 0. 0.0628 0.0427 0.0791 7 21 2. 0.0933 0.0665 0.1088 8 22 0. HO. 0.0311 0.0192 0.0367 5 23 2. 0.0459 0.0290 0.0543 5 25 0. 0.0715 0.0461 0.0815 7 23 0. 0.0548 0.0351 0.0747 8 20 2. 0.0751 0.0585 0.0909 10 13 7. 0.1048 0.0979 0.1136 11 10 9. Test 1 MVN1 MVN2. 0.0281 0.0182 0.0286 5 22 3. 0.0404 0.0252 0.0415 5 24 1. 0.0645 0.0414 0.0661 3 27 0. Unif. 0.0670 0.0639 0.0665 11 15 4 0.0452 0.0442 0.0450 6 17 7. 1000 0.0732 0.0650 0.0746 17 13 0. 0.1155 0.1007 0.1102 8 14 8. 0.0873 0.0811 0.0973 16 14 0. 0.1128 0.1057 0.1230 18 11 1. 0.1594 0.1430 0.1585 16 12 2. Test 2 MVN1 MVN2. 0.0996 0.0868 0.1028 13 15 2. 500. 0.1460 0.1255 0.1420 14 11 5. 200. HO. 0.0386 0.0384 0.0746 10 18 2. 0.0558 0.0549 0.0555 2 19 9. 0.0971 0.0893 0.0949 3 21 6. Unif. 0.0577 0.0455 0.0731 11 17 2. 0.0803 0.0684 0.0959 12 14 4. 0.1154 0.1137 0.1221 18 9 3. HO. 0.0379 0.0313 0.0410 9 18 3. 0.0563 0.0464 0.0621 5 22 3. 0.0925 0.0750 0.0976 7 19 4. 0.0696 0.0631 0.0934 14 13 3. 0.0953 0.0937 0.1115 14 9 7. 0.1321 0.1362 0.1379 17 7 6. Test 3 MVN1 MVN2. Table 5: Performance of MAD: Overall Mean MAD of All Items and the Number of Items with the Smallest Value. 0.0329 0.0276 0.0333 6 20 4. 0.0475 0.0397 0.0483 7 17 6. 0.0789 0.0630 0.0777 6 19 5. Unif.

(26) 20. no restriction upward method downward method no restriction upward method downward method. no restriction upward method downward method no restriction upward method downward method. mean over MAD of items 11-30 number of items has min MAD. no restriction upward method downward method no restriction upward method downward method. mean over MAD of items 11-30 number of items has min MAD. mean over MAD of items 11-30 number of items has min MAD. M AD. 0.0560 0.0301 0.0798 0 20 0. 0.0811 0.0500 0.1016 0 18 2. 0.1182 0.0765 0.1334 0 20 0. HO. 0.0376 0.0198 0.0457 0 20 0. 0.0556 0.0302 0.0672 0 20 0. 0.0873 0.0489 0.0987 0 20 0. 0.0730 0.0429 0.0995 0 18 2. 0.0995 0.0724 0.1185 0 13 7. 0.1358 0.1204 0.1420 1 10 9. Test 1 MVN1 MVN2. 0.0334 0.0185 0.0341 0 20 0. 0.0486 0.0257 0.0500 0 20 0. 0.0766 0.0421 0.0780 0 20 0. Unif. 0.0868 0.0818 0.0860 7 11 2 0.0582 0.0568 0.0580 3 13 4. 1000 0.1007 0.0870 0.1019 8 12 0. 0.1493 0.1260 0.1424 4 13 3. 0.1227 0.1088 0.1362 7 13 0. 0.1565 0.1390 0.1686 9 11 0. 0.2173 0.1804 0.2109 6 12 2. Test 2 MVN1 MVN2. 0.1363 0.1135 0.1389 5 14 1. 500. 0.1964 0.1586 0.1872 5 11 4. 200. HO. 0.0489 0.0486 0.1019 4 16 0. 0.0704 0.0691 0.0699 1 13 6. 0.1230 0.1114 0.1199 1 18 1. Unif. 0.0775 0.0586 0.0989 3 15 2. 0.1078 0.0880 0.1280 5 13 2. 0.1523 0.1429 0.1579 8 9 3. HO. 0.0478 0.0380 0.0523 3 15 2. 0.0714 0.0566 0.0795 2 17 1. 0.1178 0.0903 0.1235 1 16 3. 0.0956 0.0833 0.1285 4 13 3. 0.1303 0.1221 0.1506 6 9 5. 0.1780 0.1737 0.1818 7 7 6. Test 3 MVN1 MVN2. Table 6: Performance of MAD: Mean MAD of Items 11 to 30 and the Number of Items with the Smallest Value. 0.0406 0.0327 0.0412 3 16 1. 0.0588 0.0471 0.0600 2 12 6. 0.0975 0.0738 0.0955 2 16 2. Unif.

(27) 21. no restriction upward method downward method no restriction upward method downward method. no restriction upward method downward method no restriction upward method downward method. mean over MAD of items 21-30 number of items has min MAD. no restriction upward methodd downward method no restriction upward method downward method. mean over MAD of items 21-30 number of items has min MAD. mean over MAD of items 21-30 number of items has min MAD. M AD. 0.0759 0.0338 0.1272 0 10 0. 0.1109 0.0587 0.1575 0 10 0. 0.1546 0.0902 0.1916 0 10 0. HO. 0.0452 0.0194 0.0660 0 10 0. 0.0661 0.0300 0.0957 0 10 0. 0.1043 0.0485 0.1341 0 10 0. 0.0979 0.0506 0.1634 0 10 0. 0.1296 0.0812 0.1865 0 10 0. 0.1760 0.1339 0.2017 0 10 0. Test 1 MVN1 MVN2. 0.0399 0.0179 0.0454 0 10 0. 0.0577 0.0247 0.0671 0 10 0. 0.0899 0.0395 0.1016 0 10 0. Unif 0.2023 0.1561 0.1897 0 10 0 0.1175 0.1068 0.1157 0 10 0 0.0775 0.0746 0.0770 0 8 2. 500 0.1926 0.1439 0.1883 0 10 0 1000 0.1496 0.1189 0.1474 0 10 0. 0.1753 0.1366 0.1870 0 10 0. 0.2207 0.1650 0.2214 0 10 0. 0.2904 0.2033 0.2670 0 10 0. Test 2 MVN1 MVN2. 200 0.2673 0.1850 0.2455 0 10 0. HO. 0.0626 0.0620 0.1474 2 8 0. 0.0906 0.0880 0.0896 0 9 1. 0.1602 0.1388 0.1525 0 10 0. Unif. 0. 0. 0.1127 0.0778 0.1499 0 10 0. 0.1538 0.1097 0.1881 0 10 0. 0.2035 0.1625 0.2176 3 7 0. HO. 0.0612 0.0460 0.0723 0 9 1. 0.0917 0.0687 0.1098 3 10 0. 0.1513 0.1051 0.1673 0 10 0. 0.1331 0.0993 0.1842 0 10 0. 0.1768 0.1402 0.2108 1 7 2. 0.2310 0.1884 0.2398 4 6 0. Test 3 MVN1 MVN2. Table 7: Performance of MAD: Mean MAD of Items 21 to 30 and the Number of Items with the Smallest Value. 0.0510 0.0391 0.0540 1 8 1. 7. 0.0729 0.0554 0.0788. 0.1231 0.0860 0.1243 0 10 0. Unif.

(28) to 30 are used to compute MAD. In this case, comparing either the average of MADs or MAD values item by item, the upward method has the best performance. That is, upward method produces the smallest average of MADs and the largest number of items having the smallest MAD. Furthermore, when the items we considered that involve attributes increase, the difference of average of MAD between no order restriction and upward method is more significant. Next, we examine the effects of each factor on parameter estimates of the three methods using Table 5. Type of Items. Recall that the items in test 1 are all DINA items, the items in test 2 are all A-CDM items, and test 3 contains both types of items. We can see that when the percentage of DINA items in a test increases, the average of MAD decreases. Distribution of cognitive pattern. Among the four kinds of distribution of cognitive pattern, the discrete uniform distribution mostly has the smallest average of MAD, the next is the multivariate normal distribution with lower correlation, followed by higher-order distribution, and lastly the multivariate normal distribution with higher correlation has the largest average of MAD. With Figure 1, we found that if the cognitive pattern is very concentrated in a few classes, the average of MAD is also relatively large. Sample size. In the case of holding the other factors constant, the average of MAD become smaller as sample size increases. That is, as expected the larger the sample size, the more accurate the estimates are. In addition, comparing the MAD values of the same position of in Table 5 (take into account all items), Table 6 (take into account items 11-30), and Table 7 (take into account items 21-30) under various item level, the results show that that the first item level has the smallest average of MAD, and the third item level has the largest average of MAD. For example, in the case of test 1, cognitive pattern resulting from the higher-order distribution, sample size of 200, and using the unrestricted method, the averages of MAD are respectively 0.0933 in Table 5, 0.1182 in Table 6, and 0.1546 in Table 7. In other words, when the number of attributes the items involve increases, the average of MAD also becomes larger.. RMSEA The construction of Table 8 to 10 is the same as Table 5 to 7, but considering the index RMSEA in this part. In is shown in Table 8 for the first item level that no matter what the design, the upward method almost always has the smallest average of RMSEA, except only two case (test 1, multivariate normal with higher correlation, sample size of 200 and test 3, multivariate normal with higher correlation, sample size of 200). If we compare the RMSEA 22.

(29) values item by item, the method that has the largest number of RMSEA is not the upward method only in a small part of designs (8 of all 36 designs). In the majority of cases, the upward method has the largest number of items with the smallest RMSEA. According to Table 9 for the second item level, the upward method almost always yields the largest number of items having the smallest RMSEA, with the except of only two case (test 1, multivariate normal with higher correlation, sample size of 200 and test 3, multivariate normal with higher correlation, sample size of 500) that the unrestricted method having the smallest RMSEA. Table 10 presents the results for the third item level and they are almost identical with Table 10 that regardless of the design, the upward method has both the smallest average RMSEA and the largest number of items with smallest RMSEA. In addition, the effects of each factor on RMSEA of the estimates are also examined. When the percentage of DINA items in a test increases, the average of RMSEA decreases. The discrete uniform distribution of cognitive patterns has the smallest average of RMSEA, followed by the multivariate normal with lower correlation, the higher-order distribution, and lastly the multivariate normal distribution with higher correlation having the larger average of RMSEA. With increasing sample size, the average of RMSEA becomes smaller. On the other hand, comparing RMSEA values on the same position of Tables 8 to 10, we also found the value in Table 8 the smallest and that in Table 10 the largest. In other words, the average of RMSEA becomes large when the number of attributes the items involving increases. The results on RMSEA display consistent findings with those obtained from MAD. Another point worth noting is that, with respect to either the average or largest number of items on MAD or RMSEA, the estimates with order restrictions always perform better than the unrestricted ones when we only examine items involving two or more attributes.. Classification Accuracy Pca Finally, we compare the classification accuracy of the three methods in this study. From Table 11, classification accuracy index values of the upward and downward methods are shown to be higher than the estimation without order restrictions. We also find that the upward method has the highest classification accuracy among the three methods in test 1, and the downward method has the highest classification accuracy when the sample size is less than 500 in test 2, except when the distribution of cognitive patterns is discrete uniform. When the sample size is large enough (more than 1000), the upward method has the highest classification accuracy in most cases, except for only one case (test 2, multivariate normal distribution with higher correlation). In addition, the impacts of manipulated factors for classification accuracy are less salient.. 23.

(30) 24. no restriction upward method downward method no restriction upward method downward method. no restriction upward method downward method no restriction upward method downward method. mean over RMSEA of items 1-30 number of items has min RMSEA. no restriction upward method downward method no restriction upward method downward method. mean over RMSEA of items 1-30 number of items has min RMSEA. mean over RMSEA of items 1-30 number of items has min RMSEA. RM SEA. 0.0477 0.0370 0.0697 3 27 0. 0.0694 0.0615 0.0914 8 21 1. 0.1047 0.0939 0.1291 10 20 0. HO. 0.0350 0.0246 0.0417 5 23 2. 0.0514 0.0372 0.0624 4 26 0. 0.0805 0.0598 0.0949 6 24 0. 0.0634 0.0532 0.0878 8 20 2. 0.0882 0.0859 0.1090 13 10 7. 0.1254 0.1416 0.1405 19 2 9. Test 1 MVN1 MVN2. 0.0324 0.0230 0.0333 6 21 3. 0.0464 0.0319 0.0484 4 26 0. 0.0741 0.0524 0.0781 1 29 0. Unif. 1000. 500. 200. 0.0812 0.0678 0.0831 17 13 0. 0.1129 0.0897 0.1179 12 18 0. 0.1697 0.1258 0.1712 12 17 1. HO. 0.0524 0.0510 0.0523 4 18 8. 0.0775 0.0725 0.0771 8 19 3. 0.1332 0.1101 0.1301 6 20 4. 0.1050 0.0837 0.1195 17 13 0. 0.1358 0.1070 0.1538 18 11 1. 0.1922 0.1425 0.2012 14 15 1. Test 2 MVN1 MVN2. 0.0461 0.0458 0.0831 9 19 2. 0.0662 0.0649 0.0659 3 19 8. 0.1149 0.1042 0.1132 1 22 7. Unif. 0.0634 0.0532 0.0812 10 19 1. 0.0887 0.0788 0.1095 14 14 2. 0.1308 0.1282 0.1445 16 10 4. HO. 0.0430 0.0371 0.0469 10 17 3. 0.0643 0.0546 0.0714 5 22 3. 0.1057 0.0861 0.1137 7 23 0. 0.0818 0.0747 0.1105 18 8 4. 0.1133 0.1093 0.1340 19 6 5. 0.1563 0.1576 0.1687 18 10 2. Test 3 MVN1 MVN2. Table 8: Performance of RMSEA: Overall Mean RMSEA of All Items and the Number of Items with the Smallest Value. 0.0385 0.0334 0.0390 6 22 2. 0.0556 0.0479 0.0569 7 21 2. 0.0918 0.0752 0.0918 6 22 2. Unif.

(31) 25. no restriction upward method downward method no restriction upward method downward method. no restriction upward method downward method no restriction upward method downward method. mean over RMSEA of items 11-30 number of items has min RMSEA. no restriction upward method downward method no restriction upward method downward method. mean over RMSEA of items 11-30 number of items has min RMSEA. mean over RMSEA of items 11-30 number of items has min RMSEA. RM SEA. 0.0620 0.0457 0.0918 0 20 0. 0.0903 0.0774 0.1184 2 17 1. 0.1340 0.1164 0.1605 2 18 0. HO. 0.0430 0.0274 0.0527 0 20 0. 0.0631 0.0418 0.0782 0 20 0. 0.0995 0.0683 0.1167 0 20 0. 0.0855 0.0695 0.1179 0 18 2. 0.1184 0.1128 0.1436 3 10 7. 0.1655 0.1846 0.1789 9 2 9. Test 1 MVN1 MVN2. 0.0391 0.0250 0.0404 0 20 0. 0.0566 0.0349 0.0593 0 20 0. 0.0893 0.0572 0.0941 0 20 0. Unif. 1000. 500. 200. 0.1122 0.0904 0.1140 8 12 0. 0.1555 0.1166 0.1601 3 17 0. 0.2309 0.1573 0.2287 3 17 0. HO. 0.0686 0.0664 0.0683 3 14 3. 0.1017 0.0939 0.1010 4 15 1. 0.1745 0.1385 0.1706 2 18 0. 0.1487 0.1113 0.1688 7 13 0. 0.1903 0.1391 0.2132 9 11 0. 0.2650 0.1769 0.2719 4 15 1. Test 2 MVN1 MVN2. 0.0594 0.0589 0.1140 4 16 0. 0.0848 0.0830 0.0844 0 14 6. 0.1479 0.1320 0.1454 1 18 1. Unif. 0.0855 0.0695 0.1101 4 15 1. 0.1195 0.1026 0.1467 6 13 1. 0.1743 0.1629 0.1888 6 10 4. HO. 0.0550 0.0461 0.0606 3 15 2. 0.0826 0.0680 0.0926 1 18 1. 0.1365 0.1055 0.1458 0 20 0. 0.1134 0.0997 0.1531 8 8 4. 0.1565 0.1441 0.1827 11 6 3. 0.2132 0.2040 0.2251 8 10 2. Test 3 MVN1 MVN2. Table 9: Performance of RMSEA: Mean RMSEA of Items 11 to 30 and the Number of Items with the Smallest Value. 0.0483 0.0407 0.0492 1 17 2. 0.0701 0.0584 0.0718 2 16 2. 0.1153 0.0907 0.1149 1 19 0. Unif.

(32) 26. no restriction upward method downward method no restriction upward method downward method. no restriction upward method downward method no restriction upward method downward method. mean over RMSEA of items 21-30 number of items has min RMSEA. no restriction upward method downward method no restriction upward method downward method. mean over RMSEA of items 21-30 number of items has min RMSEA. mean over RMSEA of items 21-30 number of items has min RMSEA. RM SEA. 0.0847 0.0592 0.1434 0 10 0. 0.1248 0.1044 0.1799 0 10 0. 0.1783 0.1555 0.2266 2 8 0. HO. 0.0524 0.0298 0.0747 0 10 0. 0.0762 0.0456 0.1089 0 10 0. 0.1214 0.0763 0.1565 0 10 0. 0.1172 0.0953 0.1912 0 10 0. 0.1595 0.1498 0.2238 3 7 0. 0.2241 0.2428 0.2540 5 2 3. Test 1 MVN1 MVN2. 0.0473 0.0260 0.0531 0 10 0. 0.0682 0.0358 0.0784 0 10 0. 0.1066 0.0582 0.1213 0 10 0. Unif. 1000. 500. 200. 0.1678 0.1217 0.1649 0 10 0. 0.2232 0.1475 0.2183 0 10 0. 0.3225 0.1831 0.3085 0 10 0. HO. 0.0932 0.0890 0.0927 0 9 1. 0.1405 0.1244 0.1388 0 10 0. 0.2419 0.1732 0.2335 0 10 0. 0.2189 0.1395 0.2377 0 10 0. 0.2769 0.1634 0.2891 0 10 0. 0.3653 0.1989 0.3576 0 10 0. Test 2 MVN1 MVN2. 0.0779 0.0769 0.1649 2 8 0. 0.1115 0.1078 0.1105 0 9 1. 0.1966 0.1672 0.1892 0 10 0. Unif. 0.1252 0.0945 0.1652 0 10 0. 0.1725 0.1340 0.2143 2 8 0. 0.2389 0.2009 0.2626 4 6 0. HO. 0.0719 0.0575 0.0840 0 9 1. 0.1077 0.0845 0.1279 0 10 0. 0.1793 0.1267 0.1987 0 10 0. 0.1619 0.1252 0.2198 3 7 0. 0.2194 0.1777 0.2572 5 5 0. 0.2859 0.2425 0.3012 5 5 0. Test 3 MVN1 MVN2. Table 10: Performance of RMSEA: Mean RMSEA of Items 21 to 30 and the Number of Items with the Smallest Value. 0.0617 0.0498 0.0647 1 8 1. 0.0886 0.0706 0.0946 0 10 0. 0.1483 0.1083 0.1505 0 10 0. Unif.

(33) 27. 0.8506 0.9721 0.9464. 0.8714 0.9785 0.9648. 0.8767 0.9887 0.9835. no restriction upward method downward method. no restriction upward method downward method. no rectriction upward method downward method. HO. 0.8645 0.9746 0.9720. 0.8520 0.9724 0.9638. 0.8404 0.9654 0.9412. 0.8918 0.9820 0.9625. 0.8867 0.9749 0.9548. 0.8628 0.9605 0.9401. Test 1 MVN1 MVN2. 0.8586 0.9622 0.9614. 0.8530 0.9603 0.9570. 0.8322 0.9542 0.9394. Unif 0.8269 0.8663 0.8884 0.8621 0.8936 0.9332 0.8714 0.9074 0.9583. 500 0.8409 0.7987 0.9295 0.9474 0.9653 0.9506 1000 0.8557 0.8122 0.9498 0.9547 0.9234 0.9347. Test 2 MVN1 MVN2. 200 0.8036 0.7629 0.8873 0.9143 0.9234 0.9347. HO. 0.7872 0.9457 0.9291. 0.7768 0.9437 0.9435. 0.7304 0.9317 0.9291. Unif. 0.8897 0.9770 0.9258. 0.8819 0.9577 0.9600. 0.8602 0.9088 0.9258. HO. 0.8581 0.9690 0.9429. 0.8519 0.9670 0.9619. 0.8340 0.9408 0.9429. 0.9019 0.9477 0.9151. 0.8941 0.9154 0.9375. 0.8714 0.8925 0.9151. Test 3 MVN1 MVN2. 0.8383 0.9616 0.9471. 0.8308 0.9601 0.9585. 0.8049 0.9555 0.9471. Unif. Table 11: Performance of Classification Accuracy of the Three Methods for Each Distribution of Cognitive Patterns, Type of Items, and Sample Size.

(34) 5. Analysis of Empirical Data In addition to the simulation study, we also apply and illustrate the proposed method in analyzing empirical data. The data are the mathematical stage test from a senior high school in Taipei and the percentile rank of the students in this school is higher than 96. The test responses of 309 students in seven classes were collected. As shown in Table 12, the Q-matrix of this test contains 11 items and 9 attributes. Some items in the test belong to the DINA model items and some are A-CDM items, so these test data are relatively similar to the test 3 in our simulation. Table 12: Q-matrix for the Empirical Data item K1 1 0 2 1 3 1 4 1 5 1 6 1 7 1 8 1 9 1 10 1 11 0. K2 0 1 0 0 0 1 1 0 1 0 0. K3 0 1 1 0 0 0 0 0 0 0 0. K4 0 0 0 1 1 0 1 0 1 0 0. K5 0 0 0 0 0 1 0 0 0 0 1. K6 1 0 0 0 0 1 0 1 0 0 0. K7 1 1 1 0 0 0 0 1 0 0 0. K8 1 0 0 0 0 0 0 0 0 0 1. K9 1 0 1 0 0 0 0 0 0 1 1. Tables 13 and 14 respectively report the probability of getting correct answers for each corresponding cognitive pattern and the parameter estimates of the G-DINA model obtained without order restrictions. We can easily see that find some counterintuitive phenomenon in Table 13. For the example of item 3, the probability of getting a correct answer for respondents with the cognitive pattern of mastering none of the attribute item 3 involving is 0.52, whereas the probability reduces to 0.33 for respondents with the cognitive patterns of mastering only the first attribute of item 3. In other words, we have for item 3, (1, 0, 0, 0) ≻ (0, 0, 0, 0) but P1000 < P0000 . Take another example of item 7, the probability of getting a correct answer for cognitive pattern (1, 0, 0) is 1.00, but the the probability is only 0.94 for the cognitive pattern (1, 1, 1). In other words, we have for item 7, (1, 1, 1) ≻ (1, 0, 0) but P111 < P100 . All the counterintuitive phenomenon in Table 13 are marked in shaded gray. Use the relationship between δ j and P j mentioned in section 3, we present the results of δ j in Table 14, for j = 1, 2, · · · , 11 to help us directly and conveniently observe the main effects. From Table 14, we find that items 3, 8, and 10 have negative main effect estimates, and the proportion of negative main effects is slightly less than test 3 in Table 4. In addition, even for respondents who have mastered all the attributes that items 4 and 5 involve, the probabilities 28.

(35) of answering these items correctly are less than satisfactory. This could probably result from the fact that these items are so-called application questions and respondents might have mastered all the attributes but do not know how to apply them in solving a problem, therefore, the probability of answering correctly appears to be lower than expected for those who have mastered all the attributes. According to the simulation results in section 4, the upward method performs better than the downward method. Therefore, we compare only the results obtained from the method without order restrictions and the upward method while analyzing the empirical data. The resulting probability of answering correctly P j are reported in Table 15 and the estimates of δ j are reported in Table 16, for j = 1, 2, . . . , 11. We can see that the counterintuitive phenomenon in Table 13 has been solved here. In Table 15, the probabilities fully meet the order restrictions, Pj (αl(j) ) ≥ Pj αl′ (j) when αl(j) ≻ αl′ (j) , for j = 1, 2, ..., 11. We note that there exists no negative main effect estimates in Table 16.. 29.

(36) 30. P0 P00 P000 item P0000 1 0.14 2 0.00 3 0.52 4 0.05 5 0.00 6 0.05 7 0.17 8 0.05 9 0.40 10 0.27 11 0.00. P1 P10 P100 P1000 0.75 0.21 0.33 0.13 0.00 0.44 1.00 0.64 0.94 0.00 1.00. P01 P010 P0100 0.58 0.00 0.52 0.05 0.00 0.05 0.17 0.00 0.40 1.00 0.00. P11 P001 P0010 0.14 0.00 0.50 0.45 0.62 0.55 0.17 0.35 0.40 1.00 0.00 P101 P1100 0.80 1.00 0.33. 1.00 0.82 0.00 0.00 0.47. P110 P0001 1.00 0.61 0.00. 0.75 1.00 0.00 0.28 1.00. 0.05. 0.87 0.17 1.00 0.40. P011 P1010 0.75 0.21 0.85. 1.00. 0.96 0.94 1.00 1.00. P111 P1001 1.00 1.00 0.66. 0.55. P0110 0.58 0.00 0.50. 0.75. P0101 0.88 0.61 0.00. Attribute pattern. 0.00. P0011 0.51 0.61 0.87. 0.94. P1110 0.80 1.00 1.00. 0.92. P1101 0.08 0.00 0.66. 1.00. P1011 0.00 0.95 0.81. 0.00. P0111 0.66 0.61 0.87. 1.00. P1111 0.95 1.00 1.00. Table 13: Estimated Probability of Answering Correctly for Each Cognitive Pattern of the G-DINA Model without Order Restrictions for the Empirical Data.

(37) 31. δ0 δ00 δ000 item δ0000 1 0.14 2 0.00 3 0.52 4 0.05 5 0.00 6 0.05 7 0.17 8 0.05 9 0.40 10 0.27 11 0.00. δ1 δ10 δ100 δ1000 0.61 0.21 -0.19 0.08 0.00 0.39 0.83 0.58 0.54 -0.27 1.00. δ01 δ010 δ0100 0.44 0.00 0.00 0.00 0.00 0.00 0.00 -0.05 0.00 0.73 0.00. δ11 δ001 δ0010 0.00 0.00 -0.02 0.33 0.62 0.50 0.00 0.29 0.00 0.27 0.00 0.56 -0.07 -0.18 0.00 -0.93 0.71 -0.94 0.00 -0.53. 0.70 0.00 -0.58 -0.66 0.00. 0.05. δ101 δ1100 -0.39 0.79 0.00. δ110 δ0001 0.86 0.61 -0.52. δ011 δ1010 0.00 0.00 0.54. 0.48. -0.18 0.13 0.93 1.66. δ111 δ1001 -0.61 0.17 0.85. 0.00. δ0110 0.00 0.00 0.00. Attribute pattern. 0.00. δ0101 -0.55 0.00 0.00. -1.25. δ0011 -0.49 0.00 0.90. -0.50. δ1110 0.00 0.00 0.15. -0.60. δ1101 -0.42 -1.79 0.00. Table 14: Parameter Estimates of the G-DINA Model without Order Restrictions for the Empirical Data. 0.86. 0.00. δ1011 δ0111 -0.51 0.27 -0.05 0.00 -1.27 0.00. 0.53. δ1111 1.60 1.05 0.04.

(38) 32. P0 P00 P000 item P0000 1 0.04 2 0.00 3 0.56 4 0.09 5 0.02 6 0.09 7 0.35 8 0.06 9 0.33 10 0.28 11 0.00. P1 P10 P100 P1000 1.00 0.44 0.61 0.14 0.08 0.95 0.80 0.54 0.67 0.28 1.00. P01 P010 P0100 0.63 0.00 0.56 0.09 0.02 0.09 0.35 0.21 0.33 1.00 0.00. P11 P001 P0010 0.04 0.00 0.56 0.54 0.70 0.57 0.35 0.50 0.33 1.00 0.01 P101 P1100 1.00 0.48 0.61. 1.00 0.84 0.54 0.67 1.00. P110 P0001 0.04 1.00 0.56. 0.25 1.00 0.54 0.67 1.00. 0.39. 0.95 0.35 0.50 0.33. P011 P1010 1.00 0.44 1.00. 1.00. 1.00 1.00 1.00 1.00. P111 P1001 1.00 1.00 0.80. 0.57. P0110 0.63 0.00 0.56. 0.25. P0101 0.69 1.00 0.56. Attribute pattern. 0.57. P0011 0.04 1.00 0.66. 1.00. P1110 1.00 0.48 1.00. 1.00. P1101 1.00 1.00 0.80. 1.00. P1011 1.00 1.00 1.00. 0.57. P0111 0.69 1.00 0.66. 1.00. P1111 1.00 1.00 1.00. Table 15: Estimated Probability of Answering Correctly for Each Cognitive Pattern of the G-DINA Model using Upward Method for the Empirical Data.

(39) 33. item 1 2 3 4 5 6 7 8 9 10 11. δ0 δ00 δ000 δ0000 0.04 0.00 0.56 0.09 0.02 0.09 0.35 0.06 0.33 0.28 0.00. δ1 δ10 δ100 δ1000 0.96 0.44 0.05 0.05 0.05 0.85 0.45 0.48 0.35 0.00 1.00. δ01 δ010 δ0100 0.60 0.00 0.00 0.00 0.00 0.00 0.00 0.15 0.00 0.72 0.00. δ11 δ001 δ0010 0.00 0.00 0.00 0.40 0.63 0.47 0.00 0.44 0.00 0.00 0.01 δ101 δ1100 -0.59 0.03 0.00. 0.05 0.04 -0.44 0.00 -0.01. δ110 δ0001 0.00 1.00 0.00. 0.16 0.20 -0.15 0.00 0.00. 0.39. -0.47 0.00 -0.15 0.00. δ011 δ1010 0.00 0.00 0.39. -0.39. -0.11 -0.04 0.61 0.33. δ111 δ1001 0.00 -0.44 0.18. 0.00. δ0110 0.00 0.00 0.00. 0.00. δ0101 0.05 0.00 0.00. Attribute pattern. -0.16. δ0011 0.00 0.00 0.10. 0.00. δ1110 0.00 0.00 0.00. -0.05. δ1101 -0.05 -0.03 0.00. Table 16: Parameter Estimates of the G-DINA Model using Upward Method for the Empirical Data. 0.16. δ1011 0.00 0.00 -0.28. 0.00. δ0111 0.00 0.00 0.00. 0.00. δ1111 0.00 0.00 0.00.

(40) 6. Discussion and Conclusion The purpose of this study is to propose an improved algorithm for the estimation of the G-DINA model to avoid the counter-intuitive phenomena that a lower probability of getting a correct answer is obtained for the students who have mastered more attributes than those who know less. We evaluate the performance of our proposed algorithms through a simulation study of which a number of factors were manipulated, including item type, distribution of the cognitive patterns, and sample size. First of all, we can almost affirm that such a counter-intuitive phenomenon would not occur on the items which intuitively involve only one attribute. That is, for items involving only one attribute, there is no need to consider the order restrictions. Because the Q-matrix describes the relationship between the items and the attributes, the above counter-intuitive phenomenon occurring on an item involving only one attribute indicates that mastering this sole attribute in fact decreases the probability of answering the item correctly. This would imply that the attribute is very likely to be mistakenly identified as associated with the item in the Q-matrix. That is, there are factors other than the attribute affecting the probability of getting a correct answer for that item. Our results show that if we focus on the items involving two or more attributes, the estimates obtained from the upward method perform the best for both MAD and RMSEA regardless of the factorial conditions. The estimates from the upward method not only on average have smaller MAD and RMSEA than the other two methods, but also have the smallest MAD and RMSEA in most of the items involving three attributes. In other words, the greater the number of attributes an item involves, the more salient the advantages of using the upward method for estimation is. As for classification accuracy, although the classification accuracy of the downward method is higher than the upward method in some cases, the upward method is still better than estimation without restrictions in all cases. If the researcher accepts the classification accuracy of the existent estimation method without restrictions, they should find the classification accuracy of the upward method satisfactory. Moreover, the convergence of the upward method does not take relatively longer time compared to that with no order restrictions. Especially when all the estimates naturally fall in the permissible parameter space of the order restrictions, the same maximum likelihood estimates will be obtained and adding the order restrictions in the estimation algorithm has no effect on the estimates. In summary, if the Q-matrix of a test contains items involving two or more attributes, it is recommended to use the G-DINA model with order restrictions and estimate the parameters with the upward method. In fact, the problem we have raised and discussed in this study is not just for the G-DINA model, it should be applicable to some other CDMs such as the log linear cognitive diagnosis model (Henson, Templin, & Willse, 2009; Rupp & Templin, 2008; von Davier, 2005). Although the “monotonicity” property that respondents with a superior attribute pattern should have a larger probability 34.