2.5 Computerized Adaptive Testing
2.5.2 Ability Estimation
Before selecting the next item with maximum information value to administer to an examinee, provisional ability estimate should be computed based on the responses to items that have been administered. The ability for an examinee is updated after administering each selected item. Finally, the estimate of ability is calculated based on all responses to all the administered items with meeting the stopping criterion. There are three significant ability estimators, namely, the maximum likelihood estimator (MLE), the maximum a posteriori (MAP) estimator, and the expected a posteriori (EAP) estimator.
Maximum Likelihood Estimator
Assuming an examinee response to I items, one can compute the likelihood function by multiplying the probability function for all I items under the assumption of local item independence. The likelihood function can be expressed as:
( ) ( ) ( ) [
1( ) ]
1 i,where X indicates a response vector; xi =1 if examinee get a correct answer to item I; and xi =0 otherwise. Because the probability for an item is between zero and one, the product of probability for administered items will become too small to maintain precision. To overcome the difficulty of numerical computation, a natural log of likelihood functions rather than raw likelihoods is often used in further calculations.
In addition, it is hard to find the maximum of the log-likelihood by setting first derivative of the log-likelihood at zero due to absence of a closed solution form for such a function. One popular alternative way to find maximum estimates is to use an iterative Newton-Raphson procedure. We have to compute the ratio of the first derivative to second derivative of log-likelihood, and then new ability estimate is obtained by taking the previous ability estimate minus the ratio. The iterative
procedure is repeated until the ratio is less than a pre-specified value such as 0.001.
The strength of the MLE is asymptotically efficient and unbiased, but it requires a large item pool and no trail level estimate will be available until the examinee has endorsed or not endorsed at least one item (Embretson & Reise, 2000).
Maximum A Posteriori (MAP)
To overcome the critical problem with the MLE that no trait level can be estimated for examinees with all-endorsed or all-not-endorsed response vectors, one may incorporate prior information for unknown parameters into the log-likelihood function to become a posterior distribution. Assume that the distribution for latent trait level has a normal distribution N
(
µ, σ2)
, in which the hyper-parameter for the distribution of unknown parameter can be obtained from empirical information or experts’ subjective judgment, then we can combine the log-likelihood function with the prior information into the formulation as follows:(
θ)
logL(
θ)
g( )
θ ,g ∣X ∝ ∣X (2.20)
where g
( )
θ is the prior distribution for trait parameter; logL(
θ∣X)
is the log-likelihood function for latent trait given response vectors; and g(
θ∣X)
is the posterior distribution for ability parameter for that examinee. Because thelog-likelihood function and normal prior function is not conjugated, the posterior distribution has no closed form solution such that iterative Newton-Raphson
procedure has to be implemented to find the maximum value. Except for the addition part of prior information in Equation (2.20), the algorism to find the value to
maximize the posterior function is the same as the MLE. The value of estimated ability that maximizes the posterior will equal the mode, and this is why the procedure is sometimes called Bayes modal estimation.
The advantage of the MAP estimator is that the ability estimates can be obtained with whatever responses the examinees have made. Moreover, the prior distribution can increase the precision of the trail level estimation. However, there are some drawbacks with the MAP estimator. The critical problem is that the MAP estimates are always biased when the number of items is small. Another issue is that the results of ability estimates will be seriously biased and misleading if we adopt the wrong prior distribution (Embretson & Reise, 2000). In sum, the more administered items to an examinee, the less the prior information can influence the ability estimate.
Expected A Posteriori
Similar to the MAP estimator, the EAP estimator incorporates prior information for unknown parameters into the log-likelihood function. The difference between the EAP and MAP estimators, however, is that the EAP estimator does not involve an iterative procedure and its Bayesian estimator is derived from finding the mean of the posterior distribution rather than mode in the MAP estimator or MLE (Bock &
Mislevy, 1982). The formula can be expressed in the form of an expected value as follows:
( ) ( ) ( )
(
θ) ( )
θ θ θ. θθ θ
θ d
d g L
g E L
∫ ∫
= X
X X
∣
∣ ∣ (2.21)
The strength of the EAP estimator is that this procedure is non-iterative and therefore computationally faster than iterative methods but only in the conditions of UIRT models. Based on Bayesian estimation, the EAP estimator can calculate finite trait level estimates for all response patterns. Furthermore, the EAP estimator has minimum mean square error over the population of ability (Bock & Mislevy, 1982).
On the other hand, the EAP estimator will be biased and misleading when there is a finite number of items and a wrong prior distribution to be implemented (Embretson
& Reise, 2000). In addition, Wainer and Thissen (1987) found that the ability estimates are seriously regressed toward the mean of the prior distribution when the number of the administered items is small. As with the MAP estimator, the effect of the prior information reduces with the increase in the number of administered items.