The purpose of this study was to develop and validate an MCAT system for use with the biology assessment in Indonesia. The validity and reliability of the MIRT scales and the measurement precision of the system were focused on in the study. It was found that the MIRT scales of the system are valid because their content has adequate coverage of the biology curriculum according to subject experts comprised of teachers and university professors, the scales have construct validity in measuring the six biology domains, and the items of the scales have a good item-model fit with the multidimensional random coefficients multinomial logit model. The reliability of the scales is assured by the results of the covariance, correlation matrix, and variance of the MIRT scales. In addition, the item fit with the MNSQ item difficulty and the descriptive statistics of the scale ability of good skewness and kurtosis strengthened the reliability of the scales.
This indicates that the finding is similar to Wang at all (2011) for their Chinese MCAT system in which RMSE MAP outperforms other ability estimation. This also means that, this Biology MCAT system is confirmed its performance already.
72
The results of evaluation indicate that the system using RMSE of MAP estimation work better than RMSE of MLE’s. This finding is similar to Wang at all (2011) for their Chinese MCAT system in which RMSE MAP outperforms other ability estimation of MLE’s and also EAP’s. Thus, it is confirm that this Biology MCAT system can perform properly and work better with RMSE of MAP estimations. Based on the result, this MCAT’s measurement errors are small enough, and the participants require fewer items to finish the test. Since the evaluation result shows that the MAP ability estimation has a better performance than the MLE estimation, this finding is also similar to Chen (2006).
It was also found that the bias estimation was low enough after only limited items were administered. Moreover, this constructed Biology MCAT System has a small measurement error when a participant takes the biology test. When the acceptable SE criteria ranges from 0.315 ~ 0.385 (Babcock & Weiss, 2012), the test can be ended after administering 34items of the MLE SE; however, the test could be stopped even earlier after administering 28 items if the MAP SE were used because it reaches the acceptable criteria earlier than the MLE SE. This MCAT system was integrated into an online system and can be an alternative for the Indonesian NE.
The online system was designed to be as user-friendly as possible, and it has a simple interface on its webpages. An examinee is required to have a username and password to proceed with the test. Instructions for each step are given to lead examinees to the end of the test. With the simplicity of its webpage interface, it is expected that the focus of the examinees would not be disturbed during the test.
73
The report card for the MCAT gives feedback to participants soon after the test, and the information is useful for remedial instruction. Unlike some traditional tests in which examinees must wait a particular number of working days to receive their results, the report card for this MCAT is an integral part of test accomplishment. Therefore, proficiency information, which shows the strengths and weaknesses of examinees in all biology domains, can be used for remedial learning and teaching. Examinees themselves can use the information to review their competencies. They should focus more on the biology domains in which their achievements are still low. Teachers can use the information to rearrange their teaching plans for classroom activities. Furthermore, if this MCAT system is used to test students’ competencies prior to the National Examination (NE), students would have more time to remedy their weaknesses before taking the NE.
This will create more opportunities to receive a passing grade on the NE.
The Biology MCAT System is a more convenient way to perform an examination than to deliver the P&P test to different parts of Indonesia. The MCAT system also minimizes the leaking of test materials, which is commonly experienced by traditional NE P&P tests, such as the Indonesian NE. For P&P testing, the leaking could take place when test materials (items) are printed, photocopied, or transported to or from the central test center for grading. These issues would not occur with the MCAT examination, which needs no printing, photocopying, or transportation. Furthermore, the MCAT system’s adaptive function is set to follow the progression of an individual test taker’s ability. In other words, all test takers will have different items, even though they may access the test
74
at the same time and from the same place. Thus, the chance of cheating is lower, and, therefore, the test results will be more accurate.
The online MCAT is also beneficial because of its simplicity. It can make NE much simpler regarding logistical issues in consideration of Indonesia’s wide demographic area and large number of NE participants. Hence, the MCAT online system would be more reliable compared to the NE P&P test. Thus, the objective to convert the NE into a tool to improve the quality of education in Indonesia can be achieved.