2.1. Computer-based Testing
The traditional multiple-choice test, administered via paper and pencil, provides a highly constrained testing environment. With dynamic visuals, sound, and user interactivity, computer based testing are being considered as alternative means for more effective testing [5-9], compared to traditional paper-and-pencil test. Innovative computerized test items [8] which refer to item types that use the computer’s capabilities to improve measurement and assessment have been introduced over the past two decades. Zenisky and Sireci [12] gave examples to illustrate drag-and-drop, moving objects to create a tree structure, and several other response actions. In [13], a taxonomy or categorization of 28 innovative item types that may be useful in computer-based assessment are introduced, which is organized along the degree of constraint on the respondent’s options for answering or interacting with the assessment item or task.
According to [8], it is possible to view innovations as falling into three categories.
First, new item types that improve the assessment of some ability or skill can be created. Bennett et al. [7] discussed this kind of item type. The second category includes assessments designed to improve the authenticity of the assessment.
Licensing exams that use computers to create high-fidelity simulations of professional activities, such as the clinical skills and the architectural licensing exams discussed in the next section fall into this category. Assessments of constructs not easily measured by conventional tests constitute the third category. Examples include tests of musical aptitude and interpersonal skills.
2.2. Computer-based Skill Assessment
Many skills and proficiencies are not easily assessed by paper-and-pencil multiple-choice test items. Consider, for example, clinical skills [11], [14], which take a history from a patient and perform a physical examination, or the design ability of an architect [10]. In recent years, the simulation technology has been applied to the certification exams of Microsoft Office Specialist (MOS), to realize PBT. Therefore, the examinees demonstrate their software operating skills by performing a sequence of actions on specific software to finish critical IT tasks in a simulated working environment, not using the actual software [15]. They have conducted that the simulation testing is a more effective way to evaluate the examinee’s software operating skills.
However, research has shown that these performance-based testing are difficult to administer and score, and have low reliability and limited generalizability.
Therefore, it is hard to construct and maintain a PBT tester for skill assessment. In this thesis, we focus on the software operating skill assessment, and try to propose a model to describe the PBT tester.
2.3. Different Approach to Construct PBT Tester
In this section, several approaches that can be used to construct a PBT tester are surveyed, including 1) standard-based approach, 2) object-based approach, and 3) screenshot-based approach.
z Standard-based Approach
The IMS Global Consortium, an industry and academic consortium, produced
the IMS QTI (IMS Question & Test Interoperability Specification), which uses an XML file to describe a basic structure for the representation of question (item) and test (assessment) data and their corresponding results reports [16]. Some questions, such as drag & drop and spot the error, in QTI specification define an image as a test item to let the examinees perform some actions (i.e., clicking, drag object) on that image. For example, for a given picture, the question may look like “The picture illustrates four of the most popular destinations for air travelers arriving in the United Kingdom: London, Manchester, Edinburgh and Glasgow. Which one is Glasgow?” [17], and then the examinee uses mouse to click at the correct location on the given picture. However, the QTI specification did not clearly define the structure and the operating flow of simulation questions for PBT.
z Object-based Approach
In [8], Shavelson et al. designed several objects, such as batter, light bulb, and voltage, to let the examinees perform their hand-on performance task on the computer.
Due to the special effort required to implement specific domain objects, it is hard to reuse the objects in different domains.
z Screenshot-based Approach
Kinnersley et al. [13] used Adobe Flash [20] to create a PBT tester to emulate MS Word and Excel. The tester is a timeline of MS Word or Excel screenshots to represent different software run-time status so that the examinee can perform some actions (i.e., clicking, drag object) on the screenshots, where the action performed by the examinee is to move from the current screenshot to the next screenshot in the timeline. Since Flash is not originally designed for software simulation, all of the screenshots need to be gathered, cropped and sized, and the mouse events
corresponding to the actions performed by the examinee also hove to be implemented.
Therefore, it is very time consuming for a tester developer to build a PBT tester.
Adobe Captivate [21] was used to create a tester for software skill exam. When the teacher wants to create a tester, he/she opens the target software to perform a sequence of actions from the starting state of the software to the final state of the software. During the sequence of actions, the Captivate automatically captures the screenshot and creates the sequence of screenshot based on the actions performed by the teacher. However, to model different operating paths, objects such as button or menu and the corresponding mouse events have to be added manually. Thus, it is hard to construct, reuse, and maintain the more complex simulation.