PBT Tester Revision and Combination

With the defined FSM model for PBT tester, the examinee can perform a sequence of actions on the PBT tester that can be used to assess the examinee’s software operating skills. The PBT tester revision and combination are used to design a PBT tester with different functionalities, where the revision is to revise the original PBT tester functionality or add new functionality to a PBT tester; and the combination is to combine different PBT tester by FSM union or concatenation to provide comprehensive software operating skill assessment. The revision and combination of a PBT tester are described as follows.

(1) Reusing an Existing PBT Tester

Algorithm: Revise the Functionality of a PBT Tester Input: The FSL constructed in Example 2.

Output: A PBT tester with revised functionality.

Parameter: ai is the action that needs to be revised.

Step 1. Import an existing FSL into a new PBT tester and name the FSL f1. Step 2. For each terminals t_i in f₁, find the terminals for a_i.

Step 3. For each ai, revise the corresponding action routines attached in the terminals found in Step 2.

Step 3. For each revised action routines in Step 3, configure the corresponding parameters for a_i.

Step 4. Apply parser generator to generate the corresponding PBT tester based on the f₁.

Example 4: Modify the functionality of an existing PBT tester.

Assume that the original PBT test item is “Set the title in bold”, the teacher want to apply the same test scenario to evaluate the examinee’s software operating skills and with a modification to “Set the title in italic”. For this example, we have to revise the terminal symbol e in Example 2, which is used to define the action type “list clicking on bold”. The details of symbol definitions are shown in Table 5. The terminal symbol e is revised to e’, which represents the examinee click the “italic” in the font style list.

Table 5: The descriptions of symbols used in “Set the title in italic”

Type Symbol Description

Non-terminal S, A, B The definitions are the same as those in Example 2 b-d The definitions are the same as those in Example 2 e’(revised) Click the “italic” in the font style list

Terminal

f-h The definitions are the same as those in Example 2

#a1 The definitions are the same as that in Example 2 Action Symbol

#a2 The definitions are the same as that in Example 2

Figure 9 and Figure 10 are the FSL and corresponding FSM, respectively. In the FSL, we need to redefine the terminal symbol e and configure the parameters for action routine so that the software run-time status can be visualized correctly.

S Æ b _{#a1 #a2} S | c _{#a1 #a2} S | d _#a2 A | h _#a2 A Æ e’ #a2 B

B Æ f _{#a1 #a2} B | g _{#a1 #a2} S

Figure 9: The FSL for “Set the title in italic”

click(c_#a2) click(d_#a2)

click(e_{#a1 #a2}) / arg_sytle

click(f_{#a1 #a2})/arg_sytle select(b_{#a1 #a2})

Figure 10: The FSM for “Set the title in italic”

Algorithm: Add New Functionality of a PBT Tester Input: Given the FSL in Example 2.

Output: A new PBT tester with new functionality.

Step 1. Import an existing FSL into a new PBT tester and name the FSL f2.

Step 2. According to MS Word Scene Ontology, add non-terminals to f_2.

Step 3. For each non-terminals in Step 2, add terminals and related action routines.

Step 4. For each action routines in Step 3, configure the corresponding parameters for visualization.

Step 5. Apply parser generator to generate the corresponding PBT tester based on the f2.

Example 5: Reuse an existing PBT tester and add new functionality

Assume that the original PBT test item is “Set the title in bold” and the teacher wants to add new functionality to original one “Set the title in bold and use Times New Roman as default font type”. The above algorithm can be applied.

In this example, we have to add one non-terminal C to represent the software run-time status for the font type setting. Besides, three terminals, i, j, and k, are added

which represent that the examinee clicks the font type dropdown list and selects the

“Times New Roman” font type or others. The details of symbol definitions are shown in Table 6.

Table 6: The descriptions of symbols used in “Set the title in bold and use Times New Roman as default font type”

Type Symbol Description

S, A, B The definitions are the same as those in Example 2 Non-terminal

C The state of showing the format menu.

b-h The definitions are the same as those in Example 2 i Click the “Font Type” dropdown list.

j Click the “Times New Roman” font type.

Terminal

K Click the other font type.

#a1 The definition is the same as that in Example 2 Action Symbol

#a2 The definition is the same as that in Example 2

Figure 11 shows the FSL of this example, we add several production rules to model the action sequence of setting the font type. Figure 12 shows the corresponding FSM.

S Æ b #a1 #a2 S | c #a1 #a2 S | d #a2 A | h #a2

A Æ e #a2 B

B Æ f #a1 #a2 B | g #a1 #a2 S | i #a1 #a2 C C Æ j #a1 #a2 B | k #a1 #a2 C

Figure 11: The FSL for “Set the title in bold and use Times New Roman as default font type”

click(c_#a2) click(d_#a2)

click(f_{#a1 #a2})/argsytle, type

select(b_{#a1 #a2}) click(e_{#a1 #a2})

/ arg_sytle ^click(k_{#a1 #a2})

/ arg_type

click(j_{#a1 #a2})/ arg_type click(i_{#a1 #a2})/ arg_type

Figure 12: The FSM for “Set the title in bold and use Times New Roman as default font type”

(2) The PBT Tester Combination

Algorithm: Design a PBT tester with different paths

Input: An FSM named, named m, and other FSMs, named mi, with the same start and final states.

Output: A new PBT tester with different path to complete the task.

Step 1. Import m and mi into a new PBT tester.

Step 2. For each transition t_i connect to the start state of m_i, add t_i to the start state of m; delete ti and start state of mi.

Step 3. For each transition t_i’connect to the final state of m_i, add t_i’ to the final state of m; delete ti’ and final state of mi.

Example 6: Construct a PBT tester with two paths to complete the task.

PBT allows the examinees to come up with the correct answer by different action sequences. For example, to insert a symbol (i.e., a comma) in a paragraph, the examinee has two options: 1) Click the “Insert” menu and insert a comma in the

“Symbol” window, or 2) Click the ”View” menu, enable the toolbar, and insert a comma. The examinee can complete the task by one of action paths above. Suppose

we have two PBT testers, one describes the action sequence in the first option and the other describes the second option. We can union these two PBT testers to generate a new one, as shown in Figure 13.

…

… Click_insert

Click_view

Click_comma

Click_comma

Start Finish

Action path

Figure 13: A PBT tester with two action paths

Algorithm: Design a PBT tester with trap paths

Input: An FSM named, named m, and other FSMs, named mi.The mi model the trap path that the examinee may navigate through in certain state of m.

Output: A new PBT tester with trap paths.

Step 1. Import m and mi into a new PBT tester.

Step 2. For each state in m, find the corresponding state that the trap path may occur.

Step 3. For each state found in Step 2, add a state transition the start state of mi.

Example 7: Construct a PBT tester with a trap path.

When the examinee performs a sequence of actions to complete the task, some misconceptions may occur. For example, to format the text, if the examinee forgets to select the text first, the format of the text will not change. Therefore, we can design a

“trap path” and combine with the correct path. That is, the examinee can also navigate through the “trap path”, but the action sequence performed on the “trap path” will not affect the results. In order to highlight the misconception, we can also design action routines to record what the examinee performs. Figure 14 shows corresponding PBT tester.

Start _… Finish

Click_Format

Select_Text

Trap path

Correct path Click_Format

Click_OK Click_OK

Figure 14: The PBT tester with a trap path

Algorithm: Design a PBT Tester with Comprehensive Skill Assessment.

Input: An FSM named, named m, and other FSMs, named mi, where mi is a PBT tester that Is used to assess different software operating capabilities.

Output: A new PBT tester with comprehensive software operating skills.

Step 1. Import m and mi into a new PBT tester.

Step 2. For each m_i, add a state transition from the finish state of m_i-1 to the start state of mi.

Step 3. Configure the corresponding visualization of m_i if necessary

Example 8: Construct a PBT Tester with comprehensive skill assessment.

Some performance-based test item may contain comprehensive software

operating skills. For example, to create a business document, the examinee may need to know how to create a table to show the sales amount in the end the month, format the related text, and create a Table of Contents etc. To design a PBT tester for this kind of test item, we can combine several PBT testers to generate a more complex one, as shown in Figure 15.

… Format the text

Merge/Split cell Start

Finish Click_Format

Select_Text

Click_Table

Figure 15: The PBT tester combination