Experiment and Evaluation

6.2.1 Experiment Design

This thesis used the authoring tool to generate the different paper-review processes for journal or conference, and compared this thesis with current existing template-based paper/conference systems such as OJS, OCS, MyReview, and the traditional paper system such as IJDLT in supportability. For a reconfigurable paper-review system generator, there were many criteria of the dynamic system supportability. Thus, we also compared the system in our thesis with other template-based paper-review system in these criteria.

At first, we briefly introduced each paper-review process would be generate.

They might be collected by referencing the existing paper as well as collecting by an interview with the chief of a specific conference or journal.

Case 1: Paper-review process in [13]

The process was for a journal paper system. There were 3 roles in the system:

editor, reviewer, and author. User could register as an author then submit paper. Editor received the papers and dispatched them to no more than 3 other reviewers or editor for reviewing. The paper might be judge as accept, reject, major revise, or minor revise. When paper was accepted, it entered the proofreading process. Finally, it waited for being collected into the journal. There were several types mail sent to author, reviewer or editor automatically such as the revise deadline notification.

Editor could add/remove author or reviewer. In addition editor could define the category ontology for paper submission.

Case 2: Paper-review process in [2]

The process was for a conference management system. There were 3 roles in the system: administrator, reviewer, authors. The process was different with Case 1.

Author could withdraw a submitted paper. Editor could assign the reviewing job to review based on reviewers’ preferences. Administrator could only accept or reject the paper. There was only the judge result mail sent to author automatically. Finally, the administrator could manage the session of conference.

Case 3: Paper-review process in [3]

Fan et al. [3] presented an anonymous paper-review mechanism. In their paper-review process, there existed the anonymity between author, editor, and review each other. In case 1, all users were not anonymous with editor. (That is, editor could know who the author of a paper is or who the review of a paper was.) This thesis just

46

presented an anonymous paper-review process model. It didn’t propose a real system.

Case 4: Paper-review process in Conference A

This process was collected by interviewing with a conference chief editor. In their conference, there existed a hierarchical role definition. That is, chief editor could generate several session chairs for different topic of conference. Different session chairs could generate reviewers themselves. When author submitted paper to different topic in the conference, the corresponding session chair could dispatch the paper to reviewer under this topic. After each session chair published the accept papers. They could send back to the chief editor for the publication.

Case 5: Paper-review process in Journal B

This process was collected by interviewing with a journal editor. In their journal system, there exists another role “assistant editor”. Assistant editor was not responsible for paper dispatching. This role was only responsible for the proof reading task. On the other hand, when author submitted paper, the journal system asked user provide the information of number of words it helped end-user.

By observation of Case 1~5, we summarized different functionality of a paper-review system and listed the supportability in different case with different functionality as Table 6.1.

In addition to the supportability for the existing paper-review process, we compared this thesis with current existing template-based paper-review systems in the criteria of dynamic system supportability, these criteria are stated as follows.

a. Setup time: The construction time when start to construct the paper-review system a user needs.

b. Custom user profile: The ability of determining the user information should be provided when author register.

c. Custom paper attribute: The ability of determining the manuscript information should be provided when author submit paper.

d. Custom paper-review criteria: The ability of determining the criteria for evaluating paper when reviewing.

e. Dynamic roles definition: The ability of defining the role types in the paper-review system.

f. Knowledge sharing: The ability of constructing a paper-review system based on the others’ paper-review process.

g. Custom mailing rules: The ability of defining the automatically mailing rules and mail template.

47

h. Custom process control: The ability of defining the different paper-review workflow.

Table 6.1: The functionality in different paper-review process cases

Case 1 Case 2 Case 3 Case 4 Case 5

editor, reviewer,

author Yes Yes Yes Yes Yes

Assistant editor No No No No Yes

Hierarchical reviewing

process

No No No Yes No

Paper category

management Yes Not

mentioned

Not mentioned

Not

mentioned Yes

Submission Yes Yes Yes Yes Yes

Number of words

of paper No No No No Yes

Dispatching Yes Yes Yes Yes Yes

Reviewing Yes Yes Yes Yes Yes

Proof Reading Yes No No Yes Yes

Publish to journal Yes No No No Yes

Publish to

conference No Yes No Yes No

48 6.2.2 Experiment Result and Discussion

Table 6.2 showed the functionality supportability of this thesis and another paper-review system with the current existing paper review process.(Table 6.1) In Table 6.2, because the other paper-review system restricted the role definition in their systems, they could not generate the role “assistant editor” or define the hierarchical reviewing process. The MyReview System, OJS, OCS were partially support the paper category management because that they could only define one-level of paper categories. They could not define a paper sub-category under a specific paper category. For other systems, they were developed for the specific purpose, so they could not support to generate the journal system or conference system as users’ wish.

Table 6.2: The supportability in different researches with different functionality

RPS MyReview OJS OCS IJDLT

editor, reviewer,

author Yes Yes Yes Yes Yes

Assistant editor Yes No Yes Yes No

Hierarchical

reviewing process Yes No No No No

Paper category

management Yes Partially Partially Partially Yes

Submission Yes Yes Yes Yes Yes

Number of words of

paper Yes Yes Yes Yes No

Dispatching Yes Yes Yes Yes Yes

Reviewing Yes Yes Yes Yes Yes

Proof Reading Yes Yes Yes Yes Yes

Publish to journal Yes No Yes No Yes

Publish to

conference Yes Yes No Yes No

49

The comparison with the other template-based paper review systems in the criteria of dynamic system supportability was shown in Table 6.3.

Table 6.3: The comparison among different researches in the criteria of dynamic system supportability

RPS MyReview OJS OCS

Setup time cost Lower Higher Higher Higher

Custom user profile Support Not Support Not Support Not Support Dynamic roles

definition

Support Not Support Not Support Not Support

Custom paper attribute

Support Support Support Support

Custom paper-review criteria

Support Support Support Support

Knowledge sharing Support Not Support Not Support Not Support Custom process

control

Support Partially Not Support Not Support

Mailing template Support Support Support Support Custom mailing rules Support Not Support Not Support Not Support

For the criteria that this thesis powerful/weak than other system, we would describe why we had the following evaluating result.

Setup time cost

When users constructed their paper-review system through the other systems, they had to setup the web server and database environment first and then install the system and configure the system. In the system of this thesis, users just need make their configuration online. They did not spend any time on the web and database environment setup.

Custom user profile

For different research group, the users’ profile in their systems may be different because some information of user is not so concerned. By the help of frame system, we could make users define their own user profile definition.

Dynamic roles definition

Because other system thought that there were only 3 roles in a paper-review process, they thought that there is no need to generate other roles. However, by

50

defining different roles, we could define more paper-review process complicate such as hierarchical reviewing process.

Knowledge sharing

For the other systems, different outputted paper-review systems were installed in different server. Moreover, the system configuration and the data were stored in the database together. Therefore, they could not support the knowledge sharing. That is, they could not use the configuration made by other user and modify it to generate a new paper-review system.

Custom process control

For MyReview system, it could only define the papers’ judge-result in the output paper-review system. However, there were no dependency on the judge-result and the process control in MyReview system. Therefore, user still could not customize the paper-review process. In this thesis, by defining the paper state transition rules after each action in the paper-review process, users could generate their own paper review processes.

Custom mailing rules

For the other systems, they could define the mailing template and insert dynamic fields such as user’s name or paper’s title. However, except mailing to author at the time of paper was judged, they could only send mails manually. Based on the mailing rules in this thesis, users’ paper-review system could send the specific mails automatically at the specific time.

51