The Design and Implementation of Intelligent Assessment Management System

The Design and Implementation of Intelligent

Assessment Management System

Abstract—Designing a high-quality assessment system is a

challenge for college-level educators. An effective assessment system requires mastering the professional knowledge and skills involved. The field of assessment and evaluation has developed many important concepts, principles, and methods to guide practice. This study proposes an intelligent assessment management system for the summative assessment of college courses. It integrates a virtual online assessment management system and in-classroom examinations. Before an exam, the scope of the examination is selected by teachers, and students who will take the exam are listed. According to the number of students, the system configures the range of examination rooms. Students can go to any of the examination rooms to take the exam during the exam period. Each table has a near-field communication (NFC)-enabled tablet for student identification via a student APP. The identification process uses digital certificates to ensure uniqueness and to avoid substitute testers. Invigilators use an exam management APP to access information about the students taking the exam in the exam room, and give feedback regarding the real-time situation to the system. Instructors use the assessment management APP to check students’ situation after the exam. The system ensures that all students receive fair treatment during the exam. The proposed system improves the effectiveness of assessment.

Keywords: assessment management system; summative

assessment; in-classroom examinations; near-field communication (NFC); APP; digital certificate; assessment effectiveness

I. INTRODUCTION

As on-line education has recently become one of the popular subjects in higher education research and application, many online learning system [1], [2] based on the web-based education mode have been used to help participation of students with materials of some specific courses. To cover all the phases of learning and assessment processes, these systems are usually comprised of some functional subsystems such as the Web Course System, Online Q&A (Question and Answer) System, and Online Testing System. However, typical online learning systems [3], [4], [5], [6] arrange and carry out the learning resources and the assessment activities mainly through virtual interactions and environment. It is difficult for these systems to meet the needs of students from different backgrounds and to be consistent with the rules of teaching and assessment. It is hard for on-line systems to confirm the personal identification effectively, and monitor students’ activities during the exam, because they cannot cope with the

violation activities, such as taking a test for someone else or cheating in a test.

In the case of learning and student development, a detailed understanding of the functioning of orientation, curriculum, instruction, academic advising, and other key educational processes is necessary for the improvement of institutional results. In fact, students’ learning results which generated from the typical online learning system are lack of the correctness and the fairness, and a great deal of data accumulated during the teaching and learning process have not been made full use, which contains much meaningful information of students, teachers and the education process. Thus an effective system should be established to promote the teaching and learning activities as well as to improve the performance of teaching and learning. In the paper, an Intelligent Assessment Management System is constructed. Aimed to the practice of current summative evaluations such as midterms and final exams, this new system provides a hybrid structure for integrating virtual online assessment management system and in-classroom examinations.

II. SYSTEM DESIGN

Compared with other assessment or evaluation systems, the Intelligent Assessment Management System is distinguished for its major features as follow:

A. A comprehensive and integrated framework

Intelligent Assessment Management System

Database Assessment Management APP Student APP Invigilator APP

Intelligent Assessment Management Framework

Location Management

Module Authentication Module Assessment Management Module Management ModuleExamination Space Examination Activities Course Instructor Assessment Manage i-Tag( NFC Tag) Digital Certificate

Figure 1. System Architecture of the Intelligent Assessment Management System

Yu-Chun Lu*, Yu-Sheng Yang**, Ping-Chun Chang*, Chu-Sing Yang*

*Department of Electrical Engineering, National Cheng Kung University, Tainan, Taiwan. lur.ncku@gmail.com

What the term “comprehensive” means is that we apply different technologies and applications, such as tablet, digital certificate technique, NFC (Near Field Communication) technology and various management APPs, for promoting the performance of the system. On the other hand, the term “integrated” implies that during the progress of evaluation, the system gets information from not only students but also course instructors and invigilators, and the fairness and effectiveness of the assessment will be improved. Figure 1 shows the system architecture of our system.

B. An adaptive evaluation module for both students and invigilators

In the Intelligent Assessment Management System, meaningful information is no longer limited in the static information, but will be gathered from the users’ behavior and actions, which is relatively dynamic. As figure 2 shows, when course instructors design the summative assessment, the system provides assessment management App to them as a tool for planning and making questions for the midterms and finals. Students who take the exam can freely choose any qualified examination rooms during specified exam time period without the class limitation, and then they can take the exam with the student APP after checking the i-Tag for the location and signing a digital signature for the personal identification. It means students now can take exams of different subjects in the same exam room. During the exam period, the invigilator in the exam room will use exam management APP to access real-time and related information of students who take exam in the exam room. At the same time, invigilators will be responsible for student identification check and report any violations during the exam with the invigilator APP.

qualified examination room

i-Tag( NFC Tag)

Student APP

Invigilator APP Assessment Management APP

Qualified Examination Rooms

Figure 2. Application Context of the Intelligent Assessment Management System

Next, we will illustrate the design of the system. As shown in the figure 3, course instructors, students, invigilators, and examination room managers are four main users of the system. Examination room managers among them have responsibility for administrating and maintaining the examination rooms and related facilities. The physical and virtual objects of the system consist of examination rooms, seats, and other related resources, and they are connected by introducing techniques of

i-TAG (intelligent Tag, a tag applied NFC communication technology), Digital Certificate, and Digital Signature

Users

Links

Resources

Figure 3. Relationships between managed objects

Figure 4 presents the four phases of exam management workflow, includes configuring examination space, arranging related affairs of examination, proceeding examinations, and processing follow-up activities after-the-exam.

Exam Space Configuration

• Configure i-Tag of examination rooms • Allot i-Tag of examination seats

Arrange Exam Affairs

• Develop test items of different courses • Arrange affairs of examination

Proceed Examinations

• Students choose any one of examination rooms to take test

• Invigilators check student identification and report violations during the exam • Course instructors monitor test information of their courses

Follow-up Activities

• Calculate test score

• Gather statistics and anlysis of effectiveness of learning and teaching • Provide resources of remidal learning

Figure 4. Workflow of the Intelligent Assessment Management System

 Phase 1: allotting examination rooms. Available and administrable examination rooms should be allotted before developing the examinations. Exam-rooms managers will configure i-Tag of each room and seat, so that users can access resources of the examination by using their mobile devices to induct i-Tag.

 Phase 2: arranging related affairs of examination. After allotting available examination rooms, course instructors decide the scope of the examination and select items from question bank of the system to generate a test. Then, they choose available examination time and rooms to reserve the exam event.

 Phase 3: proceeding examinations. In this phase, our system mainly provides different services and functions for students, invigilators, and course instructors. During examination period, students can choose any of available examination room and seat to take an exam. They will be positioned and identification checked by the system when they use mobile devices to induct i-Tag of the seat. When students use digital certificate to log in the system, system will assign a personal test with random items from assigned scope for each student. Students’

answers will be signed with digital signature and upload to the system after finishing the examination. On the other hand, invigilators have responsibility for keeping fairness and stability of the examination. Their main task is verifying student identification and personal information, and return verified results to the system. Invigilators also record the violation of exam regulations, and report the event to the system. Besides, course instructors can use the system to inquire the location of the student who takes the course exam, they also can figure out the answer condition of the student timely.

 Phase 4: follow-up activities after the exam. When examinations come to the end of the stage, course instructors can use the system to do online checking. System will automatically offer test results to the instructor for the question types who has standard answers (e.g. alternative response, multiple choice questions, or completion type); instructors do hand scoring for other question types (e.g. essay question, or calculation questions). Students will get personal grade and all correct answers of a test after the course instructor public the results of the examination. Moreover, related remedial resources will be provided to each student.

The main goal of each design of our system is for promoting evaluation effectiveness of the exam. In order to achieve this goal, the design of personalized test, i-Tag position, digital signature, and invigilator App which used to reduce cheating behaviors and other violations are used to ensure the fairness of the examination. System implementation will be described in the following section.

III. SYSTEM INPLEMENTATION

Intelligent Assessment Management System

Assessment Management APP Student APP Invigilator APP

Intelligent Assessment Management Framework

Location Management

Module Authentication Module Assessment Management Module Management ModuleExamination Space

Figure 5. System Components

The structure of our intelligent assessment management system contains of four main objects: Location Management Module、Authentication Module、Assessment Management Module、Examination Room Management Module, which are managed by Intelligent Assessment Management Framework.

 Examination Space Management Module: This module provides functions of arranging available examination time, rooms and seats, and corresponding i-Tag of each room and seat.

 Authentication Module: This module provides services for user login, verification of digital certificate and signature, and user profile management.

 Assessment Management Module: This module provides functions of question bank establishment, test developing, test grading, and results analysis and feedback.

 Location Management Module: This module provides information of user location and related location records.

Excluding above four modules, three Applications are developed on the Android platform for helping user to access each services: Assessment Management APP, Student APP, and Invigilator APP. We will make a description of the relationship between system and each user APP.

A. Examination space configuration

Figure 6. Qualified Exam room illustration diagram

As Figure 6 shown, before a general classroom can be utilized as a manageable examination room, exam room managers have to configure the i-Tag of each room and seat. Thus, users can access resources of the examination by using their mobile devices to induct the i-Tag. The “inducting” behavior also helps the management system to collect the position information of users. Figure 7 and Figure 8 illustrate are screenshots of using Assessment Management APP to configure i-Tag data of examination rooms and seats.

Figure 7. Configuring i-Tag information for a Exam Room

Figure 8. Configuring i-Tag information for the Seats in the Exam Room

B. Arrange examination affairs

Add New Exam Activity _{Set up Exam Info}

Set up Questions Scope and Partition

Manual Selection

Set up Exam Parameters of Time and Location

Course Instructor

Random Selection

Submit New

Exam Activity 學生可參與此項測驗

Figure 9. Workflow for Arranging a New Exam Activity As shown in Figure 9, when the course instructor wants to create a new exam, basic exam information including topic and participated classes should be confirmed firstly. Then the instructor has to develop the content of exam, such as exam scope, question types, difficulty, and numbers and partitions of questions. A test can be generated either by the instructor selects test items manually from question bank, or by the system automatically based on given parameters. Finally, the course instructor decides the date, time, and classrooms of the

examination according the system information of available examination rooms. The screenshots of planning exam activity is depicted in Figure 10. Figure 11 shows the summative information of the exam after creating a new exam.

Figure 10. Configuring New Exam Activity

Figure 11. Summative Information of Examination Activity In addition to provide question bank to instructors generate the exam content, our system also provide the function of formulate questions so that the instructor can add new question to question bank.

Create New

Questions Select Question Type

Add Question Description

Add Answer to the Question

Setup Question Subject, Difficulty and other

parameters

Question Bank

Submit New Questions

Course Instructor

Figure 12. Workflow for Adding New Questions

As shown in Figure 12, when the course instructor wants to add a new question, the type of question should be determined in the first. The system provides True/False questions, Multiple Choice, Fill-in-the-blank, Essay items, and Calculation

questions for user choosing. After deciding question type, the course instructor input the content and solution of the new item, and other parameters including scope and difficultly into the question bank. Figure 13 presents the screenshot of the course instructor add a new question by using the question bank management function in the APP

Figure 13. Adding New Exam Questions

C. Proceeding Examinations

In this section, we will describe the operations of the system by classifying users.

1) The Students

Sense i-Tag on the Desk Participate the Exam

student

Taking the Exam

Submit answer signed with digital Signature database Login with Signature Location Management Module Authentication Module

Figure 14. Workflow for Taking the Exam

Before taking the exam, the student can get exam information by using the Student APP to induct the examination room i-Tag which is configured on the door of each examination room. Figure 14 depicts the student have to use Student App to induct the seat i-Tag before starting to answer the test. While the student log in the system with the digital certificate and signature (as shown in Figure 15 and 16), Authentication Module will verify personal identification, and send i-Tag info to the Location Management Module for locating the position. Then the student can start to answer question after finishing verification.

Figure 15. User Digital Certificate Configuration Process

Figure 16. User Login with the Digital Signature

When the student complete all questions and press the submit button on the screen, Student APP will ask him to sign the answers with digital signature and upload information to the Assessment Management Module. The screenshot of the student use the answer function of the APP to answer questions is shown in Figure 17.

2) The Invigilators

Invigilators have the responsibility for keeping the fairness of the examination. As shown in Figure 18, the invigilator checks the personal picture and profile of the student which is provided by APP, and sends the verified results to the system server.

Figure 18. Examinee Identity Verification

Besides, invigilators have to record any of violation of examination regulations, and report related content to the system. Figure 19 presents the invigilator reported a violation of mobile phone rings during the examination.

Figure 19. Reporting a Violation (phone rings during the exam)

3) The Course Insturctors

During the examination period, Assessment Management APP provides the function for course instructors to inquire real-time information, such as the location of exam takers, student condition of answering questions, or the incident of violation. The screenshot of the course instructor inquire the real-time information of students who have entered examination rooms is depicted in Figure 20

Figure 20. Realtime information of students arriaving the examination rooms

D. Follow-up Activities

The system provides the follow-up activities to course instructors and students after the end of examinations by the Assessment Management APP and Student APP. We will explain t the operations of the system by classifying users.

1) The Course Instructors

The instructor can use Assessment Management App to score a test. System will automatically offer test results to the instructor for the question types who has standard answers (e.g. True/False questions, Multiple Choice, or Fill-in-the-blank); and instructors do hand scoring for other question types (e.g. essay question, or calculation questions). Besides, the system provides the statistics and analysis information of answers to the instructors, shown as Figure 21, so that they can realize learning and teaching effects. The information also can help instructors to modify the teaching activities and provide remedial learning.

Figure 21. The Statistics information of Students’ Answers

2) The Students

Students will receive their grade and correct answers of questions by Student App after the instructor chooses to public the results of the exam. System will provide personalized remedial resource to each student automatically according the

results of exam. Figure 22 presents the student acquired his answers and correct solution by using the student APP.

Figure 22. Exam results and remedial material on the Student APP

IV. CONCLUSIONS

This paper proposed an intelligent assessment management system that combines a virtual online exam management system and in-classroom examination. The system integrates both advantages of digital and traditional assessments and guarantees fairness. To improve the effectiveness of assessment, the system collects feedback based on an adaptive module. Instructors can recognize each student’s needs accurately and provide individual remedies. The system expands the range of objects for evaluation and gives a more precise assessment.

ACKNOWLEDGMENT

This work was supported by the Center for Technologies of Ubiquitous Computing and Humanity, National Cheng Kung University, under projects from the Ministry of Education and the National Science Council (NSC-100-2218-E-006-033 and NSC-100-2218-E-006-017) of Taiwan.

REFERENCES

[1] Xiaofei Liu, Abdulmotaleb El Saddik, Nicolas D. Georganas, "An implementable architecture of an e-learning system," in IEEE Electrical and Computer Engineering Canadian Conference, vol.2, pp. 717-720, May 2003.

[2] Gang Chen, “The Research on Architecture of Intelligent E-learning System, ” in IEEE International Conference on Computer Science & Education, Nanning, pp. 1079-1081, July 2009.

[3] Hua Pang, Shu Yang, Liying Bian, “A Web Services Based Online Training and Exam System, ” in IEEE International Conference on Wireless Communications Networking and Mobile Computing, Dalian, pp. 1-4, October 2008.

[4] Bo Hang, “The Design and Implementation of On-line Examination System, ” in IEEE International Symposium on Computer Science and Society, Kota Kinabalu, pp. 227-230, July 2011.

[5] Wang Xiuying, Zhou Fang, “The Online Testing System With Explain Function As Students Answer Mistake, ” in IEEE International Symposium on IT in Medicine and Education, Cuangzhou , vol. 2, pp. 456-460, December 2011.

[6] Huiping Wang, Ruowu Zhong, “Design and Implementation of Online Exam system Based on Data Mining, ” in IEEE International Conference on Internet Computing and Information Services, Hong Kong, pp. 207-209, September 2011.