Per sonalized E-learning System by Using Item Response Theory

全文

(1)Per sonalized E-lear ning System by Using Item Response Theor y Hahn-Ming Lee1, Chih-Ming Chen2 and Ya-Hui Chen1 Department of Computer Science and Information Engineering National Taiwan University of Science and Technology, Taipei, Taiwan1 Institute of Information Science Academia Sinica, Taipei, Taiwan2 E-mail: hmlee@mail.ntust.edu.tw. Abstr act Personalized service is an important issue on the Web, especially on Web-based learning. In general, most personalized systems only consider learners’ preferences, interests, and browsing behaviors to provide personalized services. They do not consider learners’ abilities as an important factor to implement personalized mechanism. Besides, too many hyperlink structures on Web-based learning systems bring a lot of information burdens to learners. Hence, in Web-based learning, disorientation (losing in hyperspace), cognitive overload, lack of adaptive mechanism, the information overload problem, and the adaptation of courses materials difficulties are main research issues. Thus, we consider both the difficulties of course materials and learners’ abilities to provide personalized learning. This study adopts Item Response Theory (IRT) to estimate the learners’ abilities and rank appropriate course materials to learners. We also propose a collaborative voting approach to adjust the difficulties of the course materials in order to determine the difficulties of the course materials more objectively. Furthermore, learners’ abilities can be adjusted according to learners’ explicit feedback to achieve the goal of the personalized course materials’ recommendation. Experimental results show that IRT applied to Web learning can achieve personalized learning and assist learners to learn more effectively and efficiently.. 1.

(2) 1. Introduction In recent years, many kinds of applications are developed on the Web, such as portal websites [1], news websites [2], various commercial websites [3], and so on, while Internet is more and more maturity. Consequently, a fast growing information on the Web [4] results in information overloading problem [5] such that Internet users cannot find needed information exactly [6]. In order to help Internet users search more efficiently, many powerful search tools are proposed, such as Google search engine [1], Citeseer website [7]. Most of them provide personalized mechanism to prevent too many uninterested or irrelevant searching results for users. Restated, to consider personalized service has received considerable attention [8] in recent years because different users have different information needs. Furthermore, some recent surveys of network behaviors on Yam [9] show that most users apply search engines to find their desired information, and learning via Web environment is a growing trend. Learning via electronic appliances with Internet is called e-learning, also called distance learning, online learning (training) or Web-based learning, which helps users to learn by themselves through Internet. According to analyses at International Data Corporation (IDC) [10], the worldwide corporate e-learning market will exceed US$ 24 billion by 2004. The reason of e-learning becoming a trend is that it can provide convenient and efficient learning environments and practical utilities at anytime and anywhere. Many universities, corporations, and educational organization develop platforms of distance learning to provide course materials for Web-based learning. They are also often used for on-line employee training in business or distance learning in school. Similar to searching via Internet, Webbased learning needs personalized mechanism to help users learning more efficiently. Therefore, many researchers have taken efforts on providing personalization mechanism for Web-based learning in recent years [11][12]. Nowadays, most recommendation systems [13][14][12] only consider learners’ preferences, interests, and browsing behaviors to analyze users’ behaviors for personalized services. They do not consider learners’ abilities as an important factor to implement personalized mechanism. 2.

(3) Also, some researchers emphasized on the different levels of users knowledge in personalization, especially in learning aspect [11]. That is, each person may have different ability based on his or her major fields and subjects. If we can consider the users’ abilities, the performance of personalized learning might be promoted. Item Response Theory is a robust theory in education measurement. It is usually applied on Computerized Adaptive Test (CAT) [15][16] to select the most appropriate item to examinees based on various users’ abilities. At present, CAT has been applied to replace the traditional measurement instruments (which are typically fixed-length, fixed-content and paper-pencil tests) in several realworld applications, such as TOELF [17], GRE [18], and GMAT [19]. Based on previous analyses, a personalized e-learning system based on Item Response Theory (IRT) [15][16] which is called PEL-IRT is proposed to provide personalized e-learning service on the Web. In our approach, the abilities of learners and the difficulties of course materials are taken into account simultaneously. PEL-IRT can estimate users’ abilities by collecting the explicit feedback information after users have studied one course material. After performing the estimation of learners’ abilities, the system will recommend the appropriate course to the learners based on their abilities. In this manner, using our proposed e-learning system, users can get most appropriate course materials based on learners’ abilities and learn more effectively.. 2. Per sonalized Cour se Recommendation System In this section, we will describe our system architecture and personalized mechanism implemented by using Item Response Theory (IRT). We will first give the overview of our system architecture in Section 2.1. In Sections 2.2 and 2.3, we will describe the system’s components in detail. 2.1 System Architecture In our study, a personalized e-learning system based on Item Response Theory (PEL-IRT) is proposed to provide adaptive learning. Figure 1 depicts our system architecture. PEL-IRT can be 3.

(4) divided into two main parts according to system operation procedures, i.e. front-end and back-end parts. Front-end part manages communication with learners and records learners’ behaviors. On the other hand, back-end part aims at analyzing learners’ abilities and selecting appropriate courses according to the estimated values of learners’ abilities. In Figure 1, interface agent belongs to front-end part. It identifies learners’ status, transfers learners’ queries to system and returns the suggested results to learners. It can be served as a human-machine interactive interface. On the other hand, Item Response Theory (IRT) agent manages back-end operation. It can be divided into two separated agents: feedback agent and courses recommendation agent. Feedback agent aim at collecting learners’ feedback information, updates learners’ abilities, and adjusts the difficulties of course materials. On the other hand, courses recommendation agent aims at selecting most appropriate course materials to learners. Our system provides a searching and browsing interface that learners can retrieve course materials in a specified course unit. Everyone can browse course materials without login. However, personalized service now only is provided to registered learners. In what follows, the process of personalized learning will be illustrated. At the beginning, learners can login our system by registered accounts to obtain the personalization services. In our system, each course material is classified into a predefined course unit. Thus, learners must first select interested course unit or use keywords to search needed course materials. While a new learner visits our system, our system will assign general courses to learners. If learners click course materials and reply the predefined questionnaires, then personalized learning services will be started. Feedback agent will estimate learners’ abilities and adjust the difficulties of the course materials based on learners’ feedback information. Recommendation course agent will then use learners’ new abilities to select appropriate course materials to learners. The information function [16][20] is applied to select appropriate course materials. That is, course recommendation agent ranks course materials based on the information function’s values. While learners click the recommended course materials, IRT agent will repeat the recommended action until 4.

(5) learners give other query terms or logout this system. Additionally, PEL-IRT includes three databases: user account database, user profile database and courses database. In order to identify learners’ status, user account database records learners’ e-mail addresses. User profile database contains learners’ query terms, clicked behaviors, the responses of questionnaires, learners’ abilities and the difficulties of the clicked course materials. That is, all browsing information about learners is stored in user profile database. Courses database contains courses materials with different difficulties, clicked times of various difficulties, course category, course unit, course title and brief description of course material. Furthermore, the difficulties of course materials determined by the experts and learners’ collaborative voting approach [20] are detailed in Section 2.3. Contain courses with various difficulties Courses Database. 10. User Account Database IRT Agent 12 2 Courses Recommendation Agent. 4 1 User. 5/15. Interface Agent. 14. 3 a.Very Hard b.Hard c.Middle d.Easy e.Very Easy Yes or No System Architecture. User Profile 13 Database. 11 Feedback Agent. 6. 16. 9. 7. 8. Mapping learners' feedbacks into quantitative information.. Figure 1. PEL-IRT Architecture Moreover, the process of our proposed system can be depicted in Figure 2. First, system identifies. 5.

(6) learners’ status, if learners use the system at the first time, the system will provide the original course material list (non-personalization list) based on the retrieved results from learners’ query term. After learners visit some course materials and respond the given questionnaires, our system will estimates learners’ abilities, adjusts the difficulty of the selected course material, and recommend appropriate course materials to learners until learners logout this system. Detail description will be introduced in next section.. Beginner. Provide original relevant information by system Respond questionnaire. Visited pages Experienced User. Provide personal information according to the profile. Feedback agent. Analyzed result. Ranking Courses display. Courses recommendation agent. Profile and difficulty modification. Figure 2. The Process of PEL-IRT 2.2 Inter face Agent Interface agent provides a friendly interface to interact with learners and is served as an information channel to communicate with IRT agent. Interface agent includes the mechanisms of account management, authorization and query searching. While learners visit this system, they can select interested course categories and units in course database, and might give appropriate keywords to search course materials. If learners visit this system at the first time, they need to register in our system. At the beginning, our system only depends the query term to recommend course materials to learner according to the selected course category and unit. After learners login our system successfully and browses some interested course materials, they must reply some assigned questionnaires. Then, these replied answers are sent to the IRT agent to infer learners’ new abilities and suggest appropriate course 6.

(7) materials. These questionnaires contain two questions: one is “How do you think about the difficulty of this course material? ”, another is “Can you understand the content of the course material? ”. The first question includes five levels of choices: “very hard”, “hard”, “middle”, “easy” and “very easy”. The second question has two crisp options: “yes“ or “no“. Learners’ responses will be sent to the IRT agent to reevaluated the learners’ abilities and modify the difficulty of the browsed course materials. The related descriptions of the two questionnaires will be illustrated in Section 2.3. 2.3 IRT Agent After learners respond the given questionnaires, their responses are sent to IRT agent. IRT agent contains feedback agent and courses recommendation agent as shown in Figure 1. Feedback agent aims at recording learners’ feedback information and sending these learners’ responses to courses recommendation agent in order to evaluate learners’ new abilities values. After new abilities of learners are evaluated by feedback agent, course recommendation agent selects and suggests appropriate course materials to learners. In next subsections, we will first describe feedback agent, then illustrate courses recommendation agent in detail. 2.3.1 Feedback Agent Feedback agent records learners’ responses, analyzes learners’ abilities, and tunes the difficulties of course materials. It can communicate with interface agent and courses recommendation agent simultaneously. Also it contains three main operations: collecting learners’ feedback information, reevaluating learners’ abilities based on feedback information and updating course difficulties in course database. The detailed flowchart of feedback agent is shown as Figure 3. The collected information from interface agent includes learners’ e-mail addresses, the clicked courses ids and the learners’ answers to questionnaires. In PER-IRT system, the difficulties of courses materials are tuned based on the collaborative voting approach [21] and the learners’ new abilities are reevaluated by applying 7.

(8) maximal likelihood function [16][20]. The corresponding updated information will be sent to user profile database and courses database, respectively. Since learners’ abilities are reevaluated according to their feedback information, learners’ new abilities can be adjusted dynamically. In the meanwhile, learners’ new abilities are also sent to course recommendation agent as an index to rank course materials in course database based on information function [20]. Next, we will describe how to adjust the difficulties of course materials and how to estimate learners’ abilities.. Interface Agent. User Profile Database. Collect feedback information. Courses Database. Estimate user ability by using maximal likelihood function. Reevaluate courses difficulty. Feedback Agent. Courses Recommendation Agent. List of Courses materials. Figure 3. Operation Flowchart of Feedback Agent 2.3.1.1 Adjusting the Difficulty of Cour se Mater ials In order to recommend appropriate course materials to learners based on personalized requirement, Item Response Theory with single parameter, i.e., difficulty, is used to model a course material. In our 8.

(9) system, we consider both the difficulties of course materials and learners’ abilities because they will affect the learners’ interests and the learning results. We think that too hard or too easy course materials will make learners lose learning interests. In general, too hard course materials make users feel frustrating. On the contrary, too easy course materials make learners feel no any challenge and waste too much time on these course materials. Thus, providing appropriate course materials to learners is an important issue for any Web-based learning systems. In most Web-based learning systems, course materials’ difficulties are determined by course experts. However, it is not an appropriate approach because most learners are not course experts. In order to meet real needs, our proposed system adjusts the difficulties of course materials based on the collaborative voting approach [21] automatically. Namely, course experts first initialize the difficulties of course materials, and then the difficulties of course materials are adjusted according to the learners’ feedback. After a large number of learners use this system, the difficulties will gradually approach to reasonable and stable status. In fact, our system can effectively reduce the effect of noise or abnormal learners’ feedback information due to the proposed collaborative voting approach. In what follows, we will describe the procedure of adjusting the difficulties of course materials. In learners’ collaborative voting approach, 5-point Likert-scale proposed by Likert in 1932 is applied in our system [22]. In past researches, Likert-scale is used in attitude surveys. Likert-scale defines the scaled answers from “strongly disagree” to “strongly agree” based on the degree of a person agreement or disagreement with the question. The most common scale measure is defined from 1 to 5. Frequently, the scale 1 stands for “strongly disagree”, 2 is “disagree”, 3 is “not sure”, 4 is “agree” and 5 is for “strongly agree”. Based on Likert-scale, we define the scales: -2 stands for “very easy”, -1 represents “easy”, 0 is “Middle”, 1 stands for “hard” and 2 represents “very hard” in our system. As a user logins our system, the system will record the user’s browsing behaviors. After the user browses a course material suggested by our system, two questions must be reply. One is “How do you think about the difficulty of this course material? ”, another is “Can you understand the content of the course 9.

(10) material? ”. The first question includes five levels’ choices: “very hard”, “hard”, “middle”, “easy” or “very easy”. The reason of using 5-point Likert scale is that too many options items will make learners fell confusion, and too few options items cannot distinct from the difficulties of course materials. The second question has two crisp options: “yes“ or “no“. The reason is that our method needs yes or no pattern to evaluate learners’ abilities. Furthermore, the tuned course difficulty is a linear combination of the course difficulty defined by course experts and course difficulty replied by learners with different weight. In order to describe our proposed method, three definitions about the collaborative voting approach are described as follows:. Definition 3.1: Level of course material difficulty D is a vector of course material difficulty with five different difficult levels, and Di is the course difficulty of the i th difficult level, i=1… 5, D1 represents the level of course’s difficulty is very easy,. D2 stands for easy, D3 is moderate, D4 represents hard and D5 is very hard, and we quantify them as follows: D1 = −2, D2 = −1, D1 = 0, D1 = 1, D1 = 2 .. Definition 3.2: The average difficulty of the j th course material based on users’ collaborativevoting 5 n (1) b j (voting ) = ∑ ij Di i =1 N j where b j (voting) is the average difficulty of the j th course material after users’ collaborative voting,. nij is the number of users who give the feedback responses of the i th difficult level for the j th course 5. material, and N j denotes total number of users to rate the j th course material, and N j = ∑ nij . i =1. Definition 3.3: The tuned difficulty of course material b j (tuned ) = w × b j (initial ) + (1 − w) × b j (voting). (2). where b j (tuned ) is the tuned difficulty of the j th course material based on users’ collaborative voting , b j (initial ) is the difficulty of the j th course material given by course experts, and w is an 10.

(11) adjustable weight. Finally, our system can use Equation (2) to tune the difficulties of course materials in course database automatically. Equation (2) is a linear combination of the courses’ difficulties defined by course experts and the courses’ difficulties derived from learners’ collaborative voting. Moreover, the time complexity of computing the tuned difficulty of course material is constant because our system preserves all old voting results. 2.3.1.2 The Estimation of Lear ner s’ Abilities Before discussing how to estimate learners’ abilities, we first describe some assumptions in Item Response Theory. Assume that a randomly chosen learner responds a set of n items with response pattern (U1 , U 2 ,...,U j ,...,U n ) , where U j is either 1 or 0 on the j th course material. By the assumption of local independence, the joint probability of observing the response pattern is the product of. the. probabilities. of. observing. each. learner. response,. that. is,. P (U 1 , U 2 ,..., U j ,..., U n | θ ) = P (U 1 | θ ) P (U 2 | θ )...P (U j | θ )...P (U n | θ ), which may be expressed more compactly as, n. P (U1 ,U 2 ,...,U j ,..., U n | θ ) = Π P (U j | θ ), j =1. Since U j is either 1 or 0, this can be taken into account by writing the likelihood function as, n. P (U 1 , U 2 ,..., U j ,..., U n | θ ) = Π P (U j | θ ) j [1 − P (U j | θ )]. 1− U. U. j. ,. j =1. or simplify as, n. P(U1,U2 ,L,Un | θ ) = ∏Pj i Qj U. j =1. where Pj = Pj (U j | θ ) and Q j = 1 − Pj (θ ) . 11. 1−U j. (3).

(12) Equation (3) is an expression of the joint probability of a response pattern. When the response pattern is observed, U j = u j , the probabilistic interpretation is no longer appropriate; the expression for the joint probability is now called the likelihood function and is denoted as L(u1 , u2 , L, u n | θ ) where u j is observed response to the j th item. Thus, the estimated formula of learners’ abilities based on the tuned difficulty of course material is shown as follows: n. L(u1 , u 2 , L, u n | θ ) = ∏ P j (θ ) j Q j (θ ) u. 1−u j. (4). j =1. where. P j (θ ) =. (θ −b ( tuned )). e j , (θ −b ( tuned )) 1+ e j. and Q j (θ ) = 1 − P j (θ ) .. P j (θ ) is the probability that users can completely understand the j th course material under their abilities level θ , Q j (θ ) is the probability that users cannot understand the j th course material under their abilities level θ , and U j is the answer of yes or no obtained from users’ feedback to the j th course material, i.e. if answer is yes then U j = 1 ; otherwise, U j = 0 . Since P j (θ ) and Q j (θ ) are functions of θ and the item parameters, the likelihood function is also a function of these parameters. The learner’s ability can be defined as the maximum likelihood estimation of θ for that learner [20]. Thus, the method of maximum likelihood estimation mentioned in Equation (4) is applied to estimate the learners’ abilities. The maximum likelihood function needs two input parameters to evaluate learners’ abilities: one is the tuned difficulties of the course materials based on the collaborative voting approach, another is the yes or no responses of learners to the given questionnaires. Restated, the learners must give yes or no crisp response after they browsed a course material. In our system, learners’ abilities are limited between –3 to 3. That is, learners with ability θ = −3 12.

(13) are viewed as poorest, θ = 0 are viewed as middle, and θ = 3 are viewed as best. While learners login the system at the first time, our system will recommend the course materials with matching query term to learners. Then our system will adaptively adjust learners’ abilities according to the learners’ feedbacks. If learners can understand completely the content of the suggested course material, then learners’ abilities will be promoted based on the estimated formula of learners’ abilities mentioned in Equation (4), otherwise their abilities will be descended. Our system will send the new learners’ abilities to course recommendation agent, then course recommendation agent ranks a series of appropriate course materials in course database according to the new ability. In next subsection, we will introduce the courses recommendation agent. 2.3.2 Cour ses Recommendation Agent After feedback agent reevaluates learners’ abilities, course recommendation agent can recommend course materials to learners by using the new abilities estimated by feedback agent. The relationship of course recommendation agent with feedback agent is shown as Figure 4. Based on the information function [20], shown in Equation (5), course recommendation agent recommends a series of course materials to learners according to the ranking order of information function’s value.. I j (θ ) =. [e. (1.7) 2. 1.7 (θ −b j ( tuned )). ] [1 + e. ]. −1.7 (θ −b j ( tuned )) 2. (5). where θ stands for users’ new abilities estimated after n preceding course materials, P j (θ ) is the probability of a correct response to the j th course material for users with ability θ , b j (tuned ) is the tuned difficulty of the j th course material. In Equation (5), the value of information function I i (θ ) is ranged from 0 to 0.7225. Moreover, we will obtain the largest information function value if the bi (tuned ) is equal to θ .. 13.

(14) Interface Agent. User Profile Database. Feedback Agent. Courses Database. Rank courses by using maximal information function. Courses Recommendation Agent. Figure 4. Operation Flowchart of Courses Recommendation Agent The method of maximum information selection is derived from the item information curve], and it is function of examinee’ ability and item parameters (i.e., the difficulty of item, etc.). The “information” means that the difficulty of this item can provide maximal information to learners with certain ability. Therefore, the personalized mechanism of course recommendation agent gives a recommended list of course materials according to the ranking of information function values. Using information function, course recommendation agent will choose an item i with maximal information function value to learner under learner’s ability as θ .. 3. Exper iments 3.1 Exper imental Envir onment Our system prototype is implemented on the platform of Windows 2000 with IIS 5.0 Web server. The front-end script language is PHP 4.0 and the database is Microsoft SQL 2000 server. At present, our system only contains small amounts of course materials. For the course of neural network in 14.

(15) courses database, we have predefined 3 course units and collect 43 course materials so far. A course unit in our system indicates the collection of course materials with high relevance, such as the course unit of “Back-propagation” in the course of neural network. These course materials gathered from the Web were classified into some predefined course units in our system. Each course material has its corresponding difficulty initialized by experts. Moreover, each learner has different ability in different course units. Figure 5 shows an example of search results after learner gives a query. The title (標題) indicates the subject of the course material; the difficulty (難度指數) represents the difficulty of the course material; and the description (描述) gives an abstract of the corresponding course material. The length of bar line in the column of difficulty indicates the difficult degree of the corresponding course material. The longer bar line implies a more difficult course material. On the contrary, the shorter bar line implies an easier course material.. Figure 5. An example of response to learner’s query in a unit of a category 3.2 Exper imental Results and Analysis Next, we use “Perceptron” unit in “Neural Network” (NN) to interpret the experimental results. There are 20 course materials in “Perceptron”, 18 learners login our system, and 195 records in user profile database. All of learners are Master students and 13 of them have taken the course of neural 15.

(16) network. 3.2.1 The Difficulties’Adjustment of Cour se Mater ials In our system, the difficulties of course materials can be dynamically tuned based on our proposed collaborative voting approach after learners give feedback responses. We use three course materials, Course A, Course B, and Course C, to illustrate the difficulties’ adjustment of course materials. Course A, B, and C belong to difficult, middle, and easy course materials, respectively. We normalize the values of the difficulties and learners’ abilities within -1 to 1. Figure 6 shows the tuned curves of difficulties of three different course materials. The tuned range at the beginning is large because the initial difficulty of course material can not fit learners’ abilities. We can observe that the curves approach to stable status as the clicked times gradually increase, i.e., difficulties of course materials can be correctly determined by a large amount of learners’ voting. 3.2.2 The Adaptation of Lear ner s’Abilities Learners’ abilities can be dynamically evaluated according to learners’ responses after they visit the recommended course materials. Learners’ abilities will be promoted or descended based on the learners’ responses. If learners can understand the content of the recommended course materials, their abilities will be promoted. Otherwise, learners’ abilities will be descended. We select three learners’ with various learning abilities to illustrate the experimental results. Figure 7 shows the curves of the adaptation of three various learners’ abilities. We can observe that learners’ abilities are tuned drastically at the beginning. While learners learn the appropriate course materials during the learning process, learners’ abilities will gradually approach to stable status. Moreover, we can observe the relationship of the difficulties of the clicked course materials with the adaptation of the learner A’ ability in Figure 8. In this figure, assume learner’s responses “yes” to the question of “Can you understand the content of the recommended course material?” in twenty clicked course materials. 16.

(17) We find that if learners can understand the recommended course material with higher difficulty will contribute a higher tuned value of ability. On the contrary, if learners understand the course material with lower difficulty will contribute a lower tuned value of ability. Figure 9 shows the relationship of the learners’ abilities with the difficulties of the recommended course materials. We find that the difficulties of the recommended course materials are high relevance with the learners’ abilities. This result shows that our system indeed can recommend appropriate course materials to learners based on different learners’ abilities. The difficulties adjustment of course materials Course A. Course B. Course C. Difficulty 0.6 0.5 0.4 0.3 0.2 0.1 0 -0.1 -0.2 -0.3 -0.4 -0.5 -0.6 -0.7 -0.8 -0.9 -1. 1. 2. Course A 0.33 0.5 Course B. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 0.5 0.42 0.38 0.35 0.33 0.36 0.31 0.28 0.25. 0.3 -0.28 -0.03 -0.03 0.04 0.03 0.02 0.01 -0.03 -0.03 0.03 0.04. Course C -0.67 -0.5 -0.63 -0.67 -0.69 -0.6 -0.63 -0.57 -0.59 -0.58 -0.63 -0.61 -0.65 Clicked times. Figure 6. The tuned curves the difficulties of course materials. 17.

(18) The adapation of learners' abilities Learner A Ability. Learner B. Learner C. 1. 0.5. 19. 17. 15. 13. 11. 9. 7. 5. 3. 1. 0 -0.5 -1 The number of clicked course material. Figure 7. The adaptation of learners’ abilities Difficulty v.s. Ability Difficulty. Ability Ability 1. Difficulty1 0.5. 0.5 19. 17. 15. -0.5. 13. 11. 9. 7. 5. 3. 0 1. 0. -1. -0.5 -1. Number of of clicked course materials. Figure 8.The relationship between the difficulties of clicked course materials and the adjustment of the learner A’s ability Ability v.s. The difficulty of recommend course material Ability Ability. Recommend-Diffiuclty. 1. 1. 0.5. 0.5 0 1 3 5 7 9 11 13 15 17 19. 0 -0.5. Difficulty. -1. -0.5 -1. Number of clicked course materials. Figure 9. The relationship between learner A’s ability and the difficulties of recommended course materials 18.

(19) 3.3 Satisfaction Evaluation Next, we use two different methods to evaluate the learners’ satisfactory degree for our proposed system. The first method collects learners’ responses to analyze if the recommended course materials satisfy most of learners’ requirement. We evaluate the personalized mechanism of our system from two different views, i.e. learners’ and course materials’ viewpoints. The results are illustrated in Table 1. We collect the learners’ responses about the question of “Can you understand the curse material?”. From learners’ viewpoint, the average degree of understanding the recommended course material is 0.8. This result indicates that the learners’ comprehension is high for the recommended course material. From course materials’ viewpoint, the average degree of the recommended course material that can be comprehended by learners is 0.84. This result indicates that most recommended course materials can be comprehended by learners. Moreover, from two different viewpoints, we find that the average difficulty of the course materials recommended by our system respectively is 1.764 and 1.596, which is close to 2, i.e., most learners think the recommended course materials belonging to middle difficulty. Namely, our system can recommend appropriate course materials to learners. Therefore, we find that our system indeed can satisfy most of learners’ personalized requirements. The second method adopts the investigation of learners’ satisfaction according to the four designed questionnaires after learner finishes a learning process. In this work, we adopt 5-point Likert-scale [22] to evaluate our proposed system. Answers include 5-point scales, i.e. very satisfaction, satisfaction, neutral, not satisfaction, and not very satisfaction. According to learners’ responses, the evaluated results are summarized in Table 2. We find that most learners’ satisfactions are high for the personalized services in our system.. 19.

(20) Table 1. The user satisfaction about recommended course materials during learning process (a) Can you understand the content of course materials? (1:Yes, 0:No) The comprehension degree of course material recommended by system 0.8 0.84. Viewpoint Learners’ viewpoints Course materials’ viewpoints. (b) How do you think about the difficulty of the course material? (0:Very easy, 1: Easy, 2: Moderate, 3: Hard, 4: Very hard)) Viewpoint. The difficulty of votes the course material recommended by system. Learners’ viewpoints Course materials’ viewpoints. 1.764 1.596. Table 2. The user satisfaction after learning (a) Question (1) and (2) Answer. Users’ choices. No. of users (%) Question (1) How do you feel that our system provides appropriate courses on top of course materials list? (2) How do you feel that our system gives lower ranking order for inappropriate course materials?. Very suitable. Suitable. Moderate. Unsuitable. Very unsuitable. 3 (21%). 6 (43%). 5 (36%). 0 (0%). 0 (0%). 0 (0%). 7 (50%). 4 (29%). 3 (21%). 0 (0%). (b) Question (3) and (4) Users’ choices. Answer No. of users (%) Question (3) Do you satisfy the personalized services provided? (4) Do you satisfy learning process provided?. Very satisfaction. Satisfaction. Moderate. Not satisfaction. Not very satisfaction. 2 (14%). 8 (57%). 4 (29%). 0 (0%). 0 (0%). 0 (0%). 7 (50%). 7 (50%). 0 (0%). 0 (0%). 4. Conclusion In our study, we propose a personalized e-learning system based on Item Response Theory, termed as PEL-IRT, which can online estimate learners’ abilities to recommend appropriate course materials to learners. Moreover, the difficulties of course materials are adjusted by our proposed collaborative 20.

(21) voting method. It provides personalized Web-based learning according to the visited course materials of learners and learners’ responses. Experimental results show that our proposed system can immediately provide personalized course materials’ recommendation based on learner’s abilities and speed up learners’ learning efficiency. Moreover, the difficulties of course materials can be automatically determined by using our proposed collaborative voting approach. Furthermore, learner only needs to reply simple questionnaires for personalized services.. Refer ences [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11]. [12]. [13] [14]. Google, Available at http://www.google.com. CNN, Available at http://www.cnn.com. Amazon, Available at http://www.amazon.com. Lawrence, S.; Giles, C. L., “Searching the World Wide Web,” Science, Vol. 280, No. 5360, pp. 98100, 1998. Berghel, H., “Cyberspace 2000: Dealing with Information overload,” Communications of the ACM, Vol. 40, No. 2, pp.19-24, Feb. 1997. Arasu, A.; Cho, J.; Garcia-Molina, H.; Paepcke, A.; Raghavan, S., “Searching the Web,” ACM Transactions on Internet Technology, Vol. 1, Issue: 1, pp. 97-101, 2001. NEC Research Institute ResearchIndex, Available at http://citeseer.nj.nec.com. Mobasher, B.; Cooley, R.; Srivastava, J., “Automatic Personalization Based on Web Usage Mining,” Communications of the ACM, Vol. 43, No. 8, pp. 142-151, August 2000. The investigation of Yam, Available at http://survey.yam.com. International Data Corporation, Available at http://www.idc.com. Brusilovsky, P., “Adaptive and Intelligent Technologies for Web-based Education,” In Rollinger, C.; Peylo, C. (eds.), Special Issue on Intelligent Systems and Teleteaching, Künstliche Intelligenz, Vol. 4, pp. 19-25, 1999. Kao, T. T., Personalization Courseware Navigation Using Grey Ranking Analysis, Master’s thesis, Department of Electronic Engineering, National Taiwan University of Science and Technology, Taipei, Taiwan, 2001. Balabanovic, M.; Shoham Y., “Fab: Content-based, Collaborative Recommendation,” Communications of the ACM, Vol. 40, No. 3, pp. 66-72, March 1997. Fu, X.; Budzik, J.; Hammond, J. K., “Mining Navigation History for Recommendation,” Proceedings of the 2000 International Conference on Intelligent User Interfaces, pp. 106-112, 21.

(22) [15] [16] [17] [18] [19] [20] [21]. [22]. 2000. Baker, F. B., “The Basics of Item Response Theory,” ERIC Clearinghouse on Assessment and Evaluation, University of Maryland, College Park, MD. Avaialable at: http://ericae.net/irt/baker. Hambleton, R. K., Item Response Theory: Principles and Applications, Boston, Kluwer-Nijhoff Publishe, 1985. TOEFL, Avaliable at http://www.toefl.org. GRE, Avaiable at http://www.gre.org. GMAT, Available at http://www.gmat.org. Hambleton, R. K.; Swaminathan, H.; Rogers, H. J., Fundamentals of Item Response Theory, Newbury Park: Sage, 1991. Lin, Y. T.; Tseng, S. S.; Jiang, M. F., “Voting Based Collaborative Platform for Internet Content Selection,” The 4th Global Chinese Conference on Computers in Education Proceeding, Singapore, 2000. Likert, R., A Technique for the Measurement of Attitudes, New York: Archives of Psychology, 1932.. 22.

(23)