Supporting Web-Based Instructional Design with Fuzzy Approaches for Assessing Attitude Surveys

全文

(1)Supporting Web-Based Instructional Design with Fuzzy Approaches for Assessing Attitude Surveys Cheng-Chang Jeng Department of Elementary Education, National Taitung University cjeng@cc.nttu.edu.tw. Abstract During the development process of an instructional design, attitude surveys are usually obtained from different groups of evaluators. Since the number of evaluators is small, it is difficult to obtain any significant result by using conventional statistical methods. This study utilizes fuzzy theory to overcome the difficulties caused by the small number of samples. In order to fuzzify the evaluators’ response on a certain question, computerized questionnaire design with continuous scales is introduced. Li and Yen’s method and Sugeno’s measure/integral are applied to investigate the overall quality of an instructional design. To verify the degree of agreement between different groups of evaluators, a method used to measure the degree of agreement is proposed. An example is given to demonstrate the actual computational process of an attitude survey by using the approach proposed in this paper. Keywords: Fuzzy Measure, Fuzzy Integral, Web-Based Attitude Survey, Likert Scale, Continuous Scale.. 1.. Introduction. In an instructional design, an evaluation acts as a test of validity to verify final effectiveness of the instructional activity or material, i.e., a summative evaluation; or serves as a checkpoint for further revision, i.e., a formative evaluation. In a summative evaluation of the instructional design, the learning performance of target students is usually measured by the difference between the pretest and the posttest score. The instructional design must be in its final form and the learners should be representative of the target students [1]. A formative evaluation is usually performed by instructors/designers, since these people are those who determine the necessary revisions. In a formative evaluation, students’ opinions and performance are also important. An. instructional designer usually obtains a small number of attitude data from both students and instructors to carry out a qualitative analysis on his/her prepared instructional activity or material. To assess the outcome of an evaluation, an evaluator may conduct a multi-factor tool to verify different categories of incidents or opinions, and each category consists of several homogeneous items or questions. A commonly used format for designing an item or a question is a 5-level or 3-level Likert scale, which is used to ask evaluators to indicate their opinions, for example, from “Strongly Agree” to “Strongly Disagree”. However, in most of the formative evaluations, the designer has very few data to be analyzed, because the data are normally collected from three to five evaluators. Therefore, a designer must use his/her own decision to determine what changes should be made in the prepared instructional activity or material [2]. On the other hand, since the evaluation information is obtained from different groups of evaluators, learners and specialists, the degree of agreement among the groups is important for further improvement of the prepared instructional activity or material. Although Jonassen [3] has suggested that fuzzy theory should be applied to the instructional design in the field of Instructional Technology (IT), there is still no existing literature that demonstrates exactly how to apply the fuzzy theory in IT. Since a formative evaluation only requires a small number of evaluators to evaluate intermediate products, it is difficult to obtain any result, which is of statistical significance, and to make any decision, which is based on any conventional statistical method. This paper focuses on formative evaluations to discover how to use fuzzy approaches to obtaining a meaningful conclusion from a small number of evaluators, although the proposed approach can also be applied to summative evaluations. Instead of using the statistical or intuitive knowledge to summarize the information obtained from the questionnaire in a formative evaluation, a fuzzy approach is proposed by Li and Yen [4] to measure the quality of the instructional activity or material. Li and Yen’s measure does not consider the degree of agreement, if.

(2) attitude surveys are obtained from different group of evaluators. Therefore, the measure of the degree of agreement is developed in this paper to compare the inclination among different groups of evaluators.. tween conventional Likert scale and continuous scale is introduced, and the technique for the fuzzification of continuous scale is also developed. 3.1. 2.. Introduction to Fuzzy Theory. While Aristotelian two-valued logic describes the membership of an element to a set, the possible values for the possession of the element could be either true or false. To illustrate the above situation, for a given universal set X, a membership function or characteristic function can be defined that declares which elements of X are members of a given set A or not. Mathematically, a membership function is defined,. The Conventional Likert Scale v.s. the Continuous Scale. Kochen’s experiments [6] show that the use of a continuous scale can represent the strength of belief while a evaluator is asked to make a judgment for a given statement, and may increase the degree of precision. Figure 1 demonstrates an example of the forms of a continuous scale and a conventional 5-level Likert scale commonly used in a questionnaire to measure the belief of a given question.. 1 x ∈ A X A (x ) =  x∉A 0 Which is formally expressed by X A : X → {0,1 }. This expression illustrates that for each x ∈ X, when X A (x) = 1, x is declared to be a member of A; when X A (x) = 0, x is declared as not being a member of A. However, Zadeh [5] gives an important evolution of the modern development of fuzzy theory. The theory gives another point of view to describe the membership of a given element. If A is a fuzzy set and x is a relevant element, the ownership for the element x does not need to be either true or false, but it may be true in some degree, the degree to which x belongs to A. As defined in the previous discussion for a membership function of a crisp set, the membership function of a fuzzy set A is denoted by. µ A : X → [0,1] In this case, each membership function maps elements of a given universal set X, which is always a crisp set, into real numbers in [0,1]. The concept of fuzzy sets can be used to describe the nature of vagueness. For example, the verbal words, “little”, “moderately” and “very”, illustrate some degree of vagueness. There are no exact boundaries among the words. Therefore, it is reasonable to apply fuzzy theory on the design of a questionnaire. To demonstrate the use of fuzzy theory, the design of a continuous scale, which is different from the conventional Likert scale, is proposed in the following section.. 3.. Likert Scale and Continuous Scale Design In this section, the comparisons be-. Figure 1: A continuous scale and a 5-level Likert scale 3.2. Fuzzification of the Continuous Scale. Let the left end be 0 and the right end denoted as L, where L is the length of the continuous scale. For any given mark x, a membership function can easily be defined to represent the degree of belief:. µ A (x ) =. x (1) L Therefore, µ A ∈ [0,1]. Eq. (1) determines the fuzziness of a evaluator’s response. Unlike the conventional 5-level Likert scale, which only represents 5 discrete responses, the continuous scale reflects some kind of vagueness in response. For example, if a evaluator thinks that he/she would like to give a personal judgment for a given question between level 3 and level 4, i.e., “a little bit agree with” but not “agree or strongly agree with”, the continuous scale can help a questionnaire designer measure such a kind of ambiguous judgment.. 4.. Multi-Factorial Measure and Determination of Overall Quality. The questions in an attitude survey may consist of several factors in order to retrieve subjects’ opinions from complex problem domain. The skills for determining overall quality of subjects’ opinions and measuring factors of questions are introduced in this section..

(3) 4.1. Distribution of Quality Measures. n. ∑ :X → ∑ X = ∑ a i a i * , where ai ∈ [0,1], and i =1. The design of a questionnaire that is used to measure the quality of the instructional activity or material may be formatted with several criterion or factors such as attention (interest), relevance, confidence, clarity, and satisfaction [2]. To study the outcome of a questionnaire is a multi-group and multi-factorial problem. Li and Yen [4] propose a fuzzy approach to measuring multi-factorial estimation of quality. The method is good for our use in the situation described above. The overall procedure for multi-factorial estimation of quality is as follows. Let F={f1, f2, …, fn} be a set of factors used for the measure of the quality of an instructional activity or material. Each of the factors could have several questions. Let P={p1, p2, …, pr} be a group of evaluators. For factor fi (i = 1, 2, …, n), evaluator pj (j = 1, 2, …, r) independently assigns one mark for each of the questions on the continuous scales. The range of these marks is denoted by [aij, bij], where aij and bij denote the minimum and the maximum of the jth person’s evaluation on the ith factor, respectively. Note that aij and bij ∈ µ A (X ) . Thus, for each factor i (i = 1, 2, …n), the quality measure. a *i obtained from all the r evaluators on the ith factor can be defined as r r  a ij + bij   ∑ (bij − a ij) ∑ (bij2 − a ij2 ) j=1  2  = j=1 a i* = r r 2 ∑ (bij − a ij) ∑ bij − a ij j=1. where. (. ). aij = ∧ µ A (X) and p jf i. (2). j=1. bij = p∨f µ A (X). , in. j i. which the operation ∧ represents the minimum function min( ), and the operation ∨ represents the maximum function max( ). * Let A = a1* , a * 2 ,..., a n . Then A is a distribution of quality measures on the n factors. In addition, A is also a fuzzy vector. Each component a*i in A reflects a quality estimation for the ith factor in a questionnaire. If a component. {. }. a*i obtains a higher value than the other components, it is concluded that the ith factor of the instructional activity or material is well designed such that the evaluators give the factor higher rating. Eq. (2) offers a tool to measure the quality of each factor. If the overall quality is considered, a simple maximum operator can be used.. n. ∑ a i =1. (3). i =1. where ai represents the weight or the importance of the ith factor. To transform the value obtained from the above quality evaluation equation into verbal variables like “Bad”, “Average” and “Good”, Fig. 2 depicts the transformation graphically.. Figure 2. Transform Overall Quality Measure into Verbal Variables 4.2. Measure of Overall Quality Using Sugeno’s Method. Although eq. (3) with the maximum operator provides a simple computational model to estimate the overall quality, it only simply discards the small values and does not consider the cross effects of factors. The mathematical tool proposed by Sugeno [7] is applied in this paper to measure the overall quality with a consideration of the effects of other factors in a more complex manner. Sugeno defines a fuzzy measure g over a finite set X (a universe of discourse with the subsets E, F, …), which satisfies the following conditions [7]: g(∅ ) = 0, g(X) = 1 If E⊂ F, then g(E)≤ g(F). (4) (5). A fuzzy measure is a Sugeno’s measure (or a λ-fuzzy measure), if it satisfies the following additional condition for some λ > -1 and λ≠ 0. If E ∩ F = ∅ , then gλ(E ∪ gλ(F) + λgλ(E) gλ(F). F) = gλ(E) + (6). The value of λ can be calculated regarding to the condition g(X)=1, n. λ = ∏ (1 + λg ( f i ) ) − 1 i =1. (7).

(4) In an application of any attitude survey, the set X can be defined as the set of factors, for example, X={f1, f2, f3, …, fn}, where fi denotes the ith evaluation factor in the questionnaire design. Furthermore, the fuzzy density g(fi) represents the degree of preference (i.e., the weight) of the ith factor. g(fi) is a fuzzy membership function. Then, the Sugeno’s measure gλ(E∪ F) represents the degree of importance of possible factor combination. Note that (E∪ F)∈ power(X). Then, a fuzzy integral can be defined as the following:. ∫ h (f ′). g = ∨ [h (f i′) ∧ g (H i )] n. o. i =1. H i = {f i′, f i′+1 , ..., f n′ } , 1≤ i ≤n. with (8). where f ′ ∈ X and h(f’) denotes the confidence value delivered by evaluation factors f’. h(f’) is arranged in an ascendant order, i.e., h (f 1′ ) ≤ h (f 2′ ) ≤ ... ≤ h (f n′ ) . For example, if n = 3 and h(f’) is arranged in an ascendant order, H 1 = {f 1′, f 2′ , f 3′ } , H 2 = {f 2′ , f 3′ } , and H 3 = {f 3′} . A computational example for Hi is shown in Section 6. How does a fuzzy integral be used in the evaluation of an attitude survey? h (f i′) are the components in a fuzzy vector calculated by using eq. (2). In other words, h (f i′) are the ascendantly ordered a* . Thus, a fuzzy integral is a. G AB =. A•B A B. (9). where A•B denotes the inner product of A and B, and A and B denote the norm of the two fuzzy vectors, respectively. Therefore, GAB ∈ [0,1] is the fuzzy measure of the degree of agreement between the two different groups of evaluators. Actually, according to the above discussion, GAB is a membership function, and its independent variables are two fuzzy vectors A and B. To assess the degree of agreement, Fig. 3 gives a sample to help a designer determine the degree of agreement. For example, if GAB is 0.28, the value of “Very Consistent” is 0.13, the value of “Undecided” is 0.26, and the value of “Contradictory” is 0.87. The maximal value 0.87 is chosen, and this value’s correspondent line “Contradictory” is used to address the degree of agreement estimation of GAB . In other words, while GAB is 0.28, the measure of the degree of agreement of the fuzzy vectors A and B is “contradictory”.. i. measure of the overall quality of an instructional activity or material regarding to the degree of preference of possible factor combination. The Sugeno’s measure indicates the preference of factors. If a questionnaire designer considers that some factors should be weighted more than the others, then the weights represent the preference of factors and are defined in Sugeno’s measure for calculating fuzzy integral.. 5.. Measure of the Degree of Agreement. Consider the situation of the evaluation of an instructional activity or material. Suppose that B= b1* , b*2 ,..., b*n is obtained from another group of evaluators. Then, B is also a fuzzy vector as well as a distribution of quality measures. For example, A is obtained from a group of specialists and B is obtained from a group of target students. Since Li and Yen’s measure cannot determine the degree of agreement between vectors A and B, an enhancement of Li and Yen’s measure is proposed in this section. The degree of agreement between the two fuzzy vectors A and B is calculated by defining the degree of agreement GAB as. {. }. Figure 3. Degree of agreement. 6.. An Example. The Division of Continuing Education (DCE) in Northern Illinois University offers online courses to students. To revise the products, a survey of students’ opinions about primitive prototypes is needed. There are only a small number of students and instructors, who live in different geographic areas, participate in the experimental online course. It is difficult to offer them conventional pencil-and-paper questionnaire. Therefore, an online questionnaire is developed. Table 1 demonstrates a sample online questionnaire. As shown in Table 1, each question is not evaluated by a discrete Likert scale. A scroll bar is designed to emulate a continuous scale. Online users are able to use their web browser to answer the questions. Each scroll bar can represent integer numbers from 0 to 32767, from left to right. Therefore, in eq. (1), the variable L is assigned to the maximum value 32767, and the variable X is assigned by the position of each.

(5) slider, the small movable square. There are three factors in this sample online questionnaire, i.e., “Attention”, “Relevance” and “Clarity”. After a user moves a slider to give his/her opinion about the questions, a fuzzy value of the answer for each question is immediately obtained by computing eq.(1).. transform each raw integer number measured to a fuzzy number as shown in Table 4 and 5. Table 4. Data in Table 2 transformed to fuzzy numbers. Table 1. A sample online questionnaire designed in HTML format I. Attention: To what degree did the following instructional activities hold your interest or attention? Question Attention Level Little Very Attentive 1. Online discussion board aids cooperative interaction 2. Instructor is used to encourage group cooperation to solve problem remotely. 3. Student may obtain online help to solve their questions in real time. II. Relevance: To what degree do you believe the following skills are relevant to help you get used to enroll in a future online course? Question Relevance Level Little Very Relevant 4. Recognizing how to cooperate with other online classmates. 5. Participating in an online group discussion 6. Taking online exams III. Clarity: What level of clarity do you believe the following instructional materials and activities have? Question Clarity Level Little Very Clear 7. Session Introduction 8. Objectives for session 9. Online video 10. Online recorded voice. During the formative evaluation, there are three students and two instructors who participated in the online survey. In this case, they form two groups of evaluators, students and instructors. Table 2 and Table 3 demonstrate the results obtained from the two groups of evaluators, respectively. Table 2. Survey results obtained from students. Table 3. Survey results obtained from instructors. To analyze the data, eq. (1) is used to. Table 5. Data in Table 3 transformed to fuzzy numbers. The fuzzy vector of the instructors’ group is calculated by using eq. (2): = 1 *. b. b b. * 2. * 3. (0.952 − 0.852 ) + (0.92 2 − 0.82 2 ) = 0.9 2 × [(0.95 − 0.85) + (0.92 − 0.82)]. = 0 .75 = 0.71. Thus, B=(0.9, 0.75, 0.71). Similarly, eq. (2) is used to find the fuzzy vector for the students’ group. The following results are obtained. * a1 = 0.82. a a. * 2 *. = 0.76. = 0.69 3 A=(0.82, 0.76, 0.69). Therefore, the degree of agreement of opinions between the two groups is formed by computing eq. (9): G AB =. A • B 1 . 7979 = = 0 . 999 A B 1 . 7997. Suppose that “very consistent”, “undecided” and “contradictory” are defined as in Figure 3. It is seen that the two groups of evaluators have a very strong degree of agreement for the quality of the experimental online course. However, this calculation does not give the account of whether the consistence of opinions is overall “positive” or “negative”. Suppose that eq.(3) is used with ai=1/3, i.e., the preference of each factor is regarded to be equal to each other. From student survey vector A, applying eq. (3), the overall quality measure is (1/3)*0.82+(1/3)*0.76+(1/3)*0.69=0.76. From Fig.2, it is concluded that the quality of the instructional activity or material is about “good”. Similarly, from instructor survey vector B, the value of the overall quality is obtained to be 0.79. To calculate Sugeno’s measure, X is defined as X={Attention, Relevance, Clarity}..

(6) Suppose 0.3 if f 1 = Attention  g (f i ) = 0.4 if f 2 = Relevance 0.9 if f = Clarity 3 . Since g(fi) represents the degree of preference of factors, the effect of the factor “Clarity” is more preferred than the other two in this example. λ = (1+0.3λ)(1+0.4λ)(1+0.9λ)-1 is obtained by applying eq. (7). The solutions are λ = {-6.022, -0.926, 0}. Since the conditions λ>-1 and λ ≠ 0 must be satisfied, λ=-0.926 is the only solution. The Sugeno’s measure can then be constructed as follows: Table 6. Computation of Sugeno’s Measure. Furthermore, h(Attention)=0.82, h(Relevance)=0.76, and h(Clarity)=0.69 are equivalent to the components a* in the fuzzy i. vector A, which is previously obtained from the student attitude survey. To compute the fuzzy integral, the h (f i′) are rearranged in an ascen-. dant order h (f 1′ ) ≤ h (f 2′ ) ≤ ... ≤ h (f n′ ) , f i′ ∈ X, 1≤ i≤ n, i.e., h(Clarity) ≤ h(Relevance) ≤ h(Attention). Therefore f 1′ = Clarity, f 2′ = Relevance, and. f 3′ = Attention. Note that. H 1 = {f 1′, f 2′ , f 3′ } = {Clarity, Relevance, Attention} = X H 2 = {f 2′ , f 3′ } = {Relevance, Attention}. H 3 = {f 3′} = {Attention}. Finally, from eq. (8), the fuzzy integral is obtained by computing the following:. ∫ h(f ′). g = ∨ [h (f i′) ∧ g (H i )] n. o. i =1. = ∨ [h(f1′) ∧ g({f1′, f 2′ , f3′}), h(f 2′ ) ∧ g({f 2′ , f3′}), h(f3′ ) ∧ g({f3′})] =max[min(0.69, 1), min(0.76, 0.59), min(0.82, 0.3)] =max[0.69,0.59,0.3] =0.69. It is concluded that, from the students’ point of view, the instructional activity or material is about “good” by using Fig. 2.. 7.. Conclusions. Instead of using discrete Likert scales to. measure the response to a survey, Kochen [6] suggests that the continuous scale reflects a response better than the Likert scale. It is difficult to measure the length of a response on a continuous scale, if the questionnaire is in a conventional paper-and-pencil style. It is necessary to use a ruler to obtain the length of every mark on a continuous scale in order to fuzzify an evaluator’s response. In this paper, a computerized online questionnaire is proposed and fuzzification can be completed automatically, i.e., a membership function can be defined as the ratio of the segments in the continuous scale. Since a questionnaire is usually assigned to people who have different background to investigate the effects or opinions for a prepared instructional design, a method to measure the degree of agreement among different groups is proposed in this paper. If there are m groups of evaluators, C m2 = m × ( m − 1) times of calcula2 tions are performed to compare the degree of agreement among groups. The combination of Li and Yen’s and Sugeno’s methods with the degree of agreement measure proposed in this paper helps to investigate the quality measure of an instructional activity or material. The aforementioned combined methods conquer the difficulty of applying conventional statistical or intuitive observation methods to investigate the quality of an instructional activity or material with a small number of evaluators. During the development process of an instructional design, what is concerned about is the quality of the instructional activity or material and the degree of agreement among the groups of evaluators. Consequently, applying the proposed fuzzy measure is appropriate to reflect the vagueness of the evaluation judged by human beings. Since the evaluation of human attitudes is complicated with many factors, an instructional designer should be very careful to interpret the evaluation data. In the simplified example in Section 6, there are only three factors to be surveyed. A real-life attitude survey may have more than 5 factors, and each factor may consist of more than 3 questions, i.e., more than 15 questions in an attitude survey. Moreover, a complete formative evaluation usually needs to test the students’ performance after a prototype of the instructional material has been tried out. Hence, a complete fuzzy quality measure should consider at least three results: the students’ attitude survey, the instructors’ (professional) attitude survey and the student test score. The student test score is not a homogeneous data from the other two. Most instructional designers use their “professional experience” to analyze these three different types of data. Thus, future re-.

(7) search is suggested to include the pretest, the posttest, the student’s attitude and the instructors’ attitude surveys all together to establish a complete fuzzy evaluation system.. References [1] J.D. Russell and B.L. Blake, “Formative and Summative Evaluation of Instructional Products and Learners,” Educational Technology, vol.28, no.9, pp22-28, 1988 [2] W. Dick and L. Carey, The Systematic Design of Instruction, 4th ed. NY: HarperCollins, 1996 [3] D. H. Jonnassen, et al., “Certainty, Determinism and Predictability in Theories of Instructional Design,” Educational Technology, vol.37, no.1, pp27-34, 1997 [4] H. X. Li and V. C. Yen, Fuzzy Sets and fuzzy Decision-Making, NW: CRC Press, 1995 [5] L. A. Zadeh, “Fuzzy Sets,” Information and Control, vol.8, no.3, pp338-353, 1965 [6] L. A. Zadeh, K. S. Fu, K. Tanaka and M. Shimura, Fuzzy Sets and Their Applications to Cognitive and Decision Processes, NY: Academic Press, 1975 [7] T. Terano and M. Sugeno, “Conditional Fuzzy Measures and Their Applications,” in Fuzzy Sets and their Applications to Cognitive and Decision Processes, (L. A. Zadeh, K. S. Fu, K. Tanaka, and M. Shimura, Eds.), pp151-170, NY: Academic Press, 1975.

(8)