Chapter 4 Applications of MMDE Algorithm in Technology Management
4.3 Evaluation of the E-learning Program
4.3.3 Results and Analysis
Followed the stages we described in section 4.2.2, we derived the results as follow:
Result of Stage 1
Questionnaire reliability analysis was analyzed following responses received.
According to reliability analysis results, Cronbach’s α value is higher than 0.8 and the standardized element α value is 0.977 showing questionnaire reliability to be significant and effective (Reliability analysis results shown in Table 4-2).
Table 4-2. Reliability analysis results
Source of variance Sum of sq. d.f. Mean square F-test Probability
Between people 4541.418 65 69.868
Within people 6210.810 376 1.651
Between measures 308.001 57 5.404
Residual 5902.809 371 1.593
Total 10752.229 383 2.81
Grand mean 6.973
Alpha 0.977 Standardized element alpha = 0.978
3.392 0.000
KMO and Bartlett's test was used to measure the appropriate usage of factor analysis.
According to Kaiser's research, KMO > 0.7 is middling to do factor analysis, and KMO
>0.8 is meritorious. The KMO value of this research is 0.737 (Bartlett's Test of Sphericity:
Approx. Chi-Square=4740, d.f.=1653, Significance=0.000); therefore, it is suitable for factor analysis. This method uses a correlation coefficient to test whether it is suitable and significant to use factor analysis. According to the results of KMO and Bartlett's test, this questionnaire is suitable to use factor analysis.
The principle component analysis was used to extract factors from fifty-eight criteria and the varimax method was used for factor rotation. Then, nine factors whose eigenvalue was more than 1.0 were chosen. Nine factors were named based on the loading of each factor: “Personal Characteristics and System Instruction,” “Participant Motivation and System Interaction,” “Range of Instruction Materials and Accuracy,” “Webpage Design and Display Of Instruction Materials,” “E-Learning Environment,” “Webpage Connection,” “Course Quality and Work Influence,” “Learning Records” and
“Instruction Materials” (Shown in Table 4-3).
Result of Stage 2
According to factor analysis results, some experts and professionals were invited to discuss and scored the relation between criteria of each factor based on the DEMATEL approach. Experts and professionals included system designers, webpage designers, instructors, managers, and human resources experts. Factor 1 and factor 2 managers and
human resources experts. Factor 4 was discussed with webpage designers. Factor 5 and factor 6 were discussed with system designers. Instructors were responsible to factor 3, factor 7, factor 8, and factor 9.
Table 4-3. Factor analysis result: names and components (criteria) of factors
Factor Components λa Αb Βc
1 Personal Characteristics and System Instruction
Personal Motivation, Rewards, Work Attitude, Learning Expectation, Work Characteristics, Self-Efficacy, Ability, Career Planning, Organization Culture, Instruction Goals, System Functions, System Instructions
25.98 44.8 44.8
2 Participant Motivation and System Interaction
Operating Skills, Solving Solutions, Mastery, Managerial Skills, Professional Skills, Inspire Originality,
Supervisor’s Support, Colleagues, Work Environment, Causes of Problem, Understanding Problems, Pre-Course Evaluation, Multi-Instruction, Communication Ways
4.926 8.494 53.3
3 Range of Instruction Materials and
Accuracy
Accuracy, Range of Instruction Materials, Sequence of
Instruction Materials, Usage of Multimedia 3.945 6.802 60.1 4 Webpage Design and
Display of Instruction Materials
Text & Title, Display of WebPages, Sentence Expression, Length of WebPages, Graphs and Tables, Colors of WebPages
2.533 4.368 64.5 5 E-Learning
Environment
Browser Compatibility, Browsing Tool, Path of WebPages, Transferring Time, Available, Reflection of Opinions
1.956 3.372 67.83 6 Webpage Connection Under-constructing WebPages, System Prompts,
Connecting to Main Page, Connection of WebPages 1.846 3.183 71.02 7 Course Quality and
Work Influence
Course Arrangement, Course Design, Personal Satisfaction, Technical Evaluation, Course Contents, ROI/Work Influence
1.667 2.874 73.9 8 Learning Records Learning Records, Instruction Activities, Course Subject 1.505 2.596 76.5 9 Instruction Materials Level of Instructional Materials, Update Frequency,
Readable 1.282 2.21 78.7
Extraction Method: Principal Component Analysis.
Rotation Method: Varimax with Kaiser Normalization.
a Eigenvalue.
b % of Variance.
c Cumulative %.
Thus, after experts and professionals scored the relation of criteria, the Full Direct/Indirect Influence Matrix and the impact-digraph-map of each factor was calculated and drawn. According to the results of DEMATEL, the threshold value of each factor was decided by the experts. The threshold value of each factor from factor 1 to factor 9 is 0.85, 0.47, 1.5, 2.1, 1.6, 6.5, 2.1, 3.8 and 3.5. The impact-digraph-maps of DEMATEL method results were obtained and shown as Figure 4-9.
Result of Stage 3
According to Figure 4-9, the intertwined structures of several criteria, affected by
other criteria, were illustrated. Therefore, the fuzzy measure for the final affected elements of each factor could be calculated out. Using factor 1 as an example, the criteria,
“Rewards” and “Learning Expectations,” are two final affected elements affected by other criteria, but they did not influence other criteria.
Figure 4-9. The impact-digraph-maps of nine factors derived by DEMATEL method
“Rewards” was affected by “Personal Motivation,” “Self-Efficacy,” “Career Planning,”
and “Ability;” “Learning Expectations” was affected by “Career Planning,” “Ability,”
and “Self-Efficacy.” Since these criteria have an influential relationship, the fuzzy
measure should be employed to evaluate the weights of “Rewards” and “Expectations.”
The λ value was set as 1 and -0.99, indicating different synthetic effects of criteria. Fuzzy measure results of final affected elements of factor 1 are listed in Table 4-4. The e-learning satisfaction survey could then be implemented to calculate the fuzzy integral value of each factor. For example, the satisfaction value of the criteria, “Personal Motivation,” “Self-Efficacy,” “Ability,” and “Career Planning” in program 2 are 3.597, 3.792, 3.719 and 3.370, and the integral value of “Rewards” at λ=1 is 3.589. The fuzzy integral values of the final affected elements are shown in Table 4-5. These results could be implemented to calculate final results of each program.
Result of Stage 4
Weights of nine factors and the reduced criteria were calculated out and used to find the effectiveness of each program. The final score for each program is shown in Table 4-6.
Table 4-4. Fuzzy measure for two final affected elements of factor 1
Factor Element λ Fuzzy measure
1 g1-1=0.192, g1-6=0.190, g1-7=0.190, g1-8=0.189
g(1-1,1-6)=0.416, g(1-1, 1-7)=0.416, g(1-1,1-8)=0.417, g(1-6,1-7)=0.411, g(1-6,1-8)=0.412, g(1-7,1-8)=0.412,
g(1-1, 1-6, 1-7)=0.683, g(1-1,1-7, 1-8)=0.683, g(1-1,1-6,1-8)=0.683, g(1-6,1-7,1-8)=0.678 g(1-1,1-6, 1-7, 1-8)=1
Rewards
-0.99 g1-1=0.696, g1-6=0.689, g1-7=0.689, g1-8=0.690
g(1-1,1-6)=0.910, g(1-1, 1-7)=0.910, g(1-1,1-8)=0.910, g(1-6,1-7)=0.908, g(1-6,1-8)=0.910, g(1-7,1-8)=0.908,
g(1-1, 1-6, 1-7)=0.978, g(1-1,1-7, 1-8)=0.978, g(1-1,1-6,1-8)=0.978, g(1-6,1-7,1-8)=0.978 g(1-1,1-6, 1-7, 1-8)=1
1 g1-6= 0.260, g1-7=0.260, g1-8=0.260
g(1-6,1-7)=0.587, g(1-6,1-8)=0.588, g(1-7,1-8)=0.588, g(1-6,1-7,1-8)=1
1
Learning Expectations
-0.99 g1-6= 0.792, g1-7= 0.792, g1-8=0.793
g(1-6,1-7)=0.963, g(1-6,1-8)=0.963, g(1-7,1-8)=0.963, g(1-6,1-7,1-8)=1
Elements: 1-1: “Personal Motivation”; 1-6: “Self-Efficacy”; 1-7: “Ability”; 1-8: “Career Planning”
Table 4-5. Fuzzy integral results of each element in different programs
Integral Value Factor Elements of Factor λ Value
Program 1 Program 2
Directive Impact Elements Indirective Impact Elements
1 2.475 3.589
Rewards -0.99 2.552 3.753 Self-Efficacy, Ability, Career Planning, Personal Motivation
1 2.447 3.593 1
Learning Expectations -0.99 2.476 3.764
Self-Efficacy, Ability, Career Planning 1 2.529 3.641
Managerial Skills -0.99 2.548 3.693 Understanding Problems, Operating
Skills Work Environment, Colleagues,
1 2.507 3.609
Professional Skills -0.99 2.623 3.761
Work Environment, Understanding Problems, Solving Solutions
Colleagues, Operating Skills, 2
Mastery* 2.585 3.684 Understanding Problems Work Environment, Colleagues
1 2.671 3.626
Accuracy -0.99 2.763 3.682 Sequence of Instruction Materials, Range of Instruction Materials 1 2.641 3.604
Range of Instruction
Materials -0.99 2.696 3.678 Sequence of Instruction Materials,
Accuracy 3
Usage of Multimediaa 2.484 3.745 Sequence of Instruction Materials 1 2.537 3.697
Display of WebPages -0.99 2.645 3.740
Text &Title, Graphs and Tables, Colors of WebPages
Length of WebPages,
1 2.471 3.688 Colors of WebPages
Graphs and Tables -0.99 2.577 3.739 Text &Title, Length of WebPages, Display of WebPages
1 2.508 3.736 4
Colors of WebPages -0.99 2.601 3.745
Text &Title, Display of WebPages Length of WebPages, Graphs and Tables
1 2.360 3.602 Available
5 Transferring Time -0.99 2.413 3.643 Browser Compatibility, Browsing Tool, Path of WebPages
Course Contents -0.99 2.676 3.771
Technical Evaluation, ROI/Work Influence, Personal Satisfaction, Course Design, Course Arrangement
8 Learning Recordsa 2.318 3.658 Instruction Activities Course Subject
1 2.520 3.720
9 Update Frequency -0.99 2.546 3.741 Level of Instructional Materials,
Readable
a Without synthetic effect, the element did not use the fuzzy measure and fuzzy integral for evaluation.
Table 4-6. Final score of each program
(Criterion) Elements of Factor
Program 1 Program 2 Program 1 Program 2
0.249 Rewards 2.552 3.753 2.475 3.589
0.249 Learning Expectations 2.476 3.764 2.447 3.593
0.086 Work Attitudea 2.438 3.729 2.438 3.729
0.082 Work Characteristicsa 2.517 3.666 2.517 3.666 0.084 Organization Culturea 2.451 3.537 2.451 3.537 0.085 Instruction Goalsa 2.186 3.703 2.186 3.703 0.086 System Functionsa 2.362 3.640 2.362 3.640 1
0.105
0.082 System Instructionsa 2.258 3.615 2.258 3.615 0.183 Managerial Skills 2.548 3.693 2.529 3.641 0.183 Professional Skills 2.623 3.761 2.507 3.609
0.180 Mastery b 2.585 3.684 2.585 3.684
0.077 Inspire Originalitya 2.281 3.518 2.281 3.518 0.077 Supervisor’s Supporta 2.578 3.799 2.578 3.799 0.078 Causes of Problema 2.475 3.597 2.475 3.597 0.073 Pre-Course Evaluationa 2.498 3.495 2.498 3.495 0.074 Multi-Instructiona 2.592 3.729 2.592 3.729 2
0.115
0.074 Communication Waysa 2.438 3.684 2.438 3.684
0.378 Accuracy 2.763 3.682 2.671 3.626
0.378 Range of Instruction Materials
0.167 Sentence Expressiona 2.601 3.719 2.601 3.719 0.835 Transferring Time 2.413 3.643 2.360 3.602 5 0.114 0.165 Reflection of Opinionsa 2.331 3.631 2.331 3.631
0.679 Connect To Main Page 2.498 3.620 2.498 3.608 6
0.111 0.321 Under-constructing Web Pagesa
a The criteria whose influence level did not reach the threshold value were considered independent criteria.
b Without synthetic effect, the element did not use the fuzzy measure and fuzzy integral for evaluation.