Chapter 2 Literature Review
2.4 E-Learning Programs Evaluation
2.4.1 Introduction
Internet has significantly impacted the establishment of Internet-based education, or e-learning. Internet technology evolution and e-business has affected all industrial and commercial activity and accelerated e-learning industry growth. It has also fostered the collaboration of education and Internet technology by increasing the volume and speed of information transfer and simplifying knowledge management and exchange tasks. E-learning could become an alternative way to deliver on-the-job training for many companies, saving money, employee transportation time, and other expenditures.
An e-learning platform is an emerging tool for corporate training, with many companies developing their own e-learning courses for employee on-the-job training. Employees can acquire competences and problem solving abilities via Internet learning for benefits among business enterprises, employees, and societies while at work.
Although e-learning has been developing for several years, evaluating e-learning effectiveness is critical as to whether companies will adopt e-learning systems. A considerable number of studies have been conducted emphasizing the factors to be considered for effectiveness evaluation. Several evaluation models are considered with specific aspects. The criteria used for e-learning effectiveness evaluation are numerous and influence one another.
The evaluation models however, are deficient and do not have an evaluation guideline. Effectiveness evaluation criteria must integrate learning theories, relative web site design, course design, and learning satisfaction theories to form an integrated evaluation model [6, 85-87]. Since e-learning can be evaluated according to different aspects and criteria, the MCDM approach is suitable for e-learning evaluation.
E-learning combines education functions into electronic form and provides instruction courses via information technology and Internet in e-Era. The most popular definition of e-learning as defined by the American Society for Training and Development (ASTD) is a wide set of applications and processes, such as Web-based learning, computer-based learning, virtual classrooms, and digital collaboration.
E-learning is not an innovative education idea, since Computer-Aided Training (CAT),
Computer-Based Training (CBT), and distance learning have been used as elements of e-learning for more than ten years. Research shows that students can be effective learners over the Web, and learn as much, if not more, than in traditional courses.
E-learning is currently a burgeoning educational and training tool because of its cost saving advantages, institution reusability, and learner flexibility. World governments emphasize e-learning for social and public education, and want to enlarge it as a branch of education. The European Union in 2000, proposed the e-Europe project, promoting an information society for all. Moreover, the Japanese government has proposed the e-Japan project, making e-learning one of seven main application development items. E-learning has also been used with university and enterprise education. Enterprises can introduce e-learning courses and systems into the firm, which can then be used by the human resources or research development department to do on-the-job training. When companies induce e-learning courses into their organization, they can save money otherwise used for guest lecturers, and employees can learn on demand.
Each e-learning procedure, from course design to learner response or behaviour measurement, will affect course performance. According to previous research, instructional system design process models are process-oriented rather than product-oriented and include built-in evaluation and revision systems [88]. Systematic instructional system designs follow five learner need stages: (1) analysis, (2) design, (3) development, (4) implementation, and (5) evaluation, or the ADDIE acronym model [89]. The ADDIE is usually used in mentoring as an intervention that can be linked to three primary functions: (1) organization, (2) training and development, and (3) career development [90].
The basic reason for e-learning evaluation is to find out the effectiveness, efficiency, or appropriateness of a particular course of action. E-learning effectiveness evaluation intends to highlight good or bad practice, detect error and correct mistakes, assess risk, enable optimum investment to be achieved, and allow individuals and organizations to learn [91]. Evaluation can be most effective when it informs future decisions [92]and is better used to understand events and processes for future actions, whereas accountability looks back and properly assigns praise or blame.
Over the past few years, considerable studies have been undertaken primarily to
find the dimensions or factors to be considered in evaluation effectiveness, however, with a specific perspective. Kirkpatrick proposed four levels of training evaluation criteria: (1) reactions, (2) learning, (3) behaviour, and (4) results [93, 94]. Garavaglia [95] proposed five dimensions to evaluate e-learner change: (1) supervisory report, (2) on- the-job peer surveys, (3) action plan reports, (4) observation, and (5) self-report.
Among these five methods, the observation method can avoid the possible bias a supervisor may have when reporting on a subordinate. The self-report method involves either interviews or surveys distributed or conducted two to three months after the learning session. Phillips [96] formed a logical framework to view ROI (Return on Investment) both from a human performance and business performance perspective.
Urdan [97] proposed four measure indicators, learner focused measures, performance focused measures, culture focused measures, and cost-return measures, to evaluate corporate e-learning effectiveness. Since web-based instruction has become the most engaging type for learning, four factors that affect the e-learning environment should also be identified: (1) efficacy studies, (2) technological advances, (3) pressures of competition and cost containment, and (4) professional responses to market influences [98].
2.4.2 Methods for Evaluating E-Learning Course Effectiveness
Formative evaluation and summative evaluation are two common methods for evaluating e-learning course effectiveness in recent decades. Formative evaluation is used at the onset of new instructional program implementation to assess the needs and learning goals of an organization, or for program evaluation following training to revise existing programs. Several familiar formative evaluation models prescribe a four-part evaluation procedure employing expert reviews, one-to-one evaluations, small group evaluation, and field trials [99]. Formative evaluation is typically categorized according to different processes such as design-based, expert-based, and learner-based for assessment, although.
Summative evaluation, one of the most popular methods focused on outcomes and used in classroom education. For example, the CIRO (Contents/Contexts, Inputs, Reactions and Outcomes) model which measures learning/training effectiveness by CIRO elements, both before and after training, is currently widely used in business
[100]. The strength of the CIRO model is consideration of objectives (contexts) and training equipment (inputs). The main emphasis of CIRO is measuring managerial training program effectiveness, but it does not indicate how measurement takes place.
Adopting measures during training provides the training provider with important information regarding the current training situation, leading to improvements [101].
Summative evaluation models lack consideration of other factors, such as individual characteristics, e-learning interface design, instructional system design, and course design, which may influence e-learning effectiveness.
Most evaluation models however, do not measure e-learning effectiveness from an overall perspective and ignore the interrelation among criteria. Most evaluation models concentrate on finding factors, aspects, or casual relationships between them.
Quantitative study models mainly use traditional statistic methods or linear models (e.g.
ANOVA, factor analysis and structural equation model) to find learner satisfaction or dissatisfaction via questionnaires or facial communications [102-106]. Typically, e-learning program effectiveness is evaluated by multiple intertwined and inter-affected criteria, and the perceptions of utility for learners are not monotonic. Establishing a model to evaluate all available criteria and to determine central criteria, learner utility perception about these criteria, and the future improvement direction for the programs is necessary.