After determined the Kirkpatrick’s model as the major evaluation theory, this study began to design the e-learning project’s performance evaluation framework. This study suggests that the entire e-learning project performance cannot be simply verified by the students’ final assessment. The e-learning project must be verified by all aspects of the entire project. Thus, this study searched popular strategy planning and evaluation models and chose the balanced scorecard as the major analytical framework.
The balanced scorecard (BSC) was first proposed in the 1990s by Kaplan and Norton. In it vision, mission, and objectives are decomposed into different views, or perspectives, as summarized through the eyes of business owners, customers, managers, employees, and stakeholders. The owners are represented by the ‘Financial’ perspective, customers and stakeholders are represented by the ‘Customer’ perspective, managers by the ‘Internal Business Process’ perspective, and employees by the ‘Learning and Growth’ perspective. The objectives and the correlative weights can be seen as the complete views of a business.
Compared to the traditional Bottom-Up planning process of MIS fields [48], BSC introduces the Top-Down concept for strategy planning. BSC suggests an organization first address the project’s vision and strategy by the organization’s core competence to determine the major perspectives and then downward to identify individual objectives and its objective targets to evaluate the project performance. Thus, BSC approach emphasizes that the organization should use a global view to evaluate the entire project and identify those unsuccessful objectives that have to be improved.
Figure 4 Balance Scorecard: from management viewpoint (Source: Niven;2002 [48])
The use of BSC approach in this study is that the Taiwan government promotes BSC concept to general firms. Thus, lots of Taiwanese organization managers and decision makers are familiar to BSC framework and feel easy to understand its applications. Introducing the BSC approach the e-learning project’s performance evaluation can reduce the learning time of the decision makers. Meanwhile, BSC approach possesses definite analytical procedures that a decision maker can acquire
feasible results by predefined steps. The BSC approach has been proved and well adopted by real world organizations that can be readily to be applied for most organizations without too many difficulties. Thus, the BSC approach also meets this study’s objective: “flexible”, “easy to use” and “easy to understand”.
The balanced scorecard (BSC) was first proposed in the 1990s by Kaplan and Norton. In it vision, mission, and objectives are decomposed into different views, or perspectives, as summarized through the eyes of business owners, customers, managers, employees, and stakeholders. The owners are represented by the ‘Financial’ perspective, customers and stakeholders are represented by the ‘Customer’ perspective, managers by the ‘Internal Business Process’ perspective, and employees by the ‘Learning and Growth’ perspective. The objectives and the correlative weights can be seen as the complete views of a business.
The BSC has evolved over time to become a full performance management system applicable to both private sector and public sector organizations, such as schools, government, or other non-profit organizations or institutes. Just by shifting the emphasis of the measurement of financial and non-financial performance, many researches ingeniously modified the vision, mission, and objectives of the four perspectives. For example, the e-learning mission may be the improvement of the overall knowledge management (KM) capabilities or increase the competitiveness of employees for a private sector firm. For a public sector organization like schools, the e-learning mission may be the increment of public reputations or improvement of educational qualities.
Based on the learning theory summarized in section 3.2 and the e-learning characteristics, this study refined the four perspectives of the BSC into Kirkpatrick’s theory as (also shown in Figure 5 with a detailed description in Table 2): (1) The reaction of the students or trainees is represented by the ‘Customer’ perspective; (2) Learning effect of learners is represented by the ‘Learning and Growth’ perspective; (3) Behavioural change of all participants of e-learning is represented by the ‘Internal business process’ perspective; and (4)Results (including financial or non-financial factors, learning efficiency/effectiveness, social reputations, organizational value) are represented by the ‘Value’ perspective. Based upon the modified e-learning perspectives and B & S assumption, this study designed a questionnaire composed of the qualitative and quantitative indicators as mentioned in section 2.4 to perform a case study for the final doctoral dissertation.
Figure 5 Applied BSC approach to e-learning performance evaluation
The performance evaluation of an e-learning project can be seen as a subset of tacit knowledge assessment, and is distinct from the traditional BSC approach of a business from several viewpoints:
(1) Schools and educational institutes are focused mainly on the learning evaluation rather than the financial return on investment. The term “learning evaluation”
is different to “learning assessment”. “Learning assessment” emphasizes on the
“assessment” of learning while “learning evaluation” includes qualitative factors like the behaviour change and social reputations
(2) Students (learners, trainee) play a principal role in learning activities covering all four perspectives of the BSC, and are especially representative of the
‘Customer’ perspective. However, for a nonprofits oriented firms like schools, the customers can be students, government, general publics and organization members.
(3) The cause and effect linkage of e-learning perspectives is different from the BSC’s original sequence.
A typical BSC procedure suggests that the organization first build up a strategy map to describe the relationship between four perspectives. The relationship between the four perspectives is that “learning and growth” generates the change of “internal process”, the improvement of “internal process” may obtain the positive reactions of
“Customer”, and the positive reactions of “customer” will produce the “financial return”.
However, if learning performance evaluation applies the BSC approach then the cause and effect linkage should be reorganized so that the organization anticipates the positive reaction of the e-learning system can improve the overall learning effectiveness thus changes the learner’s behaviour then improves the organization’s value as described in
Table 4 The four perspectives in the balanced scorecard of e-learning evaluation
Reaction(Customer, ex. students, trainees)
Mission: Deliver value-added knowledge/services learning.
Key question: Does e-learning fulfill the needs of the learner/students/trainee?
Objectives Examples:
C1: Increase the enjoyment-of-use of the e-learning environment.
C2: Provide a friendly user interface, style and functionality.
C3: Enhance organizational collaboration capability.
C4: Improve the communication between trainees and trainers.
C5: Increase user satisfaction.
C6: Increase the flexibility of the learning time arrangement.
C7: Provide abundant linkage to related learning materials.
C8: Provide sufficient examples and case studies.
C9: Provide vivid illustrations of rich multimedia materials.
Learning (Learning and Growth)
Mission: Deliver continuous improvement and prepare for future challenges
Key question: Is the e-learning project improving the knowledge and the services of the organization, and does it prepare the organization for potential changes and challenges?
Objectives Examples:
L1: Continuously improve knowledge by e-learning systems.
L2: Ensure consistent support by the organization for the e-learning project.
L3: Enhance the innovation and seniority capability of all organization members.
L4: Improve IT capability of all members.
L5: Improve professional skills of all members.
L6: Enhance personalized learning capabilities.
L7: Ensure a logical sequence among learning materials.
L8: Ensure that the learners realize what the learning subjects are.
L9: Increase the learners’ knowledge comprehension.
L10: Increase the usage frequency of the e-learning system.
L11: Increase operation familiarity with the e-learning system.
L12: Create satisfied learning results.
Behavior(Internal Business Processes)
Mission: Improve the internal business process in an efficient and effective manner
Key question: Does the e-learning project create, deliver and maintain its knowledge and services in a more efficient manner?
Objectives examples:
I1: Provide a differentiated e-learning environment.
I2: Provide incentive systems for the users.
I3: Generate a nurturing organization culture.
I4: Motivate effective learning activities.
I5: Improve the role playing capabilities during learning activities.
I6: Encourage knowledge sharing between members.
I7: Generate a change to better learning behaviour.
I8: Provide an efficient and effective learning environment
Value(Finance, Results)
Mission: Contribute to the value of the institute/school
Key question: Does the e-learning project improve the learning efficiency/effectiveness? Will the e-learning project accomplish its goal and contribute value to the organization?
Objectives Examples:
F1: Increase the value of the organization.
F2: Share knowledge with other organizations.
F3: Provide a high quality knowledge and information platform.
F4: Increase the reputation of the organization.
F5: Improve the organization's competitiveness.
F6: Increase knowledge absorption.
F7: Increase the number of research achievements.
F8: Increase the number of online users.
F9: Reduce learning costs.
4 APPLYING THE BLACK-SCHOLES MODEL
It has been explained by Cox and Ross (1976) that the option price is the expected value of payoff discounted at the risk-free interest rate over the risk-neutral distribution of the underlying asset [4]. In order to obtain the proper value of options, the field of finance has developed many sophisticated option pricing models to determine a contract value of the underlying asset under uncertainties. Among these option pricing models, the Black & Scholes (1973) model (B & S model) is the most popular one, and it has been widely adopted in real world applications.
Although the B & S model was originally designed to evaluate the option value, it has been widely adopted in various applications to determine the expected value of any target under uncertainties. This study applies the B & S model and makes two major contributions: (1) it provides a theoretical grounding for the B & S model so that it can be applied to BSC applications; (2) it proposes a measurement framework that enables managers to analyze and optimize e-learning performance. The final doctoral dissertation will also present an empirical case study to demonstrate the analyzing process of e-learning environments.