Modeling agent-based
performance evaluation for
e-learning systems
Hsiao-Ya Chiu and Chieh-Chung Sheng
Department of Management Information, Yu Da College of Business,
Chaochiao Township, Taiwan, and
An-Pin Chen
Institute of Information Management, National Chiao Tung University,
Chaochiao Township, Taiwan
Abstract
Purpose – The purpose of this paper is to show how rapidly evolving information technology has dramatically changed the knowledge dissemination process. However, many of them lack a generic evaluation process to verify the system’s performance. In an attempt to solve this problem, this study seeks to propose an agent-based model to provide a dynamic, flexible framework for performance evaluations of e-learning projects.
Design/methodology/approach – The paper proposes an agent-based model which comprises a learning model, balanced scorecard and option-pricing approach to evaluate the performance of an e-learning project.
Findings – E-learning could be the paradigm shift of traditional education. Thus, the paper provides organizations with a methodology to deliberately evaluate their e-learning projects by treating it as a continuous improvement process.
Originality/value – The original contributions in this paper are: application of a balance scorecard to weigh different perspectives; application of a real options approach for risk management of e-learning projects; construction of an agent-based system for autonomous qualitative/quantitative information gathering.
Keywords Computer based learning, Performance appraisal Paper type Conceptual paper
Introduction
Most e-learning projects require huge amounts of organizational resource such as money, time, and human training. It is necessary to monitor dynamic performance changes of the e-learning project to help organizational managers making immediately decisions. However, it is difficult to evaluate e-learning performance because there are too many qualitative/quantitative factors exist. Meanwhile, e-learning environments change so rapidly that decision makers have to dynamically adjust their development strategies in order to maximize the entire project’s performance. Thus, it is ideal to introduce concepts of risk management concepts to help decision maker delivering real time responses associated with uncertainties. In an attempt to provide dynamic evaluation solutions, this study proposes an agent-based framework that combines balanced scorecard and real option analysis approaches to help organizations dynamically and automatically analyze their e-learning projects” performance and make fundamental decisions.
The current issue and full text archive of this journal is available at www.emeraldinsight.com/0264-0473.htm
Performance
evaluation
345
Received 5 September 2007 Revised 24 October 2007 Accepted 31 October 2007The Electronic Library Vol. 26 No. 3, 2008 pp. 345-362
q Emerald Group Publishing Limited
0264-0473 DOI 10.1108/02640470810879491
Background and purpose
It is known to researchers that an e-learning environment changes so rapidly that administrators require dynamic monitoring models to provide real time managerial information. With this regard, it may be ideal to apply agent-based technology for real-time monitoring because such concepts have been well constructed for information gathering and performance monitoring for years (Klusch, 2001; Jonker et al., 1999; Preece et al., 1999; Beer et al., 2002). Klusch (2001) defined that an information agent is a just-in-time autonomous and computational software entity that can proactively acquires, mediates, and maintains relevant information on behalf of users or other agents no matter whether those resources are single or multiple, heterogeneous or geographically distributed. In short, the term “agent” is a type of autonomous and computational software which can be implemented to monitor complex situations. Unfortunately, such a concept has not yet been focused on a learning project’s performance evaluation. Thus, this paper tries to model autonomous e-learning performance agents for delivering real-time managerial information through collaboration, communication, and dynamic alerts between individual agents.
In order to construct a performance evaluation framework, the evaluation objectives should be first clearly defined by project administrators. Thus, this study adapts the concepts of the balanced scorecard (BSC) which was first developed by Kaplan and Norton in the early 1990s (Kaplan and Norton, 1992, 1993, 1996) and which has subsequently been widely adapted for making business strategies. The most widely adapted function of the BSC is the management performance system to align its vision and mission by demonstrating four different perspectives:
(1) financial; (2) customer;
(3) internal process; and (4) capacity.
Some research indicates the possibility to apply BSC concept to plan and evaluate the learning project (Forbes and Hamilton, 2003; Chiu et al., 2007). Meanwhile, this study determines the e-learning BSC perspectives based on Kirkpatrick’s learning evaluation theory (Kirkpatrick, 1959; Kirkpatrick and Kirkpatrick, 1994) in which are proposed four levels of learning including reaction, learning, behavior, and results. As a result, it will be easier for mapping Kirkpatrick’s theory to the general four visions of BSC.
Although the BSC model can help administrators make managerial decisions, such a model cannot reveal the most important information such as “will this project be successful?” or “what is the key success/failure objective?” Thus, this study applies real options approach (ROA) to answer these questions. The term “option” is a privilege sold by one party to another that offers the buyer the right, but not the obligation, to buy (call) or sell (put) the underlying asset at an agreed-on price during a certain period of time or on a specific date. It has been explained by Cox and Ross (1976) that the option price is the “expected value” of payoff discounted at the risk-free interest rate over the risk-neutral distribution of the underlying asset. The initial ROA framework focuses on the increased value of abandoning a project and liquidating the assets (Myers and Majd, 1990) that is based on the effectiveness of resource rearrangement to maximize the entire system performance (Brach, 2003). This study finds out that an organization can use the
EL
26,3
traditional BSC model to perform top-down strategy planning and applies the ROA approach to evaluate the e-learning project’s performance.
Performance evaluation of e-learning
Following the emergence of computer-based training (CBT) in the 1980s, internet-based learning in 1990s, and web-based live instructor-led training in the new millenniums, the term “e-learning” means an approach that facilitates and enhances learning through both computer and communications technology. Communications technology enables the use of the Internet such as distance learning, web-based learning platforms, collaborative authoring, virtual learning communities, multi-media/rich streaming media, course management software, and digital libraries, reusable learning objects. Organizations can adapt the latest technology to improve their learning environment under budget restrictions.
However, it is safer to consider “e-learning” as a significant extension of tradition learning instead of replacement (Henderson, 2003; Saeed, 2006). Newer technologies do not usually replace the older ones, just like new learning methodologies will not always replace the older methodologies. The theories of e-learning are built on existing learning methodologies with huge difference. For example, collaborate learning, active learning, or internet cognitive courses have been widely adopted along with classroom courses or been blended together with tutorial modules. E-mails, forums, web pages, multimedia, blog, or wikipedia have been applied as new tools of learning. Meanwhile, the learner’s behavior change from traditional learning to e-learning cannot be ignored. This study strongly emphasizes that learning from the internet and computer-assisted tools could be the paradigm shift of traditional education. Thus, organizations must carefully evaluate their e-learning projects by treating it as a continuous improvement of long term organizational strategies.
E-learning is expected to provide a higher quality learning experience, available anytime and anywhere, effectiveness/efficiency and with an even greater cost saving regarding the traditional learning environment. However, the effectiveness/efficiency of e-learning has become a controversial issue in the past few years. Many teachers disagree that the e-learning environment is a better way for learning, while others (mostly system developers) try to verify the performance of e-learning because “e-learning” is a fuzzy and growth domain which contains countless impacting factors which change quickly over time. For examples, e-learning materials can be instructor-led or self-directed (without instructors), scheduled or unscheduled, synchronous or asynchronous depending on the topics and organizations/trainers/learners requirements. “E-learning evaluation” is especially complex because it must integrate four distinct domains:
(1) learning evaluation; (2) IT improvement;
(3) project management; and (4) organizational management.
Although the formal or informal benefits of e-learning systems cannot be properly identified by most organizations, most organizations agree that e-learning may cause “paradigm shift” of learning behavior and thus increase the organization’s tangible or
Performance
evaluation
intangible value. Some approaches (Valcheva and Todorova, 2005) have been adopted to evaluate e-learning effectiveness such as comparison with traditional learning, tools and instruments, product evaluation, performance evaluation, return on investment, or comparison with a hypothetical system. Nevertheless, all these methods lack universal and quick analytical procedures (Chiu et al., 2007). This study tries to construct an analytical framework to evaluate e-learning project’s performance with objectives listed below:
. be easy to understand;
. be ready to use;
. be easy to analysis;
. be readily applied by most organizations;
. performs top-down analysis;
. be target-oriented;
. provides flexibility;
. provides real time analysis capability; and
. provides managerial information about the project’s success.
This study proposes a comprehensive framework with the above objectives by integrating agent-based systems, balanced scorecard, and options pricing analysis for e-learning project’s performance evaluation. The agent-based system can be packaged in a user-friendly interface which is easy to understand and analysis for decision makers. Meanwhile, the proposed framework can also provide efficient flexibility in a dynamic, complex, and continuously changing environment.
Design performance agents for e-learning project
Supposing an organization is running an e-learning project that should be periodically examined (hourly, daily, or monthly) for its performance because the decision makers have to analyze the project’s current status and ensure the success at deadline. The deadline can be an annual checkpoint for a long-term project or be a checkpoint between development phases. Moreover, the decision makers also want to dynamically identify the key success/fail factors of the project. In this case, it is ideal for an organization to apply automatic agents to identify the e-learning project’s perspectives and objectives in order to achieve the project’s goals. Meanwhile, the “satisfactory” level of each objective may change due to internal or external situations. Thus, this study applies ROA approach to analyze the BSC perspectives and related objectives to help decision makers evaluate the e-learning project’s performance under uncertainties. In order to simplify the use of such complex model, this study packages those analytical tools mentioned above into a simple agent-based system. Based on the concepts mentioned above, it is suitable to develop a performance agent for continuous e-learning projects. A performance agent for e-learning systems is assumed to satisfy one or many of the following requirements:
. autonomous information acquisition from online or off-line questionnaires for
learner satisfaction;
. information transformation including quantitative and qualitative measurement;
. intelligent assistance for performance monitoring; and
. just-in-time alarm according to the result of performance evaluation.
EL
26,3
The design of performance agents framework for e-learning project evaluation is illustrated in Figure 1.
There are four functions inside a performance agent. Each function is also a component agent which can active and proactive independently:
(1) Collaboration functions. Takes care of the higher level interaction between users and other agents including conversation, brokering, and negotiation. It also maintains the collaboration between multiple agents according to BSC main perspectives with detailed objectives for management purpose. Furthermore, decision makers of e-learning project use this function to initialize and adjust parameters of objectives including objectives item, target value, weight. Finally, this function represents the performance analysis report, result in graph, or automatic alarm to user via web report, e-mail, or pager.
(2) Communication functions. Communicate between users, other agents, middleware, specific API, and heterogeneous data source such as Microsoft ODBC (Open database connectivity), JDBC (Java database connectivity), or OKBC (open knowledge base connectivity) respectively.
(3) Knowledge function. Copes with dynamically acquired data and the pre-given parameters. Meanwhile, this study packs complex performance evaluation methodologies into performance agents such as real options analysis (this study uses original Black-Scholes model) and status check. In addition, knowledge function also contains required mechanism to deliver analytical results to decision makers. At the same time, it also retrieves preset rules from collaboration functions such as objectives, target value, weight.
(4) Task function. Focus on filtering, retrieving and monitoring information from various heterogeneous sources cooperating with communication functions.
Figure 1. Design of performance agents framework for e-learning project performance evaluation
Performance
evaluation
There are many qualitative/quantitative factors in e-learning projects based on the organizations’ vision. Therefore, communication functions are required to connecting different kinds of data source. For example, users can usually collect data of qualitative factors like “learners’ satisfaction” periodically from online/offline questionnaires while quantitative factors such as measurement of “usage frequency” or “efficiency of e-learning software” can be regularly retrieved from system log files.
Note that advance discussion of the procedures in “knowledge function” is mentioned in previous research (Chiu et al., 2007) which is partially demonstrated below.
Constructing performance agents
The administrators must firstly identify the organization’s e-learning goals then properly define the major perspectives of the e-learning project before and during project life cycle. Each perspective has several main objectives and goals that must be achieved. Balance scorecard is an efficient tool to explore the objectives from different roles in organizations. In this study, the four BSC perspectives identified based on (but not limited to[1]) Kirkpatrick’s learning model:
. reaction;
. learning;
. behavior; and
. value.
Reaction evaluation is to find out how the learners feel about their learning experience. Learning evaluation is the measurement of the increase in knowledge, skill, and attitudes changed from before to after the learning experience. Behavior evaluation is the extent to verify the behavior change after the learners applied the knowledge they learned. Value evaluation is typically the organizational key performance indicators such as volumes, values, percentages, timescales, return on investment, and other quantifiable aspects. However, in most situations on e-learning planning, the “results” are generally intangible. This study follows the “result” concept of Kirkpatrick model but replaces it with the term “Value” to combine the general qualitative and quantitative factors into e-learning project evaluations. The concept to clarify define objects (agents) integrate Kirkpatrick learning model and balance scorecard is demonstrated in Figure 2.
Meanwhile, this study uses ROA approach to obtain managerial information from BSC with Black and Scholes model (B and S model) (Black and Scholes, 1973) which have been widely adopted in real world applications to obtain option prices (Table I). The most widely adapted function of the BSC is the management performance system which can be used in various organizations to align its vision and mission by demonstrating four different dimensions: financial (result), customer (reaction), internal process (operation/behavior), and capacity (learning and growth). Several articles have found that the BSC system can be adapted to the evaluation of learning performance (Forbes and Hamilton, 2003), while others discussed its application in tacit knowledge management (Mitri, 2003; Chen and Chen, 2005). All these articles can help us to organize the confusion of e-learning evaluation by means of the BSC tool, something which has never been quite done before.
EL
26,3
Balanced Scorecard (BSC) is a methodology to solve challenges in balancing the theories of a strategy with its execution. The methodology is qualified for managing business strategy from top-down aligns strategic goals with objectives, targets, and metrics. It provides a balance between certain relatively forces:
. internal and external influences;
. leading and lagging indicators;
. financial and non-financial goals; and
. it cascades to all levels of the organization.
Kaplan and Norton originally addressed the four perspectives (learning, internal, customer, financial) that can guide companies as they translate strategies into
Figure 2. Framework of e-learning performance agents
Notation Option pricing applications BSC applications
C The theoretical call price The expected return of an objective that exceeds the target value
P The theoretical put price The expected return of an objective that fails to exceed the target value
S Current value of the underlying asset Current objective value
K Exercise price of the option contract The target value that an objective must exceed s Volatility of the underlying asset Standard deviation of an objective performance T Time to maturity Time to the next checkpoint
r Risk-less interest rate The anticipated growth rate of an objective
Table I. The notation for the B&S versus BSC
Performance
evaluation
actionable terms. But they do not obligate that these perspectives are necessary and sufficient conditions for success. However, BSC suggests that organizations can apply different perspectives that are more relevant to their missions or goals rather than the original ones.
It is important to note that in mission-driven organizations like schools, government, or non-profit-oriented organization, the mission is not limited to the financial goals (but could include financial items). A BSC planning must be designed according to the goals and missions of the organization in order to apply e-learning environment to increase the firm’s competition capabilities and values.
After successfully developed the perspectives and objectives of the e-learning project, the evaluation procedure introduces B and S model to address the expected value of each objective at the deadline. The core concept of the B and S model is that, assuming that there exists a portfolio containing a certain stock and its call option, adjusting the proper hedging ratio (ratio between stocks and its call options) can transiently maintain this portfolio in a risk-less state. If there are no arbitrage opportunities, then this portfolio merely makes risk-free returns. With this concept, Black and Scholes derived the option pricing formula:
C ¼ SN ðd1Þ 2 Ke2rTN ðd2Þ P ¼ Ke2rTN ð2d2Þ 2 SN ð2d1Þ where: d1 ¼ 1n S K þ r þs2 2 T spffiffiffiffiT d2 ¼ d1 2spffiffiffiffiT
Note that N(.) denotes the cumulated normal distribution, C is the call option price, P is the put option price, S is the current value of the underlying asset, K is the exercise price, T is the remaining time to maturity (in years), s is the volatility of this underlying asset, and r is the risk-less interest rate. This study employs the notation of the B and S parameters to perform the BSC analysis and construct an analyzing structure illustrated in Table II.
The individual objective is measured periodically to investigate its score of satisfactory and then transferred into a measure to indicate its relative current value as the objective index. The standard deviation of a certain objective measure is called volatility. Each measure of a certain objective has its target value to be achieved after T years indicating the feasible performance (satisfactory degree) requirement. Different weighting values are assigned to indicate the importance of each objective while keeping the sum of all weighting values equal to 1. If we add or remove any objective we must readjust the sum of the existing weighting values to 1:
Xn1 i¼1 WCi¼ Xn2 i¼1 Wli¼ Xn3 i¼1 WIi¼ Xn4 i¼1 WFi¼ 1
EL
26,3
352
Objective score Expected growth rate Volatility Previous measure Current measure Target value Objective weight Objective index Fail index Success value (X )( rG )( s G )( P G )( S G )( K G )( W G )( G G )( Z G )( V G ) Reaction a C1 rC1 sC 1 PC 1 SC 1 KC 1 W C 1 GC 1 ZC 1 VC 1 ... ... ... ... ... ... ... ... .. . ... Cn 1 rCn 1 sCn 1 PCn 1 SCn 1 KCn 1 WCn 1 GCn 1 ZCn 1 VCn 1 Learning b L1 rL1 sL 1 PL 1 SL 1 KL 1 W L 1 GL 1 ZL 1 VL 1 ... ... ... ... ... ... ... ... .. . ... Ln 2 rLn 2 sLn 2 PLn 2 SLn 2 KLn 2 W Ln 2 GLn 2 ZLn 2 VLn 2 Behavior c I1 . rI1 sI1 PI1 SI1 KI1 WI1 GI1 ZI1 VI1 ... ... ... ... ... ... ... ... .. . ... In3 rIn3 sIn 3 PIn 3 SIn 3 KIn 3 W In 3 GIn 3 ZIn 3 VIn 3 Value d F1 rF1 sF 1 PF 1 SF 1 KF 1 W F 1 GF 1 ZF 1 VF 1 ... ... ... ... ... ... ... ... .. . ... Fn 4 rFn 4 sFn 4 PFn 4 SFn 4 KFn 4 WFn 4 GFn 4 ZFn 4 VFn 4 Notes: a Perspective weight: PW C , Perspective index: PIC Perspective fail index: PZ C , Perspective success value: PV C ; b Perspective weight: PW L ; Perspective index: PIL , Perspective fail index: PZ L , Perspective success value: PV L ; c Perspective weight: PW I , Perspective index: PII , Perspective fail index: PZ I ,Perspective success value: PV I ; dPerspective weight: PW F ;Perspective index: PIF ,Perspective fail index: PZ F ,Perspective success value: PV F Table II. The analyzing objectives in performance agents
Performance
evaluation
and
PWcþ PWLþ PWIþ PWF ¼ 1
Each objective is measured in relative value compared to the last measuring point. For example, the proper measurement will be “I am more satisfied with our e-learning system compared to last month: score 0 to 4” rather than “I am satisfied with our e-learning system: score 0 to 4”.
The organization has to identify the investigating method (data collecting or questionnaire investigation) and configure the weighting factors for indicating the importance of each perspective PG and objectives WG. The growth rate rG can be greater (increase) or less (decrease) or equal (neutral) to zero. It is better to apply neutral perspective (rG ¼ 0) to the growth rate terms unless the objective score can be properly forecasted. The perspective weights can be determined by the consumed resource (or cost) of each perspective. The analyst can initially determine the weight of each objective by its percentage of consumed resource to the perspective.
When issuing questionnaires to e-learning users and IT staff members to collect intangible objectives information, the score is ranked as 0 (strongly disagree), 1 (disagree), 2 (neutral), 3 (agree) and 4 (strongly agree) to indicate the performance variation from the last checkpoint. All questions must be designed so that the higher score indicates the higher performance. Then, collect data to evaluate the tangible objectives. All quantitative values are then compare to the last checkpoint and transform the performance into 0 to 4 scores for each objective so as to indicate the performance from strongly decreased (score ¼ 0) to strongly increased (score ¼ 4).
The current measure is calculated with the following formula:
SG¼ PGð0:95 þ X £ 0:025
where
G[ Cf 1; . . .Cn1; L1; . . .Ln2; I1; . . .In3; F1; . . .Fn4g
and
X [ {C1; . . .Cn1; L1; . . .Ln2; I1; . . .In3; F1; . . .Fn4}
This formula indicates that the current measure will be 95 per cent of the previous measure if the score is zero and will be 105 per cent of the previous measure if the score is 4 (max variation is ^ 5 per cent). For the first time issuing of the BSC, this study subjectively set PG ¼ 10.0 as its initial value. The 5 per cent variance and initial value 10.0 is subjectively set by this study, an analyzer may flexibly adjust these values without influencing the final results. Meanwhile, the current measure is derived from the previous measure that assures the objective measure varies in a form of Xi þ 1/Xi to emulate the option’s underlying asset’s price variation in the real world. This study assumes that ln (Xi þ 1/Xi) follows a normal distribution and thus X follows a lognormal distribution as B and S model suggests.
The volatility can be derived as a standard deviation in nature log ln(.) from its previous measures. Assumes that the future volatility behaves like its previous n measures, and then the periodically standard deviation can be calculated as:
EL
26,3
sG0¼ StdðPG;t; PG;t21; . . .PG;t2nÞ
Where Std(.) denotes the standard deviation function. Suppose that the BSC is periodically measured every t days, the annual volatility will be:
sG¼
ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi
365 £ðsG0Þ2
t :
s
For a quantitative objective, the objective score can be measured as (with continuous compound interest rate method):
m1¼ e 1ny 1 y 0 ð Þ cp ; m 2¼ e 1n y 2 y 0 ð Þ cp score ¼ 2 þ cv 2 m1 m22 m1 ð Þ=2
where cvindicates the current incremental rate, cpindicates the checkpoint counts. For
example, the success condition of “user number” is “must exceed 2,000” indicates that this objective is failed if the final user number less than 2,000 while the organization will be strongly satisfied if the “user number” exceeds 2,200. Suppose that there are six months before the deadline and the user number is 1,400 at last month. The satisfactory investigation is regularly performed each month and the current user number is 1,491 at this month. Thus, the user number has to be increased 6.12 per cent (e1nð2;000=1;4006 ¼ 1.0612 implies 6.12 per cent increment) each month. However, if the
user number increased 7.82 per cent (e1nð2;200=1;4006 ¼ 7.82 per cent increment) the result
will be strongly satisfied. The current user increment rate is (1,491 2 1,400)/ 1,400 ¼ 6.5 per cent, thus the objective score is (0.065 2 0.0612)/((0.0782 2 0.0612)/2) þ 2 ¼ 2.44.
For a qualitative objective, the objective score can be measured as:
m1¼
3 £ ct þ 4 £ cs þ 2ðcp 2 ct 2 csÞ
cp ; m2¼ 4
score ¼ 2 þ cv 2 m1
ðm22 m1Þ2
Where the objective’s success point is measured as “at least ct satisfy counts and cs strongly satisfy counts cumulated” before deadline. Because the performance is
evaluated by its slope (incremental rate), the target value KGequals to the objective’s
initial value. As a result, both quantitative and qualitative objectives are normalized into the “satisfactory level” and thus can be summarized in the same scale.
An objective index is defined as a value that indicates the expected gain for each objective exceeds its target value after T years. An objective fail index is defined as the expected value that an objective fails to exceed its success condition:
Performance
evaluation
GG¼ SGN ðd1Þ 2 KGe2rGTN ðd2Þ £ XG ZG¼ KGe2rGTN ð2d2Þ 2 SGN ð2d1Þ £ WG 8 < : where d1 ¼ 1n SG KG þ rG þsG 2 T sG ffiffiffiffi T p d2 ¼ d1 2sG ffiffiffiffi T p :
Comparing the expected success and fail values can generate the objective’s success index. In order to provide a meaningful value, this study uses logarithm value Vr that positive means success (higher is better) and negative means fail:
VG¼ 1n
GG
ZG
The perspective index value indicates the total performance summarized from its objective indices: PIC ¼ PWC£ Xn1 i¼1 GCi PIL¼ PWL£ Xn2 i¼1 GLi PIF¼ PWF£ Xn4 i¼1 GFi PII ¼ PWI£ Xn3 i¼1 GIi
The perspective fail index indicates the expected fail value that the entire perspective fails, which can be summarized from its objective fail indices:
PZC ¼ PWC£ Xn1 i¼1 ZCiPZL¼ PWL£ Xn2 i¼1 ZLiPWi£ Xn3 i¼1 ZIiPZF ¼ PWF£ Xn4 i¼1 ZFi
Similarly, the perspective’s success index can be calculated as:
PVC ¼ 1n PIC PZC PVL¼ 1n PII PZI PVF ¼ 1n PIF PZF
EL
26,3
356
The BSC index value can thus be summarized from four perspective indexes in (s1), also, the BSC fail index can be summarized from four perspective’s fail indexes in (s2). Finally, the BSC success value can be calculated as (s3):
BSCindex ¼ PIC þ PIL þ PII þ PIF ðs1Þ
BSCFailIndex ¼ PZC þ PZL þ PZI þ PZF ðs2Þ
BSCSuccessValue ¼ lnðBSCindex=BSCFailIndexÞ ðs3Þ
A brief example
Suppose that the entire BSC perspectives and objectives have been determined by the decision maker. C1 represents the first objective “Increase the enjoyment-of-use of e-learning environment” of reaction perspective. The success point of C1 is “at least three cumulated satisfied score must be achieved”. The decision maker also determined the Reaction perspective weight ¼ 0.25 and objective weight of C1 is 0.6 according to the consumed resources. The previous measure of C1 at the first time and second time is 10.08 and 10.12 accordingly.
The questionnaire score of C1 equals to 2.26 in the investigation performed at the third checkpoint. Based on the success point “at least three cumulated satisfied score must be achieved”, the objective score of C1 can be calculated and thus the objective
score of C1 is 1.94. For example, the current measure of C1 is SC1¼ 10.12*
(0.95 þ 1.94*0.025) ¼ 10.11. The volatility of C1 can thus be calculated as
sC10¼ Std(10.08, 10.12, 10.11) ¼ 0.0208. Because we collect the questionnaires every
specific period, for example, 30 days (t ¼ 30), thussC1¼ 0.0727. The decision makers
want to explore if the performance can exceed the organization’s goals of the final deadline, the time to maturity can be calculated as T ¼ (checkpoint 2 – checkpoint 1)/365 ¼ 0.75 years (for example). Finally, the objective indexes of C1 can be calculated
with parameters (SC1, KC1, sC1, T, rC1) ¼ (10.11, 10.0, 0.0727, 0.75, 0) to obtain the
objective index (GC1¼ 0.1852) and fail index (ZC1¼ 0.1220). Thus obtain the success
value of C1 (VC1¼ 0.4169).After applying equation (s1) to (s3), performance agent
based on BSC analyzing sheet can be illustrated as Figure 3 and a summary report in Figure 4.
The system is originally developed via Excel, and under construction to an agent-based environment, to fulfill the continuous, dynamical, extremely change environment. Decision makers of e-learning projects can quickly figure out:
. the entire e-learning project is possible to succeed;
. the best performing perspective is value;
. the best performing objective is C1; and
. objective F1 must be carefully monitored because it has largest objective index
and fail index.
Performance
evaluation
Figure 3. A short example of performance agents of e-learning system Figure 4. The performance of e-learning project on web report
EL
26,3
Furthermore, “sensitivity analysis” in ROA is also a useful analysis process widely adopted by option traders, which are implemented in performance agent to provide advanced decision information. This analytical process measures how an option price responds to a small change in certain factors. By performing sensitivity analysis, decision makers can obtain additional information that is valuable for strategy planning and resource relocating. For example:
(1) Delta: refers to N(d1) of the B & S formula. Delta indicates the ratio that the underlying asset’s price change will affect its option price. For example,
Delta(G) ¼ 0.5 means that the call value will increase 0.5 point if SGincreases 1
point. This factor can be applied to search the most efficient objective that can improve the BSC index value. In this case, we found that I2 is the most efficient objective (Delta(I2) ¼ 0.0401) to increase the BSC index value:
Delta ¼›C
›S¼ N ðd1Þ
(2) Gamma: is used to evaluate the sensitivity of Delta or the acceleration of SG. This factor can be used to determine the potential efficiency of each objective. In this case Gamma(L9) ¼ 1.1665 obtained the largest value which indicates that L9 is the most potential objective to increase the entire e-learning project’s performance: Gamma ¼› 2C ›S2¼ N0ðd1Þ SspffiffiffiffiT where N0ðd1Þ ¼ 1ffiffiffi 2 p pe 1 2d12
(3) Vega: this factor can be used to evaluate the impact of volatility change to its call price. Vega can help the analyst address the most volatility sensitive objective in the BSC sheet. The Vega values are multiplied by the objective weight and the perspective weight to indicate the real impact on the final BSC index value. In this case, we found that I5 is most sensitive (Vega(I5) ¼ 0.2418) to the volatility change in our BSC objectives:
Vega ¼›C ›s¼ S ffiffiffiffi T p N0ðd1Þ
(4) Rho: is used to evaluate the influence of the risk-less interest rate. It can help the analyst to address the most valuable objective if the anticipated growth rate has changed:
Rho ¼›C
›r ¼ TKe
2rTN ðd2Þ
(5) Theta: is used to evaluate the impact between time to maturity and the call price. It can help the analyst to address the most sensitive objective to T. In this
Performance
evaluation
case, we multiplied the Theta with the objective and the perspective weights. This study found that I3 is the most sensitive objective (Theta(I3) ¼ 0.0351) if the e-learning check point varies from the pre-determined check point:
Theta ¼ ›C ›T¼ sSN0ðd1Þ 2pffiffiffiffiT þ rKe 2rTN ðd2Þ Conclusions
This study has proposed a framework for an agent-based performance system for e-learning project performance evaluation, which integrates the BSC approach and B and S model to evaluate an organization’s e-learning project performance with both quantitative and qualitative factors. The applications of this framework are not limited to e-learning projects but could also be applied to various digital content systems. It can be flexibly adopted to evaluate any BSC-based investigations if the objectives are designed in relative measurement methods. With the help of performance agents, the entire system delivers meaningful information to help decision makers address the most success/weakest factors of the e-learning project under uncertainties. This framework can dynamically generate required managerial information dealing with uncertainties, which can greatly reduce the risks of project fail. Administrators can now use the proposed framework to dynamically fine tune their resources to maximize their e-learning project’s effectiveness.
Note
1. There are other great learning evaluation models such as Bloom’s (1956) taxonomy and the following researches, which can also be adapt to e-learning performance evaluation. References
Beer, M.D., Huang, W. and Sixsmith, A. (2002), “Using agents to build a practical implementation of the INCA system”, in Jain, L.C. (Ed.), Intelligent Agents and Their Applications, Springer-Verlag, New York, NY, pp. 217-48.
Black, F. and Scholes, M.S. (1973), “The pricing of options and corporate liabilities”, Journal of Political Economy, Vol. 81 No. 3, pp. 637-59.
Bloom, B.S., Krathwohl, D.R., Masia, B.B. and Engelhart, M.D. (1956), Taxonomy of Educational Objectives: The Classification of Educational Goals[M], Longmans Green, New York, NY. Brach, M.A. (2003), Real Options in Practice, John Wiley & Sons, New York, NY.
Chen, M.Y. and Chen, A.P. (2005), “Integrating option model and knowledge management performance measures: an empirical study”, Journal of Information Science, Vol. 31 No. 5, pp. 281-393.
Chiu, H.Y., Sheng, C.C. and Chen, A.P. (2007), “Modeling e-learning system performance evaluation with agent-based approach”, paper presented at KES2007 11th International Conference on Knowledge-Based and Intelligent Information & Engineering Systems, Vietri sul Mare, Italy, 12-14 May 2007.
Cox, J.C. and Ross, S.A. (1976), “The valuation of options for alternative stochastic processes”, Journal of Financial Economics, Vol. 3, pp. 145-66.
EL
26,3
Forbes, L. and Hamilton, J. (2003), “Building an international student market: educational-balanced scorecard solutions for regional Australian cities”, International Education Journal, Vol. 5 No. 4, pp. 502-20.
Henderson, A.J. (2003), The E-learning Question and Answer Book, American Management Association, New York, NY.
Jonker, C.M., Lam, R.A. and Treur, J. (1999), “A multi-agent architecture for an intelligent website for insurance”, Proceedings of Cooperative Agents III (LNAI serious), Springer, New York, NY, pp. 86-100.
Kaplan, R.S. and Norton, D.P. (1992), “The balanced scorecard: measures that drive performance”, Harvard Business Review, Vol. 70 No. 1, pp. 71-80.
Kaplan, R.S. and Norton, D.P. (1993), “Putting the balanced scorecard to work”, Harvard Business Review, Vol. 71 No. 5, pp. 134-47.
Kaplan, R.S. and Norton, D.P. (1996), “Using the balanced scorecard as a strategic management system”, Harvard Business Review, Vol. 74 No. 1, pp. 75-85.
Kirkpatrick, D.L. (1959), “Techniques for evaluating training programs”, Journal of the American Society of Training Directors, Vol. 13, pp. 3-26.
Kirkpatrick, D.L. and Kirkpatrick, J.D. (1994), Evaluating Training Programs: The Four Levels, 3rd ed., Berrett-Koehler Publishers, San Francisco, CA.
Klusch, M. (2001), “Information agent technology for the internet: a survey”, Data and Knowledge Engineering, Vol. 36, pp. 337-72.
Mitri, M. (2003), “Applying tacit knowledge management techniques for performance assessment”, Computers & Education, Vol. 41, pp. 173-89.
Myers, S.C. and Majd, S. (1990), “Abandonment value and project life”, Advances in Futures and Option Research, Vol. 4 No. 1.
Preece, A.D., Hui, K.-Y., Gray, W.A., Marti, P., Bench-Capon, T.J.M., Jones, D.M. and Cu, Z. (1999), “The KRAFT architecture for knowledge fusion and transformation”, 19th SGES International Conference on Knowledge-based Systems and Applied Artificial Intelligence (ES’99), Springer, Berlin.
Saeed, R.S. (2006), “How digital libraries can support e-learning”, The Electronic Library, Vol. 24 No. 3, pp. 389-401.
Valcheva, D. and Todorova, M. (2005), “Defining a system of indicators for evaluation of the effectiveness of e-learning”, paper presented at International Conference on Computer Systems and Technologies – CompSysTech 2005.
About the authors
Hsiao-Ya Chiu is an Assistant Professor in the Department of Management Information, Yu Da College of Business, Taiwan. For several years, her main teaching subject have been information management, database application, and electronic commerce. Dr Chiu focuses research domains on the change of learning behaviour, change in higher education and organizations, and related performance evaluation methods in the network era. She is the corresponding author and can be contacted at: hychiu@webmail.ydu.edu.tw
Chieh-Chung Sheng has been an Assistant Professor for more than five years in the Department of Management Information, Yu Da College of Business, Taiwan. He majored in information management and has considerable working experience in the field of management information systems. Currently, Dr Sheng is working on applying financial engineering methodologies to solve real world problems. He also has strong interests in computational intelligence and believes that it may be the future to solve highly unstructured problems.
Performance
evaluation
Professor An-Pin Chen has been Director of the Institute of Information Management at the National Chiao Tung University, Taiwan for four years and is also the CEO of the EMBA program in the College of Management at NCTU. Professor Chen is now concentrating on integrating finance and information management related researches. His courses such as intelligence-oriented management decision systems, special topic on financial forecasting, and a seminar on information and management have become the most popular programs in the College of Management. Professor Chen is also chief consultant of the Polaris Financial Group, which is one of the largest securities company in Taiwan, and plays a major role in providing advice to the in-house dealings, new derivative financial products, and bond department throughout the entire Polaris Groups.
EL
26,3
362
To purchase reprints of this article please e-mail: reprints@emeraldinsight.com Or visit our web site for further details: www.emeraldinsight.com/reprints
