Yeh, R. C. and Tao, Y.-H., A Case Study on the continuing use of personal response system in Taiwan from the perspectives of IS success model, motivation and agency theory, The 2010 International Conference on e-Commerce, e-Administration, e-Society, e-Education,
and e-Technology, Macau, January 25-27, 2010.
A Case Study on the Continuing Use of Personal Response System in
Taiwan from the Perspectives of IS Success Model, Motivation and Agency
Theory
C. Rosa Yeha, Yu-Hui Taob*
a
Graduate Institute of International Human Resource Development,
National Taiwan Normal University, 162, Sec. 1, Ho-Ping E. Rd., Taipei 106, Taiwan
b
Department of Information Management, National University of Kaohsiung, 700, Kaohsiung University Road, Kaohsiung 811, Taiwan
*Corresponding Author: ytao@nuk.edu.tw ABSTRACT
Personal response system (PRS) is increasingly adopted in Taiwan’s higher education. As the literature mainly reports studies in UK and USA and involves few theories or models in education domain, this study attempts to conduct a small-scale case study to empirically test the perceptions of Taiwan’s college students on PRS usage from the perspectives of related theories of information system success model, motivation, and agency theory. As an initial effort in investigating PRS adoption theories from the perceptions of Taiwan’s college students, this study provides important results and implications to extend PRS usage studies to a global scope with a wider base of theoretical support.
Keywords: Personal Response System, Agency Theory, Motivation Theory, IS Success Model
1. Introduction
A recent development in instructional technology to boost classroom interaction, instant response system (IRS), includes the interactive white board (IWT) and a personal response system (PRS). PRS is sometimes labeled differently in the literature, such as audience or student response system. Since the commercialization of PRS applications in 1999, there has been several PRS review papers in the literature with the most recent one by Kay and LaSage (2009). In their paper, four other earlier major review papers were analyzed and compared, which leads Kay and LaSage (2009) to extract three categories of PRS benefits, three categories of PRS challenges, as well as four future research directions.
With the abundant existing PRS review studies in the literature, two issues in need of attention still remain. First of all, regardless of an increasing trend on PRS application in Taiwan’s higher education, there are very few related studies outside the continents of Europe, America, and Australia. This is a situation of concern since cultural or regional difference is generally recognized as having a moderating effect in the technology adoption theories or models, such as the technology acceptance model (TAM) (Davis, 1989). Secondly, some PRS studies involve theories or models in the education domain, however, none adopted the technology-based theories to explain students’usage of such educational technology in classroom. Kay and LeSage (2009) also pointed out a lack of systematic PRS research, as well as an observation that “data collection instruments are noticeably lacking in reliability and validity analysis.”Based on these arguments, we believe there is a need to explore technology-based theories and relate them to educational theories on a more systematic approach.
The link between motivational factors and the selection of instructional strategies for enhanced learning outcomes is an important aspect of education related theories. As Prensky (2003) puts it: “a motivated learner can’t be stopped.”Wlodkowskialso states,“Motivation is not only important because it is a necessary causal factor of learning, but because it mediates learning and isaconsequenceoflearning aswell”(Wlodkowski,1985,p.4).Thus, applying the motivational theories, it is valid to examine whether the use of PRS is a good motivational strategy in classroom to stimulate students’interests and to promote student engagement.
Agency theory advocates the use of incentive and control mechanism to motivate and channel employee behaviors towards organizational set goals (Bhattacherjee, 1998; Eisenhardt, 1989; Nilakant and Rao, 1994.) It has been applied towards studies in information outsourcing and project management, but few on education domain. The teacher-student relationship in classroom using PRS is similar to the principal-agent relationship often observed in business settings. Therefore, can agency theory be applied by giving some incentives as a reward to motivate students’use of PRS?
To increase the overall outcome of PRS usage, both the motivation theory and the agency theory based factors deserve more empirical investigations to clarify their effect. To address the above issues, a small-scale study was conducted with over fifty students registered in a course in a university in southern Taiwan. We believe that more effective implementation strategies or practices can be developed in Taiwan if we can understand how Taiwan’s college students perceive PRS, which helps to avoid possible cultural biases by basing only the
reported results in UK and USA. Furthermore, the theories and models investigated in this study will help broaden our understanding of PRS usage. The organization of the remaining paper is as follows: the background literature is briefly introduced in Section two, the research methods are described in Section three, the data analysis and discussion are reviewed in Section four, the extended analysis combining another pilot study is presented with implications in Section five, and the conclusions and future work are summarized in the last section.
2. Background Literature
The background literature starts with a summary of a related study by Tao and Yeh (2009) with data collected from the same study sample. After that, literature regarding the theories used in this study, including Motivation Theory, Information System (IS) Success Model and Agency Theory, are briefly reviewed before introducing the research models.
2.1 A related PRS study
Tao and Yeh (2009) reported results from an analysis of a pilot study addressing two issues: (1) to compare the perceptions of Taiwan’s college students with the PRS study results in the literature; (2) to investigate PRS technology acceptance which was lacking in the literature (Kay and LeSage, 2009). The research model integrated the Expected Confirmation Theory (ECT) and Technology Acceptance Model (TAM). The findings are summarized below. Compared with the PRS studies in the literature, Taiwan’s college students in general held positive attitude toward the use of PRS in class as well as the specific benefits of attendance and engagement, interaction, and discussion, which were also included in the review by Kay and LeSage (2009). However, some differences could still be found between the perceptions of Taiwanese students and the rest of the world. First, Taiwan’scollegestudentspreferred less time-consuming ways to prepare for class when PRS is used, i.e., not having to answer questions about class materials before each class meeting. Second, students did not believe that instructors were able to discover students’misconceptions with the use of PRS. The implication is that instructors need to invest more effort and devise appropriate strategies in designing class activities, so that PRS can be a useful aid in discovering students’ misconceptions. Third, students recognized the effects of peer instruction and discussion better when there were discussions before or after the Q&A session using PRS. Forth, students did not like the idea of using PRS to conduct the quiz evaluation toward their final grades. Even the idea of serving as a practice test before a conventional test was not well received. Fifth, students experienced more interactions with the instructor, but less among the students, which might have to do with whether peer discussions were allowed before or after the PRS Q&A session. Sixth, most students were willing to rent or buy the remote device if a
large-scale implementation was in place; however, better alternatives are mobile devices such as RFID-based remote device or a WiFi-based mobile phone, notebook computers or PDA, because these would increase the learning effects by extending the question types from simple true-false or multiple choices to other types of questions. Seventh, 21.27% of the students preferred using PRS only for two-week periods in a semester or never. This may reflect the percentage of students who did not like to see PRS in class, which was consistent with the 20-30% in the literature.
Results of the model-based empirical test are shown in Figure 1. In general, the ECT hypotheses were all supported while, among the three TAM hypotheses, only the perceived ease of use significantly impacted learning performance. This research substantiated the ECT model in PRS usage. Specifically, learning performance influences the confirmation (that learning performance is better than the prior expectation), which in turn influences the satisfaction of students. Satisfaction then leads to the intention to continue using PRS in the future. Surprisingly, what were considered the two most robust relationships in TAM, the influence of perceived ease of use on perceived usefulness and the influence of perceived usefulness on learning performance, were not supported in this PRS adoption analysis. On the contrary, the supposedly weaker relationship of perceived ease of use and learning performance was supported in this study. The authors provided some possible explanations. First, PRS has its unique features as a technology to be applied in higher education, which may require some modifications in the TAM model when conducting a model-based study on PRS. Second, a study on PRS may require a large-scale investigation with a more sophisticated research model.
Perceived ease of use Perceived usefulness Learning performance Confirmation Satisfaction Intention to continuance use H1 0.18 H2 0.372** H3 -0.006 H0.455***4
Technology Acceptance Model
Expectation Confirmation Theory
H5 0.525***
H6 0.502***
*** significantat 0.001 level; ** significant at 0.01 level
2.2 Motivation theory in instructional design
Keller (1983) defines motivation as“themagnitudeand direction ofbehavior.In otherwords, it refers to the choices people make to what experiences they will approach or avoid, and the degree of effort they will exert in that respect.”Motivation theories in educational psychology can be divided into three groups: behavior theories that include the use of reward, punishment and reinforcement to motivate, the cognitive theories that focus onthe attributional theories and equilibrium seeking motivation, and the humanistic theories that stress the need for achievement and gratification as motivators(Goh and Hooper, 2007).
Among related theories, Keller’s (1987) ARCS model for motivation and Wlodkowski’s (1985) time-continuum model are two holistic approaches to motivational design as indicated by Goh and Hooper (2007), and are two of the most cited references on motivational design. ARCS stands for Attention, Relevance, Confidence and Satisfaction, the four components of motivation which John Keller (1987a, 1987b, 1987c) synthesized from many existing human motivation theories. The ARCS model provides useful guidance on designing the motivational aspects of learning environments to enhance students’ motivation to learn. Wlodkowski’s time continuum model views students’learning motivation in temporal stages. It divides the learning process up into three critical stages: the beginning of the learning process when motivational strategies should focus on attitudes and needs, during the learning process when the emphasis of motivational strategies should be placed on stimulation and effect, and the end of the learning process when motivational strategies should focus on competence and reinforcement (Wlodkowski, 1985).
As motivation factors and learning strategies are often linked in theoretical as well empirical discussions, Pintrich, Smith, Garcia and Mckeachie (1991) developed a Motivated Strategies for Learning Questionnaire (MSLQ) “to assess college students’motivational orientations and their use of different learning strategies for a college course”(p.3). MSLQ combines measures for two broad categories, motivation and learning strategies. The motivation category is consisted of two sections, a value section and a task section, each having three subscales. The value section is consisted of the subscales of intrinsic goal orientation, extrinsic goal orientation, and task value. The task section includes subscales of control of learning beliefs, self-efficacy for learning and performance, and test anxiety. The learning strategies category is also divided into two sections. A cognitive and metacognitive strategies section includes subscales labeled rehearsal, elaboration, organization, critical thinking, and metacognitive self-regulation. A resource management strategies section includes the subscales of time and study environment management, effort regulation, peer learning and help seeking (Pintrich, et al., 1991, 1993). Most importantly, Duncan and McKeachie (2005) concluded that MSLQ has been widely adopted in many studies and thus was proven to be an
efficient, practical and ecologically valid measurement of students’motivation and learning strategies in higher education.
2.3 Information system success model
DeLone and McLean (1992) collected over 100 important papers over 7 years since 1981, and proposed six constructs to measure the successful factors of information systems (IS), including system quality, information quality, use, user satisfaction, individual impacts and organizational impacts. They think that system quality and information quality significantly impact the usage and user satisfaction while usage and user satisfaction also mutually influence each other. Then, usage and user satisfaction contribute to individual impact which later on influences the organization that this individual belongs to. Pitt et al. (1995) thought the above model proposed by DeLone and McLean (1992) emphasized on the technical view and did not consider the service role played by the IS departments. Therefore, Pitt et al. (1995) added service quality to this model as shown in Figure 2.
Service quality User satisfaction System quality Organizationa l impacts Usage Individual impacts Information quality
Figure 2. Revised IS success model (Pitt et al., 1995)
Later on, Myers, Kappelman and Prybutok (1997) added the impact of workgroup besides the individual and the organizational impacts since they thought the impacts of workgroup performance is an intermediary between individual and organizational impacts. DeLone and McLean (2003) conducted an analysis using 144 journal articles which cited their 1992 model over the ten-year period, and revised their model to include the service quality as proposed by Pitt et al.(1995) and Myers, Kappelman and Prybutok (1997), which confirmed service quality as an important factor in IS success model.
2.4 Agency theory
Agency theory originated in the 1960’s with economists trying to apply risk-sharing concept to exploring the relationships between individual and organization and between individuals. Later on, this concept was applied to the agency relationship between individuals (Jensen and Meckling, 1976). Jensen and Meckling (1976) describe the start of an agency relationship as when a principal appoints jobs to an agency by authorizing the agent certain power. This
contractual relationship is thus called agency relationship. Eisenhardt (1989) define agency relationship as the relationship between a principal and an agent who is appointed to accomplish the assigned tasks.
Agency theory assumes that human beings are boundedly rational, risk averse, and motivated by self-interest, and that organizations are characterized by goal incongruence, information asymmetry, and efficiency norms (Eisenhardt, 1989). Goal incongruence happens when the agent's goals differ from that of the principal. Information asymmetry problems occur when the principal cannot accurately observe the agent's behavior or private information utilized in such behavior. The principal and agent may also have differential risk preferences resulting in different attitudes toward risky behavior. Because of these problems within a contractual relationship, cooperative effort within organizations is often plagued by opportunistic behavior on the part of organizational members. Agency theory argues that incentive systems and control structures may help mitigate problems associated with such behavior (Bhattacherjee, 1998).
Agency theory was often applied in researches to explore how an organization, the principal, can use incentives and control mechanism to inspire or motivate the employees, the agent, to fulfill goals and objectives set by the organization (Bhattacherjee, 1998; Eisenhardt, 1989). Figure 3 shows an agency-theory based model of IT usage behavior which is influenced by outcome-based or behavior-based incentives (Bhattacherjee, 1998). The relationship between behavior-based incentives and IT usage behavior is moderated by various control structures such as monitoring. Both relationships between outcome-based or behavior-based incentives and IT usage behavior are moderated by risk aversion and goal conflict behaviors.
Monitoring behaviorRelative evaluation Outcome-based incentives IT usage behavior Multiple-period contracts Risk
aversion conflictGoal
Behavior-based incentives
The generalization of agency theory has gone beyond the traditional management studies. Any parties involved in a relationship with potential conflict of interest may be characterized as having an agency relationship. However, more agency-theory based studies were applied towards information outsourcing and project management, and few on education domain. Examples of related empirical studies are given as follows. Moynihan (2002) adopted the perspective of an agent to explore the strategies needed by information outsourcing companies to protect their own benefits, and reached a conclusion that the expectation from the outsourcing companies is to use the incentive factor to enhance the effectiveness and motivation to completing the information system development project. Mahaney and Lederer (2003) explored the high failure rate of information system projects and proposed a revised agency theory to include goal conflict, shirking, privately-held information and task programmability as the environmental variables of implicit contract. Shin (2004) explored the economics of knowledge management using resource-based, transaction-based and agency theories, and concluded that how to inspire individuals to share knowledge is an important factor. Pavlou, Liang, and Xue (2007) used agency theory to build a model to reduce the uncertainty of consumer online transaction and behavior.
3. Research Methods
This study is motivated and triggered by the impressive result of PRS usage by the authors. The findings presented in this paper are results from one of the two small case studies conducted as initial pilot studies preceding a large-scale comparative study across Taiwan, Japan and the U.S. Because of its nature as a pilot study, the objective of the study was to get an initial assessment of how Taiwan’s college students perceive the use of PRS in the classroom, and whether measurement items developed by previous researchers can be adopted reliably. Therefore, instead of a full-scale rigorous empirical research, this study was conducted on a small sample in Taiwan. The following sections briefly introduce the derivation of the integrated model and the research design.
3.1 Model derivation
The research model is an integrated model expanded from the IS success model shown in Figure 2. Since PRS is an information system for educational purpose, it is thought to be a good candidate for testing variables from cross-disciplinary theories or models. In theory, the four independent variables of the IS success model, system quality, information quality, reliability, and service quality, all have potential influences on the two outcome variables, user satisfaction and usage. Furthermore, user satisfaction and usage are thought to have mutual influences on each other. However, the ECT which explains the user’s intention to continue using the intended technology suggests that satisfaction strongly influences
continuance usage. Satisfaction also happens to be one of the four motivation components in ARCS motivation theory (Goh and Hooper, 2007). Therefore, only the causal relationship from user satisfaction to usage was retained in this study and not the vice versa, which automatically eliminated the relationships between the four independent variables and the outcome variable usage in our study. Since this study attempted to address the students’ perceptions on the continuance usage of PRS after taking courses with the use of PRS designed in class activities, the outcome variable usage was replaced by continuance usage. Furthermore, in the original model of MSLQ, the six motivation subscales are thought to influence the nine learning strategy subscales (Duncan and McKeachie, 2005). Therefore, it implies that motivation plays an important role in determining the usage of PRS in this study. Finally, continuance usage is an equivalent expression of IT usage behavior in Bhattacherjee’s (1998) Agency Theory model as seen in Figure 3. Therefore, we may assume that the outcome-based incentives as defined in the Agency Theory have some impact on the continuance usage as well. The integrated model derived from the above theoretical rationales can be seen in Figure 4, where satisfaction and continuance usage are the key factors linking IS success model to both MSLQ and Agency Theory in this proposed integrated model.
System Quality Satisfaction Incentives H1 H2 IS Success Model Agency Theory H12 Information Quality Continuance Usage H3 Intrinsic goal orientation Extrinsic goal orientation Task value Motivation Theory Risk Aversion Goal Conflict H13 H11 Service Quality H4 Control of learning beliefs Self-Efficacy for learning & performance Test anxiety H5 H6 H7 H9 H10 H8
Figure 4. The proposed integrated model As Figure 4 shows, 13 research hypotheses were derived as follows: H1. System quality is significantly related with student satisfaction.
H3. Service quality is significantly related with student satisfaction.
H4. Satisfaction is significantly related with continuance usage.
H5. Intrinsic goal orientation is significantly related with continuance usage.
H6. Extrinsic goal orientation is significantly related with continuance usage.
H7. Task value is significantly related with continuance usage.
H8. Control of learning beliefs is significantly related with continuance usage.
H9. Self-efficacy for learning and performance is significantly related with continuance
usage.
H10. Test anxiety is significantly related with continuance usage.
H11. Incentive is significantly related with continuance usage.
H12. Risk aversion significantly impacts the relationship between incentive and continuance
usage.
H13. Goal conflict significantly impacts the relationship between incentive and continuance
usage.
3.2 Research design
This PRS study was conducted in a required course--System Analysis and Design--offered by the Department of Information Management at the National University of Kaohsiung in Taiwan. In the eighteen-week course schedule, the instructor taught the first five chapters of thetextbook “System Analysis and Design in a Changing World, 4th Edition”by Stazinger, Jackson and Burd in the first five weekly class meetings, then students in 10 groups prepared and presented the remaining 10 chapters in the following 10 weeks. In each of the three-hour session, PRS was used to do the roll call at least twice and to perform at least three Q&A’s regarding the materials. The students experienced and familiarized themselves with the use of PRS in the first five meetings taught by the instructor before they were asked to design PRS applications in their own presentation. Fifty-three students registered in this course, but one dropped out after the mid-term examination. Among the remaining fifty-two students, forty-seven students filled out an online questionnaire at the end of the course.
The questionnaire was divided into three sections. The first section collected data on the items measuring the IS success model constructs (Delone McLean, 1992; Pitt et al., 1995), while the second section collected data on Motivation Theory constructs (Duncan and McKeachie, 2005), and the third section collected data on Agency Theory constructs (Bhattacherjee, 1998). All measurement items were adopted from existing literature. See the appendix for a complete list of these items and their sources by research constructs. Students were asked to respond to each questionnaire item using a 5-point Likert scale with 1 representing “extremely disagree”and 5 representing “extremely agree”.
4. Analysis and Discussion
Descriptive statistics was used to show the means and standard deviation of the 14 variables in the proposed integrated model. Simple regression was then performed to test the hypotheses. Regression is used because of the small sample size of 47 in this study, and thus the Structural Equation Modeling software such as LISREL and AMOS were not appropriate. Table 1 displays the descriptive means and standard deviation (S.D.) of all the variables.
Table 1. Variables, number of items and descriptive statistics
# Variables # of items Mean S.D. Reliability 1 System quality 3 3.34 0.64 0.62 2 Information quality 3 3.42 0.63 0.56 3 Service quality 3 3.48 0.65 0.69
4 Intrinsic goal orientation 3 3.58 0.56 0.41
5 Extrinsic goal orientation 3 3.31 0.54 0.34
6 Task value 3 3.62 0.61 0.44
7 Control of learning beliefs 3 3.51 0.69 0.84
8 Self-efficacy for learning and performance 4 3.53 0.45 0.45 9 Test anxiety 3 3.44 0.56 0.70 10 Incentives 3 3.31 0.50 0.41 11 Risk aversion 3 3.19 0.73 0.73 12 Goal conflict 3 3.33 0.57 0.53 13 Satisfaction 3 3.12 0.72 0.55 14 Continuance usage 3 3.81 0.62 0.77
The reliability scores are also shown in the last column of Table 1. Most of the variables with the reliability scores of above 0.5 may be accepted in a small-scale pilot study, but those variables with low reliability scores, such as intrinsic goal orientation, extrinsic goal orientation, task value, self-efficacy and incentives, will need to be modified in future studies to achieve a better measurement reliability. One of the possible causes of the low reliability score is that for some of the constructs some questionnaire items were trimmed down to reduce the total number of items and thus students’time to complete the questionnaire. As indicated above, five out of the fifty-two students did not fill out the questionnaires possibly because of the length of this questionnaire even after this time-saving design effort. Since all these questionnaire items were taken from well validated measurement scales and modified to fit into the need of this study, the content validity was assumed to be at acceptable level but needs to be further tested in future studies. Both the reliability and validity concerns are listed as part of the limitation of this pilot study in the conclusion section.
The hypotheses were examined by the three component theories respectively and the results were summarized in Table 2. First of all, among the four hypotheses, H1-H4, related to the IS
Success model in PRS usage, only H3 was not sustained. In other words, the data supported
the hypotheses that system quality and information quality were positively related to students’ satisfaction, while the service quality hypothesis was not supported. This may due to the fact that, unlike other functional IS in the university, PRS usage in classrooms involves virtually no service to individual students. However, the influences of system quality and information quality were only at the 0.05 significance level. Students’satisfaction was found to significantly impact the future use of PRS, which not only confirmed the IS Success Model, but also increased our confidence that students’continuance usage of PRS can be improved if their satisfaction level can be maintained through the system and the information quality level in the PRS context.
Table 2. Integrated research model and results
Theory Research Hypothesis P value
H1. System quality is significantly related with student
satisfaction.
0.023** H2. Information quality is significantly related with student
satisfaction.
0.014** H3. Service quality is significantly related with student
satisfaction.
0.838 IS success
model
H4. Satisfaction is significantly related with continuance usage. 0.001***
H5. Intrinsic goal orientation is significantly related with
continuance usage.
0.016** H6. Extrinsic goal orientation is significantly related with
continuance usage.
0.849 H7. Task value is significantly related with continuance usage. 0.403
H8. Control of learning beliefs is significantly related with
continuance usage.
0.504 H9. Self-efficacy for learning and performance is significantly
related with continuance usage.
0.056* Motivation
theory
H10. Text anxiety is significantly related with continuance
usage.
0.802 H11. Incentive is significantly related with continuance usage. 0.000***
H12. Risk aversion significantly impacts the relationship
between incentive and continuance usage.
0.702 (>3) 0.076* (<=3) Agency
theory
H13. Goal conflict significantly impacts the relationship
between incentive and continuance usage.
0.378 (>3) 0.061* (<=3)
Regarding the motivation theory, only hypotheses H5and H9, out of H5-H10, were supported.
In other words, the intrinsic goal orientation and self-efficacy for learning and performance were the two major contributing factors to motivate students’continuance usage of PRS in the future. On the other hand, the extrinsic goal orientation, task value, learning belief, self efficacy and test anxiety were not found to be related to students’intention of future PRS usage.
Finally, among the three agency theory hypotheses, H11-H13, H11 was strongly supported at
the 0.01 level. That is, with proper incentives, students can be well stimulated to use PRS to participate in their classroom activities. The moderating effect of risk aversion and goal conflict was tested by splitting the data sets into two distinctive groups and then subjected each group for a separate regression analysis. The high risk aversion or goal conflict group had an average ratings greater than 3 and the low risk aversion or goal conflict group had an average ratings less than or equal to 3. Since the p-value for H12 and H13 were slightly over
0.05 for the low risk aversion and the low goal conflict groups, and not significant for the high risk aversion and the high goal conflict groups, we may consider that risk aversion and goal conflict slightly moderated the relationship between incentives and future PRS usage. The business-oriented agency theory turned out to be an effective theory in explaining continuance usage of PRS in educational settings, which was encouraging since this study was the first such empirical test in the literature.
S ystem quality S atisfaction Incentives Information quality C ontinuance usage Intrinsic goal orientation R isk aversion G oal conflict S elf-E fficacy for learning and performance Significant at 0.05 level and above Significant at 0.1 level
Figure 5. The empirically supported research model
orientation, satisfaction, and incentives were found to strongly affect the continuance usage of PRS, while the relationship between self-efficacy for learning and performance and continuance usage of PRS only found mild support. Satisfaction was impacted by system quality and information quality. Incentives did influence continuance usage but were mildly moderated by goal conflict and risk aversion.
5. Extended Analysis and Implications
An additional synthesis may be obtained by combining the results reported above and our analyses in a previous article (Tao and Yeh, 2009) in an integrated view of the PRS continuance usage. Both research models in Figure 1 and Figure 4 include a satisfaction component that can serve as the joining point to integrate the models into one such as the extended model shown in Figure 6. All the existing models and theories in this extended model, including IS success model, TAM, ECT, motivation subscales from MSLQ and Agency theory, have been explored in a pilot study in the Taiwan context.
Figure 6. The extended research model
Based on Figure 6, four implications from our PRS pilot study can be derived as follows: First, although service quality was found not significant to satisfaction, the pilot study suggested that the reliability of PRS system and information were critical to students’ satisfaction on PRS use. This implies that even though most Taiwanese universities are not quite ready for a full-scale service support to PRS facilities, as long as the system and information remain reliable, students in Taiwan’s colleges can accept PRS use in classrooms.
S y stem quali ty
Satis fac tion
I ncenti ves Inform ation
quali ty Co ntinuanc eu sage
Intri nsi c goal o rientation
Ri sk av ersi on
G oal confl ic t Sel f-Effic ac y for
l earning and perfo rm anc e S ig ni fi cant at 0 .0 5 l evel an d a bo ve S ig ni fi cant at 0 .1 leve l C onfirm ation L earning perform anc e Percei ved ease o f use
Second, perceived ease of use was found to be the only significant factor influencing learning performance, which implies that devices as simple as a clicker operation could contribute to students’learning performance. The reason could be that PRS allows the students to compare their answers with peers in anonymity which helps students realize misconceptions instantly and motivate students to focus more and work harder to be competitive. Third, among the six motivation factors of MSLQ, only students’preference of intrinsic goal orientation and self-efficacy for learning and performance were found to significantly impact PRS continuance usage. This may imply that when facing students with other motivational preferences, the instructors may need to adjust their PRS instructional design. The school administration will also need to apply a larger-scale PRS implementation so that the classroom environment, learning and assessment benefits summarized by Kay and LeSage (2009) can be reinforced. Fourth, the finding that the incentives in Agency theory significantly influenced the PRS continuance usage implies that with proper incentives, students can be allured to continue the PRS usage. Therefore, the instructors are encouraged to give creative or generous incentives to stimulate students’acceptance and continuous use in order to achieve intended benefits of PRS usage.
6. Conclusions and Future Work
The proposed integrated model seems to be promising since the core hypotheses of the IS success model and agency theory were supported, while only intrinsic goal orientation and self-efficacy for learning and performance were found to be the two motivational factor influencing future PRS usage based on the perceptions of a small class of forty-seven students. In particular, agency theory was first used and sustained in educational domain in this study. Furthermore, the ECT and part of TAM were also sustained in this pilot study as presented by Tao and Yeh (2009). This pilot study has demonstrated its potential academic and practical contributions in furthering our knowledge on PRS usage, particularly in a Taiwanese context.
However, due to the nature of a pilot-study, the measurement scales used in this study still need to be tested for their robustness in terms of reliability and validity before the proposed integrated model can be tested empirically with accuracy. In particular, some points worthy of mentioning are as follows. First, most constructs used only partial measurement items and thus the use of the complete measurement scales is essential in future studies. Some constructs suffered from less than acceptable reliability scores, which should be tested again in future studies with the complete version of respective measurement scales. Second, this study did not distinguish between behavior-based and outcome-based incentives in the Agency Theory. Also, the moderating variables of monitoring, evaluation and multiple-period contract in the Agency Theory were not included in this pilot study. Future studies may
consider these variables to provide a complete picture of how Agency Theory contributes to our understanding of PRS usage. Third, since only two out of the six motivation theory hypotheses was sustained, efforts can be devoted to the association of motivation scales to other PRS contexts, such as the benefits and challenges summarized in Kay and LeSage (2009).
An immediate future work is to synthesize what we have learned from students’perceptions toward PRS technology in this pilot study and the findings in recent review papers (Kay and LeSage, 2009) to derive a more comprehensive model and to develop well-validated measurement scales for a comparative study among Taiwan, Japan and the U.S on the comprehensive model.
REFERENCES
Atchariyachanvanich, K., Okada, H. and Sonehara, N. 2006. What keeps online customers repurchasing through the internet? ACM SIGecom Exchanges, 6(2), 47-57.
Bhattacherjee, A. 1998. Managerial influences on intraorganizational information technology use: A principal-agent model, Decision Sciences, 29(1), 139-162.
Cheong, J.H. and Park, M.C. 2005. Mobile internet acceptance in Korea, Internet Research, 15(2), 125-140.
Davis, F. D. 1989. Perceived usefulness, perceived ease of use, and user acceptance of information technology, MIS Quarterly, 13(3), 319-340.
DeLone, W. H. and McLean, E. R. 1992. Information Systems Success: The Quest for the Dependent Variable, Information Systems Research, 3(1), 60-95.
DeLone, W. H. and McLean, E. R. 2003. The DeLone and McLean model of information systems success: A ten-year update, Journal of Management Information Systems, 19(4), 9-30.
Duncan, T. G. and McKeachie, W. J. 2005. The making of the motivated strategies for learning questionnaire, Educational Psychologist, 40(2), 117-128.
Eisenhardt, K. M. 1989. Agency theory: An assessment and review, Academy of Management Review, 14(1), 57-74.
Goh, T.-T. and Hooper, V. 2007. To txt or not to txt: That’s the puzzle, Journal of Information Technology Education, 6, 441-453.
Hong, S.-J., Thong, J.Y.L. and Tam, K. Y. 2006. The effects of post-adoption beliefs on the expectation-confirmation model for information technology continuance, International Journal of Human - Computer Studies, 64(9), 799-810.
Jensen, R., Meyer, L., and Sternberger, C. 2008. Three technological enhancements in nursing education: Informatics, instruction, personal response systems, and human
patient simulation. Nurse Education in Practice, doi:10.1016/j.nepr.2008.10.005. Kay, R. H. and LeSage, A. 2009. Examining the benefits and challenges of using audience
response systems: A review of the literature. Computers & Education, 53, 819-827. Keller, J. M. 1983. Motivational design of instruction. Instructional Design Theories and
Models: An Overview of their current status. Hillsdale, NJ: Lawrence Erlbaum.
Keller, J. M. 1987a. Development and use of the ARCS model of instructional design. Journal of Instructional Development, 10 (3), 2-10.
Keller, J. M. 1987b. Strategies for stimulating the motivation to learn. Performance and Instruction Journal, 26 (8), 1-7.
Keller, J. M. 1987c. The systematic approach of motivational design. Performance and Instruction Journal, 26 (9/10), 1-8.
Mahaney, R. C. and Lederer, A. L. 2003. Information systems project management: An agency theory interpretation, The Journal of Systems and Software, 68(1), 1-9.
Moynihan, T. 2002. Coping with client-based ‘people-problems’:thetheories-of-action of experienced IS/software project managers, Information and Management, 39(5), 377-390.
Myers, B. L., Kappelman L. A. and Prybutok V. R. 1997, A comprehensive model for assessing the quality and productivity of the information systems function: Toward a theory for information systems assessment, Information Resources Management Journal, 10(1), 6-25.
Nilakant, V. and Rao, H. 1994. Agency theory and uncertainty in organizations: An evaluation, Organization Studies, 15(5), 649-672.
Norton, W. and Moore, W. 2006. The influence of entrepreneurial risk assessment on venture launch or growth decisions, Small Business Economics, 26, 215-226.
Pavlou, P. A., Liang, H., and Xue, Y. 2007. Understanding and mitigating uncertainty in online exchange relationships: A principal-agent perspective, MIS Quarterly, 31(1), 105-136
Pintrich, P. R., Smith, D. A. F., Garcia, T. and McKeachie, W. J. 1991. A manual for the use of the Motivated Strategies for Learning Questionnaire (MSLQ). Ann Arbor: University of Michigan, National Center for Research to Improve Postsecondary Teaching and Learning.
Pintrich, P. R., Smith, D. A. F., Garcia, T. and McKeachie, W. J. 1993. Reliability and predictive validity of the Motivated Strategies for Learning Questionnaire (MSLQ). Educational and Psychological Measurement, 53, 801-813.
Pitt, L. F., Watson, R. T. and Kavan, C. B. 1995. Service quality: A measure of information systems effectiveness, MIS Quarterly, 19(2), 173-187.
Premkumar, G and Bhattacherjee, A. 2008. Explaining information technology usage: A test of competing models, Omega, 36, 64-75.
Prensky, M. 2002. The motivation of gameplay or the real 21st century learning revolution. On the Horizon, 10 (1).
Prensky, M. 2003. Digital game-based learning. ACM Computers in Entertainment, 1(1). Robbins, S.P. 1982. Personnel: The Management of Human Resource. Prentice-Hall Inc. Shin, M. 2004. A framework for evaluating economics of knowledge management systems,
Information and Management, 42(1), 179-196.
Song, S. H. and Keller, J. M. 2001. Effectiveness of motivationally adaptive computer-assisted instruction on the Dynamic Aspects of Motivation. Educational Technology Research and Development, 49(2), 5-22.
Tao, Y.-H. and Yeh, C. R. 2009. Personal response system: A model-based case study in Taiwan, 14th Annual Meeting of Asia Pacific Region of Decision Sciences Institute, July 4-8, Shanghai, China.
Venkatesh, V. and Davis, F.D. 2000. A theoretical extension of the technology acceptance model: Four longitudinal field studies, Management Science, 46(2), 184-204.
Wang, Y.-S. and Liao, Y.-W. 2008. Assessing eGovernment systems success: A validation of the DeLone and McLean model of information systems success. Government Information System, 25, 717–733
Wlodkowski, R. J. 1985. Enhancing adult motivation to learn. San Francisco: Jossey-Bass. Yen, H. R., Li, E. Y. and Niehoff, B. P. 2008. Do organizational citizenship behaviors lead to
information system success? Testing the mediation effects of integration climate and project management. Information & Management, 45, 394-402.
Yu, F.-Y., Chang, L.-J., Liu, Y.-H. and Chan, T.-W. 2002. Learning preferences towards computerised competitive modes, Journal of Computer Assisted Learning, 18, 341-350.
APPENDIX
Measurement Item Source
System quality
The personal response system functions reliably. The personal response system operates efficiently. The personal response system design satisfies my needs.
Yen, Li & Niehoff (2008)
Information quality
The personal response system provides accurate information. The personal response system provides timely information. The personal response system provides useful information.
Yen, Li & Niehoff (2008)
Service quality
I feel safe in my transactions with the personal response system service. When I have a problem, the personal response system service shows a sincere interest in solving it.
The personal response system service gives me individual attention.
Wang & Liao, (2008)
Intrinsic goal oriented
In a class like this, I prefer course material that really challenges me so I can learn new things.
The most satisfying thing for me in this course is trying to understand the content as thoroughly as possible.
When I have the opportunity in this class, I choose course assignments that I can learn from even ifthey don’tguaranteeagood grade.
Duncan & McKeachie (2005)
Extrinsic goal oriented
Getting a good course grade is the most satisfying thing for me.
If I can, I want to get better grades in this class than most of the other students.
I want to do well in this class because it is important to show my ability to my family, friends, employer, or others.
Duncan & McKeachie (2005)
Task value
I think I will be able to use what I learn in this course in other courses. I think the course material is useful for me to learn.
Understanding the subject matter of this course is very important to me.
Duncan & McKeachie (2005)
Control of learning beliefs
If I study in appropriate ways, then I will be able to learn the material in this course.
Itismy own faultifIdon’tlearnthe material in this course.
If I try hard enough, then I will understand the course material in the class using a personal response system.
Duncan & McKeachie (2005)
Self efficacy for learning and performance
I believe I will receive an excellent grade in this class.
I’m confidentIcan learn thebasicconceptstaughtin thiscourse.
I’m confidentIcan do an excellentjob on theassignmentsand testsin this course.
Duncan & McKeachie (2005)
I’m certain Ican mastertheskillsbeing taughtin thisclass. Test anxiety
When I take a test I think about how poorly I am doing compared with other students.
When ItakeatestIthink aboutitemson otherpartsofthetestIcan’tanswer. I have an uneasy, upset feeling when I take an exam.
Duncan & McKeachie (2005)
Incentives
The instructor offers incentives for using personal response system. I get rewarded when I achieve the goal in class.
The instructor generally gives us enough incentives for learning.
Bhattacherjee (2001)
Goal conflicts
I intend to use the personal response system in class.
I intend to use personal response system as much as possible.
Cheong & Park (2005) Assuming students had access to the personal response system, they intend to
use it.
Venkatesh & Davis (2000) Risk aversion
I am not willing to take risks when choosing a work environment.
I prefer a low risk/high security work environment with predictable income over a high risk and high reward environment.
I prefer to remain in an environment that has problems that I know about rather than to take the risks of a new environment that has unknown problems, even if the new environment offers greater rewards.
I view job-related risk as a situation to be avoided at all costs.
Norton & Moore (2006)
Satisfaction
It’senjoyable for me to participate in a course using personal response system.
I like to learn new skill through the instructional method using personal response system.
I hope all courses can integrate personal response system into the instructional method to practice.
Yu et al. (2002)
Continuance usage
I intend to continue using personal response system to learn new skills. Premkumar & Bhattacherjee (2008) I intend to increase my use of personal response system in the future. Hong et al.
(2006) Personal response system will become the tool of my first choice when I learn
new skills.
Atchariyachan vanich et al. (2006)
