Benefits and Challenges of Personal Response System on Student
Satisfaction and Usage Continuance Intention
C. Rosa Yeha, Yu-Hui Taob*, Shiro Uesugic
a
Graduate Institute of International Human Resource Development, National Taiwan Normal University, 162, Sec. 1, Ho-Ping E. Rd., Taipei, Taiwan
b
Department of Information Management, National University of Kaohsiung, 700, Kaohsiung University Road, Kaohsiung 811, Taiwan
c
Faculty of Business Administration, Matsuyama University 790-8578 Bunkyo 4-2, Matsuyama, Ehime, Japan
*Corresponding Author: ytao@nuk.edu.tw
ABSTRACT
Since the introduction of the personal response system (PRS) to commercial market, its adoption in higher education systems has increased. Extensive studies have been done on this tool, and corresponding PRS review papers have been published. The value of these review papers may increase, if outcomes will be empirically validated further with existing theories.As an initial effort, we investigate how the benefits and challenges of PRS, as concluded by Kay and LeSage (2009), are perceived by experienced college students in Taiwan. The research results confirmed that PRS challenges are partially while PRS benefits are partially but strongly influencing student satisfaction and the intention to continue usage. This research not only contributes by empirically validating PRS benefits and challenges derived from literature, but also by indicating that teachers may adopt PRS educational technology with fewer impacts from its limitations. This should encourage practical implementation of PRS. Some suggestions regarding PRS implementations and future research directions are also discussed.
Keyword: Personal Response System, Expectation Confirmation Theory, Satisfaction, Benefits, Challenges
1. Introduction
Personal Response System (PRS), an audience and student response system, has been studied extensively since a new generation of infrared PRS became available in 1999. The latest PRS review published by Kay and LeSage (2009) covered papers from 2000 to 2007, including several earlier review papers such as Judson and Sawada (2002), Fies and Marshall (2006), Simpson and Oliver (2007), and Caldwell (2007). These papers agree considerably on the benefits and potential challenges of using PRS
as well as on future research opportunities.
Four related PRS issues can be observed in the Kay and LeSage review (2009). The first issue is how to identify the reasons why these specific benefits and challenges influence the use of PRSs. Prior to arriving at this potential research agendum, we were more interested in the empirical verification of the impacts these benefits and challenges have on student perception. Second, while Kay and LeSage (2009) identified technology-based challenges in their research, few PRS studies actually involved technology-related theories and discussed how to establish its use in classroom setting. The TAM meta-analysis of King and He (2006) reveals that technology is treated as one potential moderating factor influencing several model relationships. This is applicable to other technology-based theories such as, Expectation Confirmation Theory (ECT) by Bhattacherjee (2001). Third, Kay and LeSage (2009) also called for the expansion of the context of PRS use to social science subjects and K-12 classrooms. Before achieving such expansion, it is more favorable to obtain generalized research results by expanding samples in a small class or single university to a larger regional or country-wide scope. Lastly, Kay and LeSage’s (2009) study listed systematic research with reliability and validity analysis as the first direction in relation to further studies on this research area.
To address the four issues identified, we formulate a research model on PRS usage and conduct a robust empirical study on PRS perceptions with Taiwan’s college students as representative sample. This procedure will guarantee a systematic research with reliability and validity analysis lacking in most PRS studies. Moreover, critical literature analysis of Kay and LeSage’s (2009) review papers is presented in Section 2, which provides the context in formulating the proposed research model and design of the empirical test in Section 3. A detailed data analysis and discussion are presented in Section 4, followed by the presentation of conclusion and limitations in Section 5.
2. Background Information
Recent studies provide adequate evidence on the positive perception of students on the use of ARS in higher education. The summary of Kay and LeSage (2009) includes the benefits, challenges, key problems, and future directions.
The benefits of ARS are grouped into three categories: classroom environment, learning, and assessment. Classroom environment benefits include attendance, attention, anonymity, participation, and engagement. Learning benefits include interaction, discussion, contingent teaching, learning performance, and quality of
learning. Assessment benefits include feedback, and formative and comparative assessment.
The challenges of ARS are grouped into three categories: technology-based, teacher-based, and student-based challenges. Technology-based challenges include bringing non-functional remote control devices and ARS. Student feedback, coverage, and question formulation are examples of teacher-based challenges. Student-based challenges include acceptability of new methods, discussion, effort, summative assessment, attendance for grades, identifying students, and negative feedback.
Key problems encountered on current ARS research include, the lack of systematic research methodology, a bias toward using the anecdotal, lack of qualitative data, excessive focus on attitudes as opposed to learning and cognitive process, and inconclusive samples derived from limited education settings.
Four future research directions for ARS were identified. These are, the need to determine why specific benefits and challenges influence the use of ARS, the need for an in-depth research analyzing the impact of specific types of question on creating a student-center learning, the need for knowledge-rich learning that builds classroom community, where the call for ARS expansion to include the social sciences subject areas and K-12 classrooms was pushed; and the necessity for more research on individual differences in the use of ARS, focusing on gender, year level, age, and learning style.
3. Research Methods
Based on the research questions and knowledge obtained from literature review, a description of the research model and hypotheses derivation is provided in Section 3.1. This is followed by a description of the design of the empirical study in Section 3.2.
3.1 Model and hypothesis development
In technology-related theories, it is natural to think of the casual consequence of PRS benefits and challenges on satisfaction as perceived by the students. In ECT (Bhattacherjee, 2001), the ultimate goal is to determine the impact on continuance usage intention mediated by satisfaction. On the contrary, the IS success model (DeLone and McLean,2003)points out the parallel relationship between satisfaction and usage as regards their common antecedents.
benefits and challenges on usage continuance intention, which, according to ECT, is mediated by satisfaction. Three hypotheses can be derived accordingly:
H1 PRS benefits are significantly related with satisfaction.
H1-1. Classroom environment benefits are significantly related with satisfaction.
H1-2. Learning benefits are significantly related with satisfaction.
H1-3. Assessment environment benefits are significantly related with satisfaction.
H2 PRS challenges are significantly related with satisfaction.
H2-1. Technology-based challenges are significantly related with satisfaction.
H2-2. Teacher-based challenges are significantly related with satisfaction.
H2-3. Student-based challenges are significantly related with satisfaction.
H3. Satisfaction is significantly related with the intention to continue usage.
Accordingly, the research hypotheses can be formulated as shown in Figure 1.
Intention to continue usage Satisfaction Technology-based challenges Teacher-based challenges Student-based challenges IRS challenge IRS benefits Classroom environment benefits Learning benefits Assessment benefits H3 H1 H2
Figure 1 The proposed integrated model
3.2 Research design
As the previous section addresses the first two of the four issues pointed in Introduction, this section addresses the remaining two issues. The objective of this study is to come up with a result representative of the 164 higher educational institutions in Taiwan. Therefore, the target population comprises college students who have used PRS systems in their classes. Majority of the research conducted on
Taiwan population divides the population into North, Central, and South. Several studies added the East geographical regions. To address this issue, we selected sample populations from three universities in each of the four regions. The sampling approach is to identify the teachers in each university who are willing to assist through distribution of the questionnaire to students in the PRS classes.
The student questionnaire has four sections. The first section consists of questionnaire items aimed at collecting data for measuring PRS benefits (Kay and LeSage, 2009). The second section intends to collect data on PRS challenges (Kay and LeSage, 2009). Both were based on the 13 benefits and 12 challenges summarized by Kay and LeSage (2009). However, there are only two items in the technology-based challenges and one of the items regarding “bringing” the remote may not be applicable to most current situations. For this reason, we added alternative items to represent questions that are not applicable. New items were also created to complete the total number of questions in this variable to three. The third section aims to collect data on satisfaction and the intention to continue usage based on Bhattacherjee (2001). Four questions on satisfaction and four questions on the intention to continue usage were included. The last section intends to collect data on the five personal profile items, including sex (male vs. female), school location (north, central, south, or east), IRS experience (1, 2, 3, and above 3 classes), and class performance (top 25%, top 50%, top 75%, or bottom 25%). See Appendix for the detailed listing of questionnaire items and corresponding sources. Students were asked to respond to each questionnaire item in the first three sections using a 7-point Likert scale, with 1 representing “extremely disagree” and 7 representing “extremely agree.”
A systematic research with reliability and validity analyses was conducted. The analytical method used for the descriptive statistics was SPSS. Partial Least Square (PLS)-based Structural Equation Modeling (SEM) was used for the path model regarding PRS benefits or challenges impacting on satisfaction and continuance of use.
4. Analysis and Discussion
A total of 407 valid questionnaires were returned. The descriptive statistics for the five profile variables are shown in Table 1. A sample profile is summarized as follows: female students are twice the percentage of the male students in our data, which may have a relation with the classes selected for data collection. Except for the school location in the south, which accounts for almost 50% of the samples, students from the three other locations have fairly even spread of percentages. Majority of students
were taking their first PRS classes at the time of the survey. While only 3.7% had taken three PRS classes, about 20% had taken two or above three PRS classes, which is a significant percentage compared with that of the first timers. Only about 5% of the students considered themselves to be at the bottom 25% of the overall class performance, while majority of them evaluated themselves to be at the range of 25% to 50%. College students’ participation was representative of the range from freshman to senior with over 10% in each year level. Only a slight 3.2% of the graduate students participated in this investigation.
Table 1 Student profile
Gender: Male 131(32.3%) Female 263(64.8%) Location: North 56 (13.8%) Center 80 (19.7%)
South 201 (49.5%) East 59 (14.5%) IRS classes: One 224 (55.2%) Two 81(20%)
Three 15 (3.7%) Above three 78 (19.2%) Expected class
performance:
Top 25% 72 (17.7%) Top 50% 223 (54.9) Top 75% 77 (19%) Bottom 25% 20 (4.9%) Year level: Freshman 87 (21.4%) Sophomore 155 (38.2%)
Junior 86 (21.2%) Senior 48 (11.8%) Graduate 13 (3.2%)
Descriptive statistics was used to derive the means and standard deviation of the eight variables in the proposed integrated model. The SmartPLS (Ringle et al., 2005) was used to analyze the path model. The second column in Table 2 displays the descriptive means and standard deviation (S.D.) of all the variables. In general, the mean scores of the three benefits that exceed 5 indicate a relatively positive perception of PRS benefits. Challenges that are slightly under 4, expressed as almost neutral perceptions, reflect a moderately positive perception of PRS challenges. Between the two outcome variables, satisfaction is positive with the mean score of 4.94, but the intention to continue usage is relatively weak with the mean score of 4.42, which is close to the neutral value of 4.0.
The SEM path model analysis includes the measurement model and model fitting. The PLS measurement model includes item reliability (factor loading), composite reliability, and average variance extract and its square root. After excluding items with factor loadings below 0.7, a minimum hurdle suggested by Fornell and Larcker (1981), the new factor loadings were derived and shown in the fourth column. All values exceeded the value 0.7 as suggested by Nunnally (1978). The convergent
validity can be further examined to determine whether the average variance extracted (AVE) is greater than 0.5 (Fornell and Larcker, 1981). The sixth column shows that all AVE values are greater than 0.5, indicating good convergent validity.
Table 2 Descriptive statistics and reliability
Variables Mean/S.D. Items Factor
loading (>0.7) Composite reliability (>0.7) Average variance extracted (AVE) (>0.5) Technology-based challenges (TBC) 3.87/1.42 TBC1 0.83 0.89 0.74 TBC2 0.88 TBC3 0.87 Teacher-based challenges (IBC) 3.87/1.42 IBC1 deleted 0.77 0.63 IBC2 0.87 IBC3 0.71 Student-based challenges (SBC) 3.99/0.94 SBC1 0.78 0.89 0.67 SBC2 0.85 SBC 3 deleted SBC 4 0.90 SBC 5 0.71 SBC 6 deleted SBC 7 deleted Classroom environment benefits (CEB) 5.08/0.95 CEB1 deleted 0.87 0.68 CEB 2 0.83 CEB 3 deleted CEB 4 0.79 CEB 5 0.87 Learning benefits (LB) 5.06/1.02 LB1 0.74 0.89 0.63 LB2 0.75 LB3 0.80 LB4 0.82 LB5 0.84 Assessment benefits (AB) 5.33/1.00 AB1 0.89 0.88 0.71 AB2 0.87 AB3 0.76 Satisfaction (SAT) 4.94/1.20 SQ1 0.94 0.97 0.88 SQ2 0.95 SQ3 0.94 SQ4 0.92 Intention to continue usage (USE) 4.41/0.84 USE1 0.95 0.93 0.88 USE2 0.93 USE3 deleted
Discriminant validity can be determined by the square root of AVE being higher than the correlations with other constructs. In Table 3, the square roots of AVE in the diagonal are all higher than the construct-pair correlations, indicating adequate discriminant validity in all variables. The evidence suggests that the overall reliability and validity of the data set with seven items deleted are adequate.
Table 3 Discriminant validity analysis
Constructs Constructs Constructs
Constructs AVEAVEAVEAVE Correlation of ConstructsCorrelation of Constructs Correlation of ConstructsCorrelation of Constructs
(1) (2) (3) (4) (5) (6) (7) (8) TBC(1) .74 .861111 IBC(2) .63 .34 .79 SBC(3) .67 .34 .61 .82 CEB(4) .68 .11 .21 .35 .82 AB(5) .71 .14 .13 .32 .60 .88 LB(6) .63 .11 .16 .37 .71 .74 .79 SAT(7) .88 .27 .35 .55 .55 .50 .59 .94 USE(8) .88 .20 .26 .48 .50 .44 .52 .81 .94
1 Each of the diagonal AVE value is required to exceed the
construct AVE value, and other related coefficients of the non-diagonal construct values.
Results of hypotheses tests are shown in Table 4 and summarized as follows: classroom environment and learning benefits are significantly related to student satisfaction at 0.001 and 0.05 levels, respectively, while assessment benefits are not. In contrast, only student-based challenges from among the challenges are significantly related with satisfaction at 0.001 level. Teacher-based challenges are not significant at all, while technology-based challenges are weakly significant at 0.1 level. Satisfaction is significantly related with continuance of usage intention at 0.001 level. Accordingly, it can be concluded that H1 are partially but strongly supported, while H2 are partially
supported. H3 is fully supported by our data, which find similarity with many ECT
research studies.
Since the significance of H2-1 at 0.05 level borders on the significant, we believe that
moderating effects influencing the relationship between technology-based challenges and satisfaction exist. To explore this possibility, profile variables of sex, class performance, year level, IRS experience, and location are added as moderating variables on the relationship of technology-based challenges and satisfaction. The other two insignificant relationships of H1-3 and H2-2 are included as well. PLS
software can easily add a moderating variable to detect the significance of the interactions of the moderating variable and independent variable with the dependent variable (Chin, 2001).
Table 4 Integrated research model and results
Research hypothesis t value Results
H1 H1-1. Classroom environment benefits are significantly related
with satisfaction.
3.24*** Strong partially supported H1-2. Learning benefits are significantly related with
satisfaction.
4.07**
H1-3. Assessment benefits are significantly related with
satisfaction.
0.90
H2 H2-1. Technology-based challenges are significantly related
with satisfaction.
1.81 Partially
supported H2-2. Teacher-based challenges are significantly related with
the satisfaction.
0.72
H2-3. Student-based challenges are significantly related with
the satisfaction.
5.12***
H3 Satisfaction is significantly related with intention to continuance
usage. 28.04***
Fully supported *** Significant at 0.001 ** Significant at 0.01 * Significant at 0.05 Significant at 0.1
The moderating variables were added one by one to the path model. Location and year level have significant influences on H2-1 at 0.05 level with t-values equal to 2.49
and 2.14, respectively. Between them, location has t-values higher than 1 on H1-3 and
H2-2 (1.4 and 1.19, respectively), which indicates location is a stronger moderating
variable than year level.
The final path model is presented in Figure 2. The R square values are 0.51 for satisfaction and 0.66 for intention to continue usage, which are judged to be at high levels according to Cohen (1977) and thus, indicate the superior explanation power of the research model.
In principle, we assumed that benefits and challenges derived from literature should have significant impacts on the dependent variables in our research model. In other words, classroom environment benefits, learning benefits, and student-based challenges should significantly influence student satisfaction, which in turn influences students’ intention to continue usage. Results are promising in the sense that two out of three hypotheses regarding benefits, and only one hypothesis and another one with moderating effect regarding challenges are statistically significant. In other words, the students perceived more benefits than challenges with regard to their satisfaction.
Intention to continue usage R2=0.66 Satisfaction R2=0.51 Classroom environment Learning Assessment PRS benefits PRS challenges Technology-based challenges Teacher-based challenges Student-based challenges
Expectation Confirmation Model
3.39** 4.07*** 0.90 1.80+ 0.90 5.22*** Grade 27.65*** Location 2.14* 2.49* *** Significant at 0.001 ** Significant at 0.01 * Significant at 0.05
Figure 2 Integrated research model and results
5. Discussion
As regards the moderating effect on, H2-1, as mentioned above, it is a positive sign
because a certain percentage of Taiwanese college students were not influenced by technology-based challenges when using PRS. On the contrary, a number of students who were not satisfied with PRS when they encountered certain technology-based issues has been observed. This implies that teachers or the school may need to provide a more technology-friendly learning environment for students to maintain satisfaction levels. Similarly, the observation that student-based challenges significantly influence student satisfaction deserves more attention from teachers and school administration. This should guide them in increasing support and facilitating student adjustment on PRS learning approach.
Two insignificant hypotheses, H1-3 and H2-2, are related to benefits and challenges,
respectively. For the assessment benefits, our observations suggest that the teachers in Taiwan failed to facilitate PRS at a level that would have allowed them to obtain regular feedback from students. Moreover, Taiwanese students did not receive enough motivation to know peer responses. Since these three measurement items are positive benefits to the teacher-student dynamic relationship identified in Western literature, gaps in PRS implementations and promotion may exist in the Taiwan context. At present, almost all higher educational institutions in Taiwan have established teachers’ resource centers, but only a few have a full-scale PRS implementation. Majority of these institutions do not provide PRS-related support to their faculty. A quick solution
to the insignificant relationship between assessment benefits and satisfaction is to provide better training and consultation venues to teachers as well as promotional activities for students notwithstanding their scale of PRS implementations.
The teacher-based challenges may be due to the inability of students to experience or understand the challenges from the teacher’s perspective (IBC1 and IBC3). It may also be attributed to the fact that the teacher could have made known these complaints or revealed any of these challenges in class (IBC2). On the other hand, it is also reasonable to believe that these benefits may impact the teachers more than the students, and thus, student perceptions on this category of challenges may not be as critical compared with the other two categories of challenges toward their satisfaction on PRS usage. This, however, is a good indicator of serious troubles particularly in cases where teacher-based challenges were detected to have significant impacts on student satisfaction in practice.
6. Conclusions
The PRS benefits and challenges classified by Kay and LeSage (2009) are first empirically validated with student satisfaction and continuance of use intention in the Taiwan context. Four issues observed by this study related to Kay and LeSage (2009) were addressed.
Through an empirical study employing a systematic approach of conducting a survey of college students covering the entire region of Taiwan, this research validated the influences of PRS challenges and benefits. Year level and location are also discovered to be effective moderating variables on the relationship between technology-based challenges and student satisfaction. Results imply that teachers should be encouraged by the less number of influences from the PRS challenges on student satisfaction when adopting or continuing the use of PRS in classrooms. Nevertheless, more effort and attention from the teachers and school administration are the keys to influence student satisfaction on assessment benefits, technology-based challenges, and teacher-based challenges.
There is a significant number of in-depth PRS research suggestions in Kay and LeSage (2009) as well as from the other review papers reported in Kay and LeSage (2009) that can be empirically validated with existing theories as proven in this research. This could broaden the understanding and effective implementation of PRS using a more systematic approach.
Bhattacherjee, A. 2001. Understanding information systems continuance: An expectation-confirmation model, MIS Quarterly, 25(3), 351-370.
Caldwell, J. E. 2007. Clickers in the large classroom: Current research and best-practice tips, Life Science Education, 6(1), 9-20.
Chin, W.W. (2001). PLS-Graph User’s Guide, Version 3.0, Soft Modeling Inc.
Cohen, J.. Statistical Power Analysis for the Behavioral Sciences. (Revised Edition). New York, Academic Press, 1977.
DeLone, W. H. and McLean, E. R. 2003. The DeLone and McLean model of information systems success: A ten-year update, Journal of Management Information Systems, 19(4), 9-30.
Fies, C and Marshall, J. 2006. Classroom response systems: A review of the literature, Journal of Science Education and Technology, 15(1), 101-109.
Judson. E. and Sawada, D. 2002. Learning from past and present: Electronic response systems in college lecture, Journal of Computers in Mathematics and Science Teaching, 21(2), 167-181.
Kay, R. H. and LeSage, A. 2009. Examining the benefits and challenges of using audience response systems: A review of the literature. Computers & Education, 53, 819-827.
Ringle, C. M., Wende, S. and Will, A. 2005. SmartPLS 2.0 (beta), www.smartpls.de. Simpson, V. and Oliver, M. 2007. Electronic voting systems for lectures then and now:
A comparison of research and practice, Australasian Journal of Educational Technology, 23(2), 187-208.
Appendix: Questionnaire Items and Sources
CEB - Classroom environment benefits (Kay and LeSage, 2009) Using PRS enables students go to class more.
Using PRS enables students to be more focused in class. Using PRS enables students to participate anonymously.
Using PRS enables students to participate with peers more in class to solve problems. Using PRS enables students to be more engaged in class.
LB - Learning benefits (Kay and LeSage, 2009)
Using PRS enables students to interact more with peers to discuss ideas.
Using PRS enables students to actively discuss misconceptions to build knowledge. Using PRS enables the instruction to be modified based feedback from students. Using PRS enables the learning performance to increase.
Using PRS enables qualitative differences in learning. AB - Assessment benefits (Kay and LeSage, 2009)
Using PRS enables the assessment to be done to improve student understanding and quality of teaching.
Using PRS enables students to compare their own response to class response. TBC - Technology-based challenges (Kay and LeSage, 2009 or this research)
Students forgot or lost remotes and could not participate in class, if this question applies. Or The remote or system setup was not ready on time and caused the class delay its progress onsite.
Remote devices did not function properly. The overall system did not function properly. IBC - Teacher-based challenges (Kay and LeSage, 2009)
I think less experienced teachers cannot adjust to student feedback.
I think using PRS often make the teachers could not cover the course content. I think giving good PRS questions will cost a lot of teacher’s time.
SBC - Student-based challenges (Kay and LeSage, 2009) Students find it difficult to shift to a new way of learning. Discussion leads to confusion or wasting time.
Too much effort is required by students when using ARSs. Using ARS for tests may not be popular with students. Students do not like ARS used for monitoring attendance. Students want to remain anonymous.
Students feel bad when receiving negative feedback. SAT - Satisfaction (Bhattacherjee, 2001)
My overall experience of PRS is very satisfied.。 My overall experience of PRS is very pleased. My overall experience of PRS is very contented. My overall experience of PRS is absolutely delighted USE - Intention to continuance usage (Bhattacherjee, 2001)
I want to continue using PRS in future classes rather than discontinue its use.
My intentions are to continue using PRS rather than any alternative means in future classes. If I could, I would like to discontinue use of PRS in future classes.
