探討教師科技導入評量的信念、評量的實務、與學生表現間關聯性的系列研究

全文

(1)國立臺灣師範大學科學教育研究所博士論文. 探討教師科技導入評量的信念、評量的實務、與學生表現間關聯性的系列研究 Three Studies of the Relationships among Science Teachers’ Beliefs about, Practice on and their Students’ Performances on a Technology-based Assessment.. 指導教授：吳心楷教授. 博士生：簡頌沛國立臺灣師範大學科學教育研究所. 中華民國一百零八年一月.

(2) 誌謝一瓶紅酒從葡萄的品種、土壤與水文等風土特徵、乃至於釀酒人的獨特堅持與理念，都會影響著最後品嘗到的風味。當然，時間也是。而清酒的釀製除了上述要素之外，還強調採收的新米必須層層磨去外殼，只保留米心用以釀酒，唯有如此，堪稱佳釀。本篇論文雖說不屬長篇鉅著，但論文的撰寫，甚或是研究者本人的成長與智識累積，其實也正如紅酒與清酒的釀製過程。值此畢業之際，更是讓人想起過程中的淬鍊與幫助我的良師益友。首先要感謝的是，指導教授吳心楷老師，小至引文格式、大到研究方向，總是不厭其煩地提醒與叮嚀。正如堅持品質的釀酒人。其中最讓我感佩的是，老師並不會強加要求旗下子弟兵的論文題目或研究方法都要一樣，反而是極大地鼓勵我們發揮自己原有的特色與風味。也因此，本論文的研究與寫作過程，雖然如同清酒釀製一般，層層磨去外殼的粗糠，但最核心的研究主張反而更能聚焦。此外，科教所的教授們則如同養分充足、排水良好的沃土一般，給予我許多做為科學教育研究者必備的研究技能與素養。不論是在楊文金老師課堂中所思辯的科學哲學，張文華老師的課程中所訓練的課程設計原則，抑或是許瑛玿老師在頂大計畫中，時常帶領成員腦力激盪如何設計研究與分析研究結果，上述都是我在科教所期間所習得的無價資產。當然，林陳涌老師常說的：「學問慢慢做，學位快快拿。」在幾次考慮休學與中輟的期間，更是我用來鼓勵自己的名言。再來想感謝的是，漫長過程中一路相伴的好同學與好朋友們。如切如磋、如琢如磨、互相漏氣求進步，不只豐富了求學過程、也豐富了我的人生。而交往時間幾乎跟我求學年限一樣久的女朋友，更是做為第一號讀者，給予我無後顧之憂的支持與包容。最後想跟天上的老爸與家中的老媽說一聲，謝謝你們。如果今日的我有任何值得讚許之處，都要感謝你們的良好模範。.

(3) Abstract This series of studies aimed at not only investigating the components and features of teachers’ beliefs about technology-based assessments (TBAs), but also revealing the relationship between teachers’ beliefs about TBAs and their assessment practices and estimating the effects of teachers’ practice and beliefs about TBAs on students’ performances. On the basis of these purposes, three studies that combined the qualitative data analysis, a structural equation modeling (SEM) analysis that combines both confirmatory factor analysis (CFA) and path analysis, and the two-level hierarchical linear modeling (HLM) were conducted. In the study 1, the main components and features of teachers’ beliefs and practice about TBAs were investigated and analyzed based on the qualitative data from 40 technology-experienced science teachers. In the study 2, I developed a specific questionnaire from the result of the study 1 to elicit another 494 high school science teachers’ beliefs about TBAs and conducted a confirmatory factor analysis to ensure unobserved beliefs could be identified by the questionnaire to take the study 1 a step further. Besides, the study 2 specified the possible relationships among teachers’ beliefs about, attitudes toward, and intention to use TBAs by conducting a SEM analysis. Finally, a HLM analysis was conducted to estimate the effects of teachers’ practice and beliefs about TBAs on students’ performances. The same 494 science teachers in the study 2 and their 1,774 students from eighth and 11th grades from 32 secondary schools participated in the study 3. Based on the definitions of teachers’ beliefs systems and the literature review, this series of studies adapted both the elements and hierarchical structure of the technology acceptance model (TAM) and decomposed theory of planned behavior (DTPB) to form the coding scheme of the study 1 to explore the substantial components of science.

(4) teachers’ TBAs beliefs and portraying the features of science teachers’ TBAs beliefs as comprehensive as possible. And then I modified TAM and DTPB based on the result of the study1 to form a prosed model to be examined in the study 2. Finally, on the basis of the study 2, I conducted a hierarchical linear modeling analysis to investigate the relationships among teachers’ beliefs about TBAs, their usage of TBAs, and their students’ performances on a TBA. The analysis of qualitative data in the study 1 showed that 10 components were substantial in the behavioral, control, and normative beliefs. While 34 teachers perceived TBAs as useful tools and identified a variety of usefulness, nearly 40% of the participants indicated the difficulties in using TBAs and their beliefs of ease of use were mainly negative. Also, teachers’ control beliefs about TBA focused on the external components such as time, supporting personnel, and infrastructure rather than the personal factors. In their normative beliefs, teachers tended to view school policies and parents’ opinions as constraints, whereas they also realized the benefits of using TBAs for learning. Furthermore, based on their usage of TBAs, teachers were identified and characterized as three groups: frequent, occasional, and non-users. Although some frequent users did not teach in resource-rich schools and faced constraints similar to those encountered by the occasional users, they seemed to actively look for more supports and solutions to overcome the lack of resources and the disapproval from the school administration. To take the study 1 a step further, the questionnaire in the study 2 was developed from the result of the study 1 to elicit the teachers’ beliefs about TBAs and then I conducted a CFA to ensure unobserved beliefs could be identified by the questionnaire. The results of CFA showed that all of the items developed in the study 2 were validated as being adequate indicators for measuring teachers’ beliefs about TBAs. Furthermore, the results of SEM analysis suggested that with the exception of teachers who had never.

(5) used TBAs previously, teachers’ beliefs about usefulness, ease of use, and compatibility were significant predictors of attitude, which could explain the intention of teachers to use TBAs. However, perceived behavior control and subjective norms beliefs did not influence teachers’ intention. On the basis of the study 2, I utilized a HLM technique to investigate the relationships between science teachers’ beliefs about TBAs, their usage of TBAs, and their students’ performances on a TBA in the study 3. The results of the study 3 showed that there was significant variation between schools in terms of student performances. The results also showed that both of the variables at the student level, such as students’ inquiry‐related laboratory engagement and their PCs experience had significant positive effects on their learning performances. However, none of the variables at the school level, such as teachers’ TBA hours and teachers’ intentions to use TBA, had significant effects on students’ learning performances. Finally, the relationship between students’ inquiry‐related laboratory engagement and their learning performances could be moderated by two different variables at the school level in different ways. In one way, Average PCs usage at the school level would positively moderate the relationship between students’ inquiry‐related laboratory engagement and their learning performances. In another way, the TBA hours at the school level would negatively moderate the relationship between students’ inquiry‐related laboratory engagement and their learning performances. The results highlight the role of teachers’ practice and its impact on students’ performances..

(6) 摘要為了因應強調培養學生推理與問題解決能力的教育改革，現今的評量工具應讓教師得以有效且穩定地測得學生的這些能力，進而設計出幫助學生培養上述能力的課程活動。誠然，科技的進步已幫助教師測量與評估許多過去無法測得的學生能力，諸如數學能力、科學概念，甚或是學生於每一個學習環節所使用的解題策略。但是，值得注意的是，不論是學校的科技資源，用於設計與執行評量的教學時數，以及教師如何執行評量，評量本身與實際教學的一致性都有可能影響科技融入評量能否在學校中順利實施。而不論是教師如何執行評量與評估評量的成效或是教師對學校資源的覺察，又都與教師信念有關。因此本系列研究透過 3 個子研究力求探索台灣科學教師對於科技融入評量的信念成分為何，驗證其信念與實務之間的關係，並評估教師對於科技融入評量的信念與實務如何調節學生的學習成效。在本系列研究的執行過程中，教師信念的相關文獻與理論如：technology acceptance model (TAM) 與 decomposed theory of planned behavior (DTPB) 則構成了貫串全文的編碼架構與理論模型。首先，在子研究 1 當中，藉由 40 位專家教師的晤談與編碼分析，我找出了 10 項主要的科技融入評量的信念成分及其特徵。此外，受訪的教師亦依據其實際使用科技融入評量的情形而被分為經常使用者、偶爾使用者、與未使用者三類。交叉比對這些老師的信念與實務則發現到，不同使用程度使用者所持有的信念亦有所不同。例如，隨著使用頻率的提高，教師所持的正向信念亦會越高；然而負面信念卻是由偶爾使用者拔得頭籌。接著，為了進一步檢驗子研究 1 所發現的 10 項信念成分是否真的是組成教師科技融入評量信念的成分，我將子研究 1 的 10 項信念成分改寫為子研究 2 的信念問卷，並邀請 494 台灣的高中科學教師填寫，並藉由因素分析的結果確認此分問卷具備足夠的信效度來測量教師的科技融入評量信念。最後藉由結構方程式.

(7) 分析 (structural equation modeling) 所提供的路徑分析，我發現了除了從未使用者之外，其餘教師的有效性信念、方便性信念、相容性信念都會顯著且正向地影響其對科技評量的態度，而態度又能決定其使用科技評量的傾向。在此系列的尾聲，為了評估教師對於科技融入評量的信念與實務如何調節學生的學習成效，子研究 3 採用 2 階層的階層線性模式的分析 (hierarchical linear modeling)，除了從學生個體層次探討其課堂參與、電腦使用經驗如何影響其學習成效，亦由學校層次探討屬於該校特色的教師科技評量使用時數、教師科技評量使用傾向、學校平均電腦使用經驗如何影響學生學習成效，更呈現學校層次的變數如何調節學生層次變數與其學習成效的關係。結果顯示，學生層次變數均對其學習成效有顯著正向影響，學校層級變數則對其學習成效有調節作用卻無直接影響。. keywords：teachers’ beliefs about technology-based assessments, teachers’ usage of technology-based assessments, students’ performances, decomposed theory of planned behavior, structural equation modeling (SEM), hierarchical linear modeling (HLM).

(8) Contents 1. 2. 3.. Introduction .......................................................................................................... 5 Research Purposes of This Series of Studies ..................................................... 8 Theoretical Background ...................................................................................... 9 3.1 The definition of teachers’ beliefs .................................................................... 9 3.2 Teachers’ beliefs about assessment ................................................................ 10 3.3 Models of teachers’ beliefs about educational technology ............................ 12 3.3.1 The technology acceptance model (TAM) .......................................... 12 3.3.2 Decomposed theory of planned behavior (DTPB).............................. 13 3.3.3 Development of a guiding framework in this series of studies ........... 14 3.4 The definitions of teachers’ practice, assessment practice, and assessment practice about TBAs ............................................................................................ 15 3.5 Factors that might affect students’ performances on a TBA .......................... 16. 4.. Methods ............................................................................................................... 17 4.1 Research design of this series of studies ........................................................ 17. 5. An Investigation of Teachers’ Beliefs and Their Use of Technology-based Assessments (The study 1) ......................................................................................... 20 5.1 Purpose and research questions ..................................................................... 20 5.2 Research design ............................................................................................. 20 5.3 Participants ..................................................................................................... 21 5.4 Development of the semi-structured interview .............................................. 22 5.5 Development of the coding scheme ............................................................... 23 5.6 Procedure of data analysis ............................................................................. 28 5.7 Findings.......................................................................................................... 28 5.7.1 Overview of teachers’ beliefs .............................................................. 28 5.7.2 Teachers’ behavioral beliefs about TBAs ........................................... 30 5.7.3 Teachers’ control beliefs about TBAs ................................................. 35 5.7.4 Teachers’ normative beliefs about TBAs ............................................ 38 5.7.5 Teachers’ usage of TBAs .................................................................... 40 5.7.6 The Interaction between Teachers’ Degrees of Usage and Their Beliefs ...................................................................................................................... 43 5.8 Conclusions .................................................................................................... 48 6. Teachers’ Beliefs about, Attitudes toward, and Intention to Use Technology-Based Assessments: a Structural Equation Modeling Approach (The Study2) ........................................................................................................................ 51 6.1 Purpose and research questions ..................................................................... 51 6.2 Design of the instrument ................................................................................ 52 1.

(9) 6.3 Participants and data collection ..................................................................... 54 6.4 Data analysis .................................................................................................. 55 6.5 Results ............................................................................................................ 56 6.5.1 Evaluation of the measurement of the proposed model ...................... 56 6.5.2 Goodness of fit of the Proposed Model .............................................. 59 6.5.3 Path analysis of the models ................................................................. 60 6.5.4 Assessment of direct and indirect effects ............................................ 67 6.6 Conclusions and discussion ........................................................................... 68 7.. Examining the impacts of science teachers’ practice and beliefs about. technology-based assessments on students’ performances: A hierarchical linear modeling approach. (The Study 3) ........................................................................... 72 7.1 Purpose and research hypotheses ................................................................... 72 7.2 Participants and data collection ..................................................................... 74 7.3 Design of the instrument and measurements of variables ............................. 75 7.4 Data analysis .................................................................................................. 77 7.5 Results ............................................................................................................ 77 7.5.1 The establishment and testing of the null model ................................ 77 7.5.2 The establishment and testing of the student-level model (Random intercept model Lv1) .................................................................................... 79 7.5.3 The establishment and test of the student-level + school level model (Random intercept model Lv1+2) ................................................................ 80 7.5.4 The establishment and testing of the full model ................................. 82 7.6 Conclusions and discussion ........................................................................... 84 8. Conclusions of This Series of Studies ............................................................... 90 9. Implications for the Models of teachers’ technology beliefs .......................... 94 10. Implications for Teacher Education and Students’ Learning ........................ 96 References ................................................................................................................. 101 Appendix 1 Standardized Factor Loadings and the Normality of Observed Items 108 Appendix 2 The Questionnaire about teachers’ beliefs about TBAs ................... 109. 2.

(10) List of Tables. Table 1 Backgrounds of the Participating Teachers ..................................... 22 Table 2 Coding Scheme and Coding Results of Teachers’ Beliefs about TBAs .................................................................................................... 25 Table 3 Coding Scheme and Coding Results of the Degrees of Use ........... 27 Table 4 Beliefs of Perceived Usefulness and Types of TBAs Teachers’ Used (Numbers of Responses) ...................................................................... 33 Table 5 Teachers’ Usage in Different Types of TBAs (Numbers of Teachers) .............................................................................................................. 41 Table 6 Latent Variables and Their Observed Items .................................... 53 Table 7 Reliability and Convergent Validity of the Measurement Model ... 57 Table 8 Discriminant Validity of the Measurement Model .......................... 58 Table 9 Model Fit Indices of the Proposed Models ..................................... 60 Table 10 Summary of Results of Path Analysis for Frequent Users ............ 62 Table 11 Summary of Results of Path Analysis for Occasional Users ........ 64 Table 12 Summary of Results of Path Analysis for Nonusers ..................... 66 Table 13 Direct and Indirect Effects of the Exogenous Latent Variables .... 68 Table 14 The Variance, ICC value, and Gap value of Models ..................... 79 Table 15 The Fixed Effects of Student-level Model .................................... 80 Table 16 The Fixed Effects of Student-level + School level Model ............ 81 Table 17 The Fixed Effects of full Model .................................................... 84 Table 18 Means and SD of usage between different TBAs ......................... 87 Table 19 Differences of usage between four types of TBAs ....................... 89. 3.

(11) List of Figures. Figure 1. The main constructs and the design of this series of studies. ......... 9 Figure 2. Three models of teachers’ beliefs and technology usage. ............ 14 Figure 3. The interaction of teachers’ beliefs and the degrees of use. ......... 44 Figure 4. The model proposed in the study 2. .............................................. 52 Figure 5. The structural equation model for frequent users (n=105). .......... 61 Figure 6. The structural equation model for occasional users (n=211)........ 63 Figure 7. The structural equation model for nonusers (n=178). .................. 65 Figure 8. A two-level model of the impacts of science teachers’ practice and beliefs on students’ performances. ....................................................... 74 Figure 9. A two-level models of teachers’ beliefs about educational technology. ........................................................................................... 95. 4.

(12) 1. Introduction In order to echo the educational reform that is focused on improving students’ abilities of reasoning and problem solving, the assessment tools nowadays should enable teachers to evaluate these abilities of their students (Pellegrino, 2001). At the same time, during the whole process of assessment, teachers not only need to observe and record their students’ behavior when performing specific tasks, but also need to evaluate their students’ abilities on the basis of their performance of these tasks. Of course, the justification for doing the evaluation should be the theory of students’ cognition and learning (Pellegrino, 2001). However, the problem is that even though teachers can evaluate students’ performance on the basis of the learning theory, it is almost impossible for any teacher to record students’ every single action and manage the longitudinal tracking data of students without the help of technology tools (Brown, Hinze, & Pellegrino, 2008). It is worth noting that the improvements in information and communications technology in education make both measuring and evaluating students’ performance easier and more reliable (Brown et al., 2008). Firstly, technology-based assessments (TBAs) can help teachers to collect students’ data or observable performance more easily by decreasing the time required to collect and input test data (Brown et al., 2008; Kuo & Wu, 2013; Pellegrino, 2001). The observable performance includes students’ interactions with the simulation tasks during their processes of problem solving and the submitted answers as final solutions to the problem (De Klerk, Veldkamp, & Eggen, 2015). Secondly, teachers can diagnose students’ learning problems by retrieving and comparing the process data of the target performance (Brown et al., 2008). All of these potentials of TBAs make the measurement of complex abilities or skills possible and reliable. 5.

(13) Furthermore, an increasing number of TBAs have been adopted by large-scale and international evaluation programs in recent years (Quellmalz, Timms, Silberglitt, & Buckley, 2012), such as the Programme for International Student Assessment. This means that the study of TBAs has become an important issue for both researchers and teachers. However, the infrastructure needed, the preparation time requirements (Cheon, Lee, Crooks, & Song, 2012; Pellegrino, 2001) and especially teachers’ beliefs about assessment could all have an influence on how TBAs are implemented (Black & Wiliam, 1998; Cheon et al., 2012). Generally, previous educational studies have articulated that teachers make decisions on the basis of their beliefs (Hart, 2002; Pajares, 1992) and their practice is guided by their beliefs about teaching and learning (van der Schaaf, Stokking, & Verloop, 2008). Similarly, other studies have also indicated that teachers’ beliefs impact on their acceptance and implementation of innovations in curricula, especially on their use of new technology tools (De Smet, Bourgonjon, De Wever, Schellens, & Valcke, 2012; Pynoo et al., 2011; Teo, 2011). Furthermore, some studies have contributed to portraying the relationships between teachers’ beliefs about TBAs and their assessment practice (Cheon et al., 2012; Graham, 2005; Stiggins, 2004). On the one hand, Lee, Feldman, and Beatty (2012) found that teachers without the necessary resources and time tended not to try to implement TBAs. Teachers with enough resources but who did not believe in the usefulness of TBAs would also not integrate TBAs into their assessment practice. Only those teachers who could both perceive the usefulness of TBAs and find the way to overcome the problem of resources would actively use TBAs in their classrooms. On the other hand, Stiggins (2004) asserted that the passive beliefs held by teachers may constrain their involvement in the use of TBAs. 6.

(14) It is worth noting that Stiggins’ (2004) assertion was based on his review, and the findings of Graham (2005) and Lee et al. (2012) were based on their coding and interpretation of the qualitative data. Even though these studies revealed that teachers’ beliefs about TBAs might relate to their assessment practice, and articulated some features of teachers’ beliefs about TBAs, they did not articulate the causal relationship between teachers’ beliefs about TBAs and their assessment practice. In other words, if researchers want to not only portray the features of teachers’ beliefs about TBAs, but also test the relationship between teachers’ beliefs about TBAs and their assessment practices, and identify the mechanism of how they interact with each other, a study combining qualitative data analysis, confirmatory factor analysis (CFA) , regression analysis, and path analysis is needed. At the same time, the studies nowadays have also 1. Teachers’ beliefs impact on their acceptance and implementation of innovations in curricula, especially on their use of new technology tools (Churchill, 2006; Kim, Kim, Lee, Spector, & DeMeester, 2013; Shin, 2015; Stylianidou, Boohan, & Ogborn, 2005; Zacharia, 2003). 2. Several theoretical models such as the technology acceptance model (TAM) and the theory of planned behavior (TPB) have been utilized to analyze and estimate the effects of beliefs on how teachers adopt technology for use in instruction (Ajzen, 1991; Davis & Venkatesh, 1996; Liu, 2012; Taylor & Todd, 1995; Teo, 2011). 3. Some studies have indicated that teachers’ technology-infused practice has an effect on their students’ learning performances (Comi, Argentin, Gui, Origo, & Pagani, 2017; McNeill, Pimentel, & Strauss, 2013).. 7.

(15) 4. However, compared to the substantial amount of research on how teachers use technology in general, there has been relatively little research into how teachers use technology-based assessments in their teaching. To summarize the above considerations, compared to the substantial amount of research on how teachers use technology in general, there has been relatively little research into teachers’ beliefs about TBAs and their usage about TBAs. Thus, this series of studies aimed at filling this gap and the purpose of each study was listed as below. Besides, a mixed method design combining qualitative data analysis, a structural equation modeling (SEM) analysis that combines both CFA and path analysis and the hierarchical linear modeling (HLM) was conducted in this series of studies to achieve these aims. More details about the research design will be introduced in the section of methods. 2. Research Purposes of This Series of Studies According to the introduction above, I conducted a series of three studies to fill the gap with the aim of investigating the relationships between science teachers’ beliefs about TBAs, their usage of TBAs, and their students’ performances. The main constructs and the design of this series of studies are shown in the Figure 1. The following research purposes guided the studies: 1. Study 1 focused on exploring the substantial components of science teachers’ TBA beliefs and portraying the features of these beliefs. 2. Study 2 aimed at revealing the relationships among science teachers’ beliefs about, attitudes toward, and intention to use TBAs, and to compare differences in the relationships among teachers with different degrees of TBA usage. 3. Study 3 was designed to investigate the relations between science teachers’ beliefs about TBAs, their usage of TBAs, and their students’ performances on a TBA. 8.

(16) Figure 1. The main constructs and the design of this series of studies.. Since teachers’ beliefs and practice about TBAs was the cores of this series of studies, in the next section of theoretical background, the definition of teachers’ beliefs and practice will be introduced first. 3. Theoretical Background 3.1 The definition of teachers’ beliefs Teachers’ beliefs can be viewed as the assumptions that they hold about teaching and learning (Pajares, 1992). Rather than having single and disconnected beliefs, teachers develop a belief system that houses relevant beliefs. This kind of personal belief systems develop based on experience, and these systems generally contain multiple, non-independent, interrelated beliefs rather than having single and disconnected ones (Rokeach, 1968). Previous studies have shown that the different beliefs held by teachers interact with each other (De Smet et al., 2012; Teo, 2011) or 9.

(17) combine with other beliefs (van Driel, Bulte, & Verloop, 2005). In addition, there is a complicated relationship between teachers’ beliefs, practices and the mechanism of how the beliefs interact with practices (Graham, 2005). Some studies have asserted that teachers can interpret the events they encounter and make decisions about teaching based on their belief systems (Hart, 2002; Pajares, 1992; Quellmalz et al., 2012), so that their behaviors and practices may be guided by their own specific beliefs about teaching (van der Schaaf et al., 2008). However, other studies showed that positive beliefs or attitudes provide no guarantee that innovative practices will be implemented (Pintó, 2005; Viennot, Chauvet, Colin, & Rebmann, 2005). More research is therefore needed to reveal the complexity of teachers’ beliefs systems and the interactions between beliefs and practices. This is the reason why I conducted this series of studies. 3.2 Teachers’ beliefs about assessment Previous studies have shown that teachers’ beliefs about assessment could have an influence on how the assessment is implemented (Black & Wiliam, 1998; Cheon et al., 2012; Lyon, 2011). Furthermore, Ramesal (2011) indicated that teachers’ belief systems about assessments might be composed of different, and sometimes even contrasting, beliefs about teaching, learning, and their perceived approval and disapproval from important others. This is the reason why Ramesal (2011) articulated that teachers’ belief systems about assessments should be analyzed on the continuous spectrum, varying from the societal (external) side to the pedagogical (personal) side. The components of teachers’ belief systems about assessments were not only proposed by Ramesal (2011), but could also be seen in other research. On one hand, some studies asserted that from the point of view of the purpose of assessments, teachers’ beliefs about assessment should be considered from the following aspects 10.

(18) and elements (Lyon, 2011; Pellegrino, 2001). Firstly, the element of cognition is teachers’ belief about students’ learning and the way that students represent their knowledge. Secondly, the element of observation refers to teachers’ belief about what teachers should observe and collect as the evidence to examine the process and results of students’ learning. Thirdly, the element of interpretation is teachers’ belief about how to interpret their students’ learning performance (Lyon, 2011; Pellegrino, 2001). On the other hand, studies about teachers’ assessment literacy indicated that it can be viewed as part of the pedagogical content knowledge and includes teachers’ knowledge and perspectives on assessment (Wang, Wang, & Huang, 2008), such as knowledge about testing data (e.g., average and standard deviation), knowledge about constructing and administering tests (e.g., general steps of administering tests), and knowledge about purposes of assessments (e.g., discriminating students’ learning effects). Ogan-Bekiroglu (2009) further identified difficulties in implementing alternative assessments, and reported factors that affect teachers’ attitudes toward assessment, such as time management and inadequate training. Generally speaking, previous studies about teachers’ assessment beliefs or assessment literacy reveal many important features and factors of teachers’ belief systems about assessment. However, none of these studies systematically examined teachers’ beliefs about assessment or examined the possible relationships between these beliefs and their implementation of assessment. At the same time, another branch of studies has focused on teachers’ beliefs about educational technology, and has established and validated some models to explain the relationships between teachers’ beliefs about educational technology and their implementation of educational technology. Hence, in the next section I will introduce two widely used and powerful hierarchical models of teachers’ beliefs about educational technology first and then adapt these two models to propose a systematic model of teachers’ 11.

(19) beliefs about assessment, and especially their beliefs about TBAs. 3.3 Models of teachers’ beliefs about educational technology According to the definition of teachers’ beliefs and the interrelated nature of beliefs in teachers’ belief systems, the analysis should be based on a systematic model. Below I review two powerful hierarchical models that have been validated and used to portray belief systems. I then present the novel model proposed and examined in this series of studies. 3.3.1 The technology acceptance model (TAM) The technology acceptance model has been one of the most popular models used to explain how two beliefs of users—perceived usefulness and perceived ease of use—affect their behavioral intention, which in turn determines the actual use of technology (Davis & Venkatesh, 1996). Perceived usefulness is the belief that the use of new technology could increase the user’s job performance (Davis, 1989), while perceived ease of use refers to the belief that the use of new technology will be free of effort (Davis, 1989). Research has also shown that the perceived ease of use affects the perceived usefulness (Davis, 1993; Davis & Venkatesh, 1996). In Davis (1993) the attitude was added as an mediator between two beliefs and intention. While the technology acceptance model has been validated and adopted widely in the fields of psychology and education, limitations of this model have also been reported. Taylor and Todd (1995) argued that because the technology acceptance model contains only two personal beliefs about the technology itself, it excludes other important external factors such as the available resources and approvals from colleagues. Thus, the technology acceptance model might not be sufficiently comprehensive to capture the effects of external variables or social factors on the actual use of technology (Smarkola, 2008; Taylor & Todd, 1995). Additionally, the perceived 12.

(20) usefulness and ease of use may vary along with the context of the technology use (Cheon et al., 2012), but the context (e.g., voluntary settings and mandatory settings) has usually been neglected in studies that have employed the technology acceptance model (Benbasat & Barki, 2007). Some of the studies performed in the past decade have elaborated the technology acceptance model by adding new variables or new paths to the model in accordance with the specific context of technology use. For example, Sanchez and Hueros (2010) integrated the variable of technical support into their technology acceptance model, because the Moodle course management system that they used may require supporting personnel to maintain a website or train the users. In other studies, in addition to the perceived usefulness and ease of use, the facilitating conditions that would significantly influence teachers’ usage of specific technology tools were also identified, such as the technical support (Pynoo et al., 2011; Sanchez & Hueros, 2010), environment settings (Chen, 2010), and expectations of superiors (Pynoo et al., 2011; Teo, 2011). 3.3.2 Decomposed theory of planned behavior (DTPB) As an extension of the theory of planned behavior (TPB), the decomposed theory of planned behavior (DTPB) incorporates both personal and external factors (Figure 2). The attitudes of users are decomposed into three behavioral beliefs in the decomposed theory of planned behavior: perceived usefulness, compatibility, and perceived ease of use (Kriek & Stols, 2010; Taylor & Todd, 1995). The self-efficacy, technology-facilitating conditions, and resource-facilitating conditions are control beliefs that affect perceived behavior control (PBC). The subjective norms (SN) are determined by normative beliefs, and include influences from peers and superiors (Taylor & Todd, 1995). The decomposed theory of planned behavior allows researchers to identify various external conditions and personal beliefs when teachers need to make decisions about technology use (Smarkola, 2008). Moreover, the theory suggests that 13.

(21) the intention of usage is not influenced by only a few disconnected beliefs, instead being affected by many antecedents and interactions between the beliefs (Benbasat & Barki, 2007; Cheon et al., 2012).. Figure 2. Three models of teachers’ beliefs and technology usage.. 3.3.3 Development of a guiding framework in this series of studies Based on the above-mentioned review and the basic definitions of teachers’ beliefs systems, both the elements and hierarchical structure of TAM and DTPB were adapted to form the coding scheme of Study 1 (Table 2 and 3) to explore the substantial components of science teachers’ TBA beliefs and to portray the features of their beliefs as comprehensively as possible. I then modified TAM and DTPB based on the results of Study 1 as a proposed model to be examined in Study 2 (Figure 4). The proposed model has the structure of the decomposed theory of planned behavior, integrates internal and external factors, and includes the direct and indirect effects of the perceived ease of use 14.

(22) on attitudes, as suggested by the TAM. It is worth noting that the definition of attitude in this study is teachers’ feelings about using TBAs. Additionally, the definition of intention in this study is teachers’ willingness to use TBAs. This comprehensive model facilitated my analysis of the relationships among teachers’ beliefs about and intention to use TBAs. Finally, on the basis of Study 2, I conducted a hierarchical linear modeling analysis to investigate the relationship between teachers’ beliefs about TBAs, their usage of TBAs, and their students’ performances on a TBA. 3.4 The definitions of teachers’ practice, assessment practice, and assessment practice about TBAs Generally, teachers’ practice includes their preparation of classroom material (Ottenbreit-Leftwich, Glazewski, Newby, & Ertmer, 2010), their strategies used to improve the classroom operation, their implementation and enactment of specific teaching material, and their assessment of their students (Ertmer, Ottenbreit-Leftwich, Sadik, Sendurur, & Sendurur, 2012). Assessment practice can be divided into three elements according to Lyon’s (2011) framework, namely the elements of cognition, observation, and interpretation. Firstly, the element of cognition is teachers’ assessment design which was built on the basis of their beliefs about students’ learning. Secondly, the element of observation refers to what kind of students’ behavior teachers observe, what kind of students’ artifacts they collect as evidence, and what kinds of tools they use to measure the process and results of students’ learning. Thirdly, the element of interpretation is about how teachers actually interpret their students’ learning performance (Lyon, 2011; Pellegrino, 2001). This series of studies was designed to investigate the relationship between teachers’ beliefs about TBAs and their implementation of TBAs. In other words, the 15.

(23) focus was on what kinds of assessment tools teachers used to measure their students’ learning performance. Thus, participating teachers’ average hours of using different kinds of TBAs was defined as the main factor of teachers’ TBA practice in these studies. 3.5 Factors that might affect students’ performances on a TBA As a part of teachers’ teaching practice, the implement of specific curriculum could affect students’ learning performance in different ways (McNeill et al., 2013). The relationship between teachers’ technology-infused practice and students’ learning performance could also be found in previous study which indicated that teachers’ technology-infused practice could increase their students’ learning performances (Comi et al., 2017). The literature in the above sections also supported that the teachers’ multiple beliefs about technology and accessible resources could affect their technology-infused practice (Davis & Venkatesh, 1996; Nikou & Economides, 2017; Sánchez-Prieto, Olmos-Migueláñez, & García-Peñalvo, 2017; Teo, 2010). However, the way of how teachers’ beliefs and practice influence students’ learning performances is rarely studied. Thus, my study 3 was designed to fill this gap by using a hierarchical linear modeling technique and aims at investigating the relations between science teachers’ beliefs about TBAs, their usage of TBAs, and students’ performances on a TBA. Before conducting the hierarchical linear modeling analysis, the complicated beliefs in the teachers’ belief systems have to be aggregated. Since all the beliefs that teachers held might affect their practice according to the review in the above sections, it was difficult to deal with all the effects of each belief on the practice at the same time. Turner, Kitchenham, Brereton, Charters, and Budgen (2010) reviewed and compared 73 empirical studies and indicated that among all the beliefs and variables 16.

(24) that might affect teachers’ practice, teachers’ intention to use the technology as an aggregation of beliefs was the most important factor to predict teachers’ practical usage of technology tools. Thus, I used participating teachers’ intention to use TBAs as the main factor of teachers’ beliefs on the basis of the result of Turner et al. (2010) in the study 3. Overall, this series of studies aims at analyzing the possible effect that teachers’ TBAs practice and their intention to use TBAs might have on their students’ learning. 4. Methods 4.1 Research design of this series of studies On the basis of purposes 1 and 2 of this series of studies, the measurement of teachers’ beliefs and their usage of TBAs were the first things that needed to be dealt with. Only when the measurements of these constructs were both reliable and adequate could the analysis of the relationships between them be convincing. However, previous studies on the measurements of teachers’ beliefs showed two different approaches. On the one hand, some studies have argued that teachers’ beliefs are highly personal and context-related, and these beliefs exist implicitly as tacit knowledge, so teachers’ beliefs should be inferred from their verbal expressions and predispositions to action, instead of relying only on questionnaire data (Ertmer, 2005; Pajares, 1992). On the other hand, in order to make sure that some implicit constructs such as teachers’ beliefs could be measured both reliably and adequately, some quantitative studies asserted that teachers’ unobserved beliefs could be identified using items in the questionnaire, and then the confirmatory factor analysis (CFA) can be employed to examine the links between unobserved beliefs and observed items (Lay, Chi, Hsieh, & Chen, 2013; Teo, 2011). Additionally, the results of the CFA could also explain 17.

(25) whether or not these items are reliable indicators of the unobserved beliefs (Teo, 2011). Furthermore, the results of CFA could be combined with regression analysis or path analysis to establish the possible relationships among beliefs, attitudes, and intention. It should then be possible to estimate the interactions and correlations between these variables (Chen, 2010; Pynoo et al., 2011). Hence, conducting the technique of SEM that combines both CFA and path analysis should allow me to estimate the direct and indirect effects among variables through path analysis (Chen, 2010; Teo, 2011). In other words, the possible relationships among beliefs, attitudes, and intention could be specified, and a series of relationships could be estimated simultaneously by performing an SEM path analysis (Lay et al., 2013). Thus, if researchers want to not only portray the features of teachers’ beliefs about TBAs, but also test the relationship between teachers’ beliefs about TBAs and their assessment practices, and identify the mechanism of how they interact with each other, a study combining qualitative data analysis, a SEM that combines both CFA and path analysis is needed. Finally, according to the purpose 3, I also want to estimate the effects of teachers’ practice and beliefs about TBAs on students’ performances. Thus, a hierarchical linear modeling (HLM) technique which including the comparison of the differences between and within different schools should be conducted in these series of studies to provide a more comprehensive estimation about the effects of teachers’ practice and beliefs about TBAs on students’ performances. To summarize the above considerations, a series of studies that combined qualitative data analysis, SEM analysis and two-level HLM were conducted. In the study 1, the main components and features of teachers’ beliefs and practice about TBAs were investigated and analyzed based on the qualitative data of 40 technology-experienced science teachers. In the study 2, I developed a specific 18.

(26) questionnaire from the result of the study 1 to elicit another 494 high school science teachers’ beliefs about TBAs and conducted a CFA to ensure unobserved beliefs could be identified by my questionnaire to take the study 1 a step further. Besides, the study 2 specified the possible relationships among teachers’ beliefs about, attitudes toward, and intention to use TBAs by conducting a SEM analysis. Finally, a HLM analysis was conducted to estimate the effects of teachers’ practice and beliefs about TBAs on students’ performances. The same 494 science teachers in the study 2 and their 1984 students from eighth and eleventh grades from 32 secondary schools participated in this study 3.The detailed research design and the primary results of my three studies will be presented sequentially in the next sections.. 19.

(27) 5. An Investigation of Teachers’ Beliefs and Their Use of Technology-based Assessments (The study 1)1 5.1 Purpose and research questions In the context of school settings, when implementing educational innovations such as technology-based assessments, teachers take a variety of factors into consideration including the resources provided by their school, the approval of parents, and students’ needs. To explore teachers’ beliefs and relevant factors about TBAs, therefore, the study 1 adapted the framework from DTPB, and aimed at portraying the features of teachers’ TBAs beliefs and exploring the possible interaction between the beliefs and the degrees of actual use. The research questions that guided the study 1 were as follows. 1.. What are the substantial components and their features in teachers’ beliefs about technology-based assessments?. 2.. To what degrees do teachers use technology-based assessments in classrooms?. 3.. Is there an interaction between teachers’ beliefs and their degrees of the use of TBAs? If so, what is the nature of the interaction?. 5.2 Research design Since teachers’ beliefs are highly personal and context-related, and these beliefs are existed implicitly as the tacit knowledge, so both Pajares (1992) and Ertmer (2005) suggested that teachers’ beliefs should be inferred from their verbal expressions and 1. This study has been published on the journal “Computers in Human Behavior”. Chien, S. P., Wu, H. K., & Hsu, Y. S. (2014). An investigation of teachers' beliefs and their use of technology-based assessments. Computers in Human Behavior, 31, 198-210. doi:10.1016/j.chb.2013.10.037 20.

(28) predispositions to action, instead of relying only on questionnaire data. Fang (1996) also claimed that teachers’ responses to questionnaires may reflect their judgments about the questionnaire items, rather than their thinking process. Therefore, qualitative methods that document teachers’ verbal expressions and their interpretations of teaching behaviors were appropriate for the studies of beliefs, especially for the studies of which the purpose was to gain more insights into specific beliefs (Pajares, 1992). Based on the methodological suggestions from previous studies, the study 1 employed a qualitative research design to investigate teachers’ beliefs about TBAs as detailed as possible. Semi-structured interviews, which included questions about teachers’ self-reported assessment practices and their comparison between TBAs and the assessments that they frequently use, were used to elicit teachers’ beliefs. Transcripts from the interviews were the main data source used for the analysis. 5.3 Participants In order to explore teachers’ beliefs about TBAs, 40 teachers including 15 females and 25 males who had more than 5 years of experiences in using technologies in classrooms were recruited to participate in the study 1. The shortest experience in using technologies of them was 3 years and the longest was over 15 years. All the teachers came from urban school districts. And 90% of them taught science in public school (36 out of 40). These teachers taught science subjects in either junior or senior high schools and their backgrounds are described in Table 1.The reason I chose science teachers was because deep understanding of science at the secondary-school level involves understandings of complex systems and dynamic processes, and new technologies have the potential to provide accurate measures of such understanding (Kuo & Wu, 2013; Quellmalz et al., 2012). These science teachers may have more opportunities to access to TBAs. Additionally, even though teachers with experiences of using instructional 21.

(29) technologies may or may not use TBAs in classrooms, because of their rich experiences with technologies, they could provide more thoughts and information about TBAs than inexperienced teachers. The average age of the 40 teachers was 43 and they had an average of 10-year teaching experience.. Table 1 Backgrounds of the Participating Teachers Subject Taught Grade level Junior high school Senior high school. Physic. Chemistry. Earth science. Biology. 5 5. 5 5. 4 6. 5 5. 5.4 Development of the semi-structured interview To explore teachers’ beliefs and their use of TBAs, a set of interview questions were constructed on the basis of previous research (Ogan-Bekiroglu, 2009; Smarkola, 2008; Wang et al., 2008) and focused on teachers’ beliefs about and their use of TBAs. A pilot interview was conducted to refine and improve the questions. Also, a group of specialists, including university professors and in-service teachers, reviewed the interview questions to ensure that these questions were appropriate to elicit teachers’ beliefs and their use of TBAs. Besides, several pilot interviews and discussions were also conducted among three interviewers (1 post-doctoral researcher and 2 doctoral students) to ensure the consistency among different interviewers. In the final version, examples of the questions included: (1) What do you know about technology-based assessments? (2) Have you ever used these assessments to assess your students’ performances? If so, how? (3) What do you think the differences between technology-based assessments and the assessments you are using? (4) How do you 22.

(30) think that technology-based assessments can be used to help you know more about students’ learning? Any unclear responses were questioned further and the interviewed teachers were asked to provide examples to illustrate their statements. The participating teachers were interviewed individually in their school offices. All the participants agreed to be audio-recorded and were fully informed that their interviews would be transcribed and analyzed for research purposes. The average time of each interview was approximately 45 minutes. 5.5 Development of the coding scheme To answer the research questions, the coding scheme was generated through an iterative process and divided into two parts. The first part focused on teachers’ beliefs about TBAs, and I adopted the three categories of beliefs (i.e., behavioral, control, and normative) and related components identified in DTPB for the trial coding. The coding results and new components of beliefs emerged from the data guided me to reframe the scheme and a refined scheme was then created for another trial coding. The refining process was repeated until the scheme accurately portrayed teachers’ beliefs of TBAs. The final version thus included 10 components (Table 2) mentioned by at least 10% of the teachers and some beliefs were not identified by the literature. For example, according to the teachers, the time and financial support in the control beliefs were crucial in school settings, and the normative beliefs in the study 1 were decomposed into the perceptions about student learning, parents, and policy. The second part was used to analyze the various degrees of use of TBAs by the teachers (Table 3). In this study, teachers’ degrees of use were divided into three groups: non-user, occasional user, and frequent user and were determined by teachers’ self-reported usage frequencies. This division was established because previous studies have indicated that expert teachers, in-service teachers, and pre-service 23.

(31) teachers showed different extent of integration of technology-based teaching (Churchill, 2006; Ertmer et al., 2012; Lee & Lee, 2014). Thus, in this study, I would like to find out whether teachers’ degrees of TBAs usage would have similar effects on their extent of integration of TBAs or not. In this study, teachers who used TBAs regularly and more than once per semester were coded as frequent users; occasional users were those who used TBAs fewer than once per semester; and non-users had never used TBAs before. This cutting point was based on the teachers’ responses. In addition to the frequencies, I also characterized the features of their use as suggested by previous studies (Benbasat & Barki, 2007; Chuttur, 2009).. 24.

(32) Table 2 Coding Scheme and Coding Results of Teachers’ Beliefs about TBAs Category. Component. Usefulness (34a / 119b ). Behavioral Beliefs (36/161) Ease of use (15/27). Compatibility (13/23). Belief Bu1: Offering immediate responses Bu2: Providing students’ records and log files Bu3: Analyzing results Bu4: Supporting multiple representations of test items Bu5: Measuring and revealing students’ understandings as well as their process skills Bu6: Assembling a test based on teachers’ purposes Bu7: Allowing students to take the test repetitively Bu8: Allowing students to take the test at any place and any time Bu9: Allowing students to share answers with others Be1: The interface of TBAs is difficult to understand Be2: The Hardware of TBAs is difficult to operate Be3: The web-based platform of TBAs is difficult to manipulate Be4: The Design of TBAs system is complex to learn Bc1: TBAs fit well with the teachers’ current needs Bc2: TBAs do not fit well with teachers’ current needs due to previous experience Bc3: TBAs do not fit well with teachers’ current needs due to personal averseness. 16 15 14 12. Number of responses coded 25 23 20 19. 11. 13. 9 7. 11 8. 7. 7. 6 10 4. 7 15 7. 1. 3. 1 3. 1 3. 5. 8. 7. 12. Number of teachers coded. 25.

(33) Table 2 (continued) Category. Component. Belief. Number of teachers coded. Number of responses coded 2 32 14 8. Ct1: Facilitating conditions 2 Time (21/34) Ct2: Constraining conditions 21 Cs1: Facilitating conditions 11 Supporting personnel (18/22) Cs2: Constraining conditions 8 Control Beliefs Infrastructure: Ci1: Facilitating conditions 3 3 (29/81) hardware and Ci2: Constraining conditions 14 18 software (15/21) 1 1 Financial support Cf1: Facilitating conditions (4/6) Cf2: Constraining conditions 3 5 Perception about Nsl1: Advantages of TBAs for student learning 13 16 student learning Nsl2: Disadvantages of TBAs for student learning 7 10 (19/28) Normative Perception about Npo1: Supported by school and national policies 0 0 Beliefs policy Npo2: Discouraged by school and national policies 7 10 (26/51) (7/10) Perception about Npa1: Approval from parents 1 1 parent Npa2: Disapproval from parents 5 6 (5/7) a b Note. The number refers to the total number of teachers coded. The number refers to the total number of teachers’ responses coded; because one response could be coded by more than one belief, this number is smaller than the total of the numbers of responses coded (the last column) in the category. 26.

(34) Table 3 Coding Scheme and Coding Results of the Degrees of Use. Degree of use. Characteristics. The teacher does not know anything about TBAs and has never used them. Non-user The teacher can identify examples of TBAs but has never used them. The teacher asserts that he or she has used TBAs once but will not use them again. The teacher accepts TBAs as an assessment tool but asserts that traditional assessments are still the Occasional user dominant ways of classroom assessment. The teacher accepts TBAs as an assessment tool but only use them occasionally (once per semester). The teacher is accustomed to using TBAs in their classroom activities. Frequent user The teacher is accustomed to using TBAs and has revised or designed a TBA to meet their needs.. Number of teachers coded. Total Number of teachers. 2 4. Average experiences of teaching More than 10-15 years. Average experiences in using technologies 5-10 years. 2 7 3. Less than 10-15 years. More than 5-10 years. 10-15 years. More than 5-10 years. 17. 7 13 19 6. 27.

(35) 5.6 Procedure of data analysis The qualitative data were analyzed followed the analytical procedures suggested by Erickson (1998). The interviews of 40 teachers were tape-recorded and transcribed verbatim. I used “response” as an analysis unit that included a coherent answer to an interview question. The transcripts were then coded through the iterative process mentioned above, and each response could be specified by more than one code if the response involved more than one belief. The coding scheme was refined several times and a PhD student majored in science education was invited to code the data with me by using the final version of the scheme. The inter-rater agreement between two coders was 90.5%. After the process of inter-rater checking, the coding scheme was finalized and I coded the whole data formally. A qualitative data analysis package NVivo was used for coding and managing data. I reviewed the coded transcripts repeatedly, counted the numbers of coded responses for each belief to reveal the prevalence of substantial beliefs, and made comparisons between data coded under different beliefs and degrees of use (Bazeley, 2007). I then generated analytical notes and searched for assertions to answer the research questions. Each assertion was validated by confirming evidence from the data corpus. 5.7 Findings 5.7.1 Overview of teachers’ beliefs Table 2 shows that among the three categories, teachers’ behavioral beliefs of TBAs were the most prevalent. Nearly all the 40 participants made comments on this dimension and generated a total of 161 responses. Additionally, among the ten. 28.

(36) components of the three categories, usefulness (34 teachers/119 responses), time (21/34), and perception about student learning (21/34) were the most considered ones. It is worthy to note that some components identified in the study 1 (Table 2) were different from the ones in the DTPB model (Figure 2) because my analysis focused on teachers’ beliefs in a classroom context rather than the beliefs of general technology users. As teachers, participants were concerned more about students, parents, and educational policies. For example, the superior’s influence in the normative beliefs (Figure 2) was separated into the perception about policy and perception about parents (Table 2) because according to the teachers’ responses, both types of perceptions were critical to their decision making. Similarly, I divided and rearranged the technology and resource facilitating conditions in the control beliefs into four conditional components, including time, supporting personnel, infrastructure, and financial support. In addition to decomposing DTPB into components related to the teaching context, I also added or removed components in DTPB based on their prevalence in our interview data. For instance, the perception about student learning was added into the category of normative beliefs because about half of the participants mentioned it. On the other hand, I removed the component of peers’ influence from normative beliefs because less than 5% of the teachers expressed such consideration. By focusing on teachers’ beliefs, therefore, the scheme in Table 2 articulated the DTPB model and indicated the important beliefs about technology-based assessments. Because each category involved three to four components mentioned by at least four teachers (10% of the participants), we took all the three categories into consideration without neglecting any of them in order to portray a complete picture of teachers’ beliefs about TBAs. Below we present the major themes emerging from the teachers’ responses and begin with the findings of teachers’ behavioral beliefs. 29.

(37) 5.7.2 Teachers’ behavioral beliefs about TBAs Four themes emerged from the analysis of teachers’ behavior beliefs. Firstly, as expected, 85%of participating teachers (34 out of 40), who were technology-experienced, perceived TBAs as useful tools and indicated a variety of usefulness of TBAs. Secondly, the analysis showed that teachers’ beliefs about some kinds of usefulness were linked to certain types of TBAs. Furthermore, despite their beliefs on the usefulness of TBAs, 38% of participants (15 out of 40) also recognized the difficulties in using TBAs and their beliefs of ease of use were mainly negative (Table 2). Finally, teachers’ beliefs about compatibility involved personal averseness (e.g., I don’t like the computer-based assessment) as well as the evaluation of TBAs based on their previous experience. Detailed descriptions and evidence to support the themes are presented as follows. 5.7.2.1 Most teachers perceived a variety of usefulness of TBAs As Table 2 illustrates, 34 teachers mentioned the usefulness of TBAs. Moreover, 88% of the 34 teachers (30 out o 34) described more than two kinds of usefulness in the interview. It appeared that most of the teachers not only regarded TBAs as useful tools for teaching and learning, but also perceived multiple kinds of usefulness of them. Examples of teachers’ responses are presented below. T26: Compared to traditional assessments, web-based assessments could provide more detailed analysis (Bu3 in Table 2). For example, I could immediately know (Bu1) the distribution of students’ answers (Bu2) after the test was completed. According to the teachers, traditional assessments referred to paper-and-pencil tests and teachers’ questioning in classrooms. In the comments above, T26 indicated that TBAs afford teachers to analyze the test results more comprehensively and 30.

(38) quickly, and can provide multiple representations of a test item. Overall, T26’s comments included four different beliefs of usefulness (Bu1-4) in Table 2. 5.7.2.2 Some teachers’ beliefs of usefulness were linked to certain types of TBAs In addition to the usefulness of TBAs, teachers usually provided examples of TBAs to illustrate their point. I thus analyzed the possible links between the kinds of usefulness teachers perceived and the types of TBAs they used. Table 4 shows the results. According to teachers’ responses, there were seven types of TBAs used by them (from the most mentioned to the least): web-based assessment (the number of responses = 32), immediate response system (IRS; the number of responses = 21), test item bank (17), interactive simulation (16), discussion forum (15), teacher questioning with multimedia (3), and game-based assessment (1). If the teachers did not mention any specific type of TBAs, their responses were assigned to the category of not specified. Boldfaced and underlined numbers are the maximum value of the row and represent the most frequently mentioned assessment in a belief of usefulness. The analysis revealed that some beliefs of usefulness, including offering immediate responses (Bu1), analyzing results (Bu3), measuring knowledge & skills (Bu5), assembling the test (Bu6), repetitively testing (Bu7), and sharing answers with others (Bu9) seemed linked to certain types of assessments. As seen in Table 4, teachers believed that the IRS and web-based assessments are useful for offering immediate responses (Bu1) and that interactive simulations could help measure or reveal students’ knowledge and skills (Bu5). Also, 55% of the responses (12 out of 22) about the belief of analyzing results (Bu3) involved web-based assessments because the web-based assessment platforms usually provide a function for statistical analysis. Furthermore, teachers believed that test item banks afford them to assemble the test easily and use it repetitively (Bu6 and Bu7). When it came to the discussion forum, 31.

(39) teachers thought that it is suitable for promoting students’ sharing of answers (Bu9). Examples of teachers’ comments about the Bu1 and Bu5 are presented below. T34: I still think that IRS is the most effective in detecting whether students have understood the learning materials or not (Bu1). T11: Many students can correctly answer the questions about the operation of the microscope in a traditional assessment, but few of them can really operate a microscope to observe a real sample. But it would also be difficult for teachers to know what students’ problems are when they operate a real microscope. Thus, if students use the interactive simulation software, then it will be easier to reveal their learning difficulty (Bu5). Just like what T34 said, the IRS was usually perceived as an effective tool in providing teachers with students’ immediate responses to learning materials. According to T11, interactive simulations were helpful to reveal students’ difficulties by having students manipulate a virtual science instrument on the software so teachers could measure students’ process skills. Taken together, this theme suggests that a majority of participating teachers had experiences with a variety of TBAs and were able to identify the affordances and usefulness of certain types of TBAs.. 32.

(40) Table 4 Beliefs of Perceived Usefulness and Types of TBAs Teachers’ Used (Numbers of Responses) Discussion. Web-based. Interactive. Test item. Game-based. Teacher questioning. Not. forum. assessment. simulation. bank. assessment. with multimedia. specified. 10. 0. 9. 0. 2. 0. 1. 3. 25. 3. 7. 5. 3. 1. 1. 0. 3. 23. 6. 2. 12. 0. 0. 0. 0. 2. 20. 0. 1. 2. 2. 0. 0. 1. 13. 19. 0. 0. 0. 9. 0. 0. 1. 3. 13. Bu6: Assembling the test. 0. 0. 3. 0. 6. 0. 0. 2. 11. Bu7: Repetitively testing. 0. 0. 0. 2. 5. 0. 0. 1. 8. 0. 1. 1. 0. 3. 0. 0. 2. 7. 2. 4. 0. 0. 0. 0. 0. 1. 7. 21. 15. 32. 16. 17. 1. 3. 30. Beliefs of Usefulness. Total. IRS. Bu1: Offering immediate responses Bu2: Providing records and log files Bu3: Analyzing results Bu4: Supporting multiple representations Bu5: Measuring students’ understanding. Bu8: Free of time and space Bu9: Sharing answers with others Total. 33.

(41) 5.7.2.3 Teachers’ beliefs about the ease of use of TBAs were mainly negative Despite teachers believed in the usefulness or the potentials of TBAs, participants also recognized the issues and challenges in using TBAs, such as difficulties caused by the interface, hardware, platform, and design of a system. Among the beliefs in ease of use, the beliefs about the interface of TBAs (Be1) were the most considerable one (Table 2). Moreover, all of the ten teachers with Be1 stated comments on restrictions of the interface of TBAs. These teachers thought that TBAs were complicated for both students and teachers to use, as the two teachers stated below. T04: I feel that compared to a traditional assessment task, a simulation task needs students to make special efforts to understand how to operate the simulation. T29: When we are assembling the test in the “Moodle,” the assembling process requires us to write the programming language. It is not easy for every teacher to accomplish. Similar to what T04 stated, 38% of participants believed that not only teachers but also students need to overcome the technical challenges and learn how to use the interface, hardware, or platform of a TBA before they can use a TBA as an assessment tool, and that this process required extra effort and time. Moreover, the use of TBAs could be even more complicated if it required teachers to write programming language in order to set up their TBAs (see T29’s comment). All four beliefs in the ease of use component were related to restrictions and difficulties of TBAs. In other words, nearly all the teachers with these beliefs in the component perceived the complicacy and inconvenience of TBAs.. 34.

(42) 5.7.2.4 Teachers’ beliefs about compatibility involved previous experience as well as personal averseness to TBAs Among the three beliefs of compatibility, one of them was positive and only three teachers asserted that TBAs are compatible with their current needs in assessment (Bc1 in Table 2). On the other hand, two of the beliefs (Bc2 and Bc3) were negative, and the total number of responses with the negative beliefs (20) was much higher than the number of positive responses (3). The analysis of teachers’ responses revealed that their beliefs involved personal averseness as well as the evaluation of TBAs based on their previous experiences. Examples of the Bc2 and Bc3 are presented below. T37: I have used some of these (TBAs) before, such as IRS. However, I still feel that it is nothing more than a superficial and fancy tool. T01: I don’t like the TBAs. Of course, I want to know more about students’ understanding, but the assessment items should be aligned with my teaching. I won’t use a TBA, unless I read the items beforehand and teach the concepts of these items intentionally. Both T37 and T01 expressed their negative beliefs about TBAs. T37’s comment was based on his previous experience and he did not think that TBAs could meet his needs. T01, as one of the non-users, did not use TBAs due to her personal averseness as well as her beliefs about the misalignment between the content of her teaching and a TBA. 5.7.3 Teachers’ control beliefs about TBAs Overall, the analysis showed that teachers’ control beliefs about TBA focused on the social and external components such as time, supporting personnel, infrastructure, and financial support (Table 2) rather than the internal or personal factors (e.g., self-efficacy). In addition, although these external components could either facilitate 35.

(43) or constrain the use of TBAs, 3 of the 4 components (i.e., time, infrastructure, and financial support) were mostly regarded as constraining conditions. The only belief component that included more facilitating conditions than constraining ones was supporting personnel. This is probably either because participants believed that supporting personnel usually played a positive role in the use of TBAs or because compared to other components, the lack of supporting personnel was easier to overcome. 5.7.3.1 Teachers felt the constraints of time, infrastructure, and financial support The component of time was recognized as a constraining condition by 53% of teachers (Ct2 in Table 2). Twenty-one teachers asserted that they needed more time to prepare the TBAs before using them. Below are the two typical comments from teachers. T18: I mean that the incentive is too small to use it (TBA), because I may spend two hours on preparing and the product is merely a small quiz. T21: Yes, students would like it (TBA). In addition, the immediate display is good. However, you have to design the assessment and this takes a lot of time. We need to spend most of our time on preparing teaching materials and use the rest of time to prepare the assessment task. The typical comments indicated that more time was needed to design (T18’s comment) or prepare (T21’s comment) the TBAs. While the participating teachers usually placed lesson planning as a higher priority, they did not have much time left for designing assessments. Moreover, another external constraint identified by 35% of teachers (14 out of 40) was the infrastructure (Ci2 in Table 2). T03: Just as what I said, you need to provide all the 30 students with computers (when using a TBA). However, it is impossible for me to offer so 36.