Several studies have suggested that socio-cultural backgrounds have significant influence on the assessment (e.g., Bell, 2007). Moreover, in Taiwan, the high-stakes examinations at both school and national level continue to place importance on evaluating students’ performance, especially in science-related fields. Lee, Johanson, and Tsai (2008) and Tsai (2004) suggested that Taiwanese high school students seem to view learning science as “testing.” That is, the major purpose to learn science for them is to get better scores in the high-stakes college entrance examinations.
Consequently, exploring students’ conceptions of science assessment in this examination-oriented culture is meaningful.
43
Table 2.7. The studies conducted the conceptions of assessment
Studies Subject Research method The conceptions of assessment proposed in study Brown (2004) Teachers Self-reported inventory
developed through literature review
z Assessment is for the improvement of teaching and learning z Assessment holds students accountable for their learning progress z Assessment is irrelevant
z Assessment makes schools accountable
Brown and
z Assessment makes students accountable
z Assessment is irrelevant because it is bad or unfair z Assessment improves the quality of learning z Assessment in enjoyable
Brown, et al.
z Assessment improves learning
z Assessment makes students accountable z Assessment is negative and irrelevant z Assessment is liked
Li and Hui (2007)
College lecturers Questionnaire developed through literature review
z Improvement of teaching and learning z Certification of students’ learning z Accountability of schools and teachers
z Treatment of assessment as irrelevant to the life and work of the teachers and students
44
(Continued) Studies Subject Research method The conceptions of assessment proposed in study
Watkins, Dahlin, and Ekholm (2005)
University teachers
Phenomenographic method z Teaching and assessment were seen externally related
z Teaching/assessment relation was seen as external, but awareness of backwash effect was focused on the content what is learnt.
z The awareness of backwash effects was focused on the higher order understandings.
z The awareness of backwash effect was focused on the content what is learnt, but teaching/assessment relation appeared to be
simultaneously as external and internal.
z Teaching/assessment relation appeared to be simultaneously as external and internal, but awareness of the backwash effect included the abilities that students were expected to develop z Students’ learning strategies were included in the backwash effect,
and the grounds of partly external/partly internal categorization were of different kinds.
z Teaching/assessment relation was seen as completely internal.
z Teaching/assessment relation was seen as completely internal, and awareness of the backwash effect included specific references to students’ learning strategies, not only to the results of learning
45
(Continued) Studies Subject Research method The conceptions of assessment proposed in study
Peterson and Irving (2008)
Secondary school students
Interview z Assessment is for the improvement of student learning and may inform teaching
z Assessment does not make students accountable for their learning progress, but it does indicate their learning progress
z Assessment is irrelevant
z Assessment does not make school accountable, but it does hold teachers accountable for student learning
46
II.6. Theoretical Model
By summarizing of the literature review above, the nested ecology regarding science learning, involving the scientific epistemological beliefs, metacognitive awareness, conceptions of learning science, and conceptions of science assessment, was used to multidimensionally investigate the nature of students’ science learning.
The theoretical model for nested ecology regarding science learning is presented in Figure 2.3. As shown in Figure 2.3, this study hypothesized the interrelationships among those four components as below:
(1) The scientific epistemological beliefs are related to the conceptions of learning science.
(2) The scientific epistemological beliefs are related to the conceptions of science assessment.
(3) The conceptions of learning science are related to the conceptions of science assessment.
(4) The metacognitive awareness plays as a mediator in the nested ecology; in which students with higher metacognitive awareness have mature scientific
epistemological beliefs, conceptions of learning science, and conceptions of science assessment.
Through answering the research questions raised in section I.4, the hypothetical nested ecology model will be verified. Thus, to realize the relation between the theoretical model and the research questions, this study placed each research question into Figure 2.3.
47
Note: Q#: research question number (quantitative part of study) Q#: research question number (qualitative part of study)
Figure 2.3. The hypothetical nested ecology regarding science learning
Scientific Epistemological Beliefs
Conceptions of Science Assessment Conceptions of
Learning Science
Metacognitive Awareness
Interrelate
Interrelate Interrelate
Q1; Q5
Q3; Q6 Q7
Q2 Q4
Q4
Q4
Q8
Q8 Q8
Q9: The solid line Q10: The dotted line
48
CHAPTER III METHODOLOGY
This chapter will first describe the general research design used in this study.
And, the introduction about the subjects will be presented next. Finally, the data collection and data analysis will be presented.
III.1. General Research Design
Before verifying the nested ecology regarding science learning proposed in this study and examining the role of metacognitive awareness on scientific
epistemological beliefs, conceptions of learning science, and conceptions of science assessment, the careful investigations about these variables were be conducted first.
To this end, both quantitative and qualitative methods were utilized in this study.
In general, this study consisted of four variables. Ideally, the usages of quantitative survey and interview for each variable were much better. However, as aforementioned, there is no suitable questionnaire for examining students’
metacognitive awareness. And, the issue about students’ conceptions of science assessment is initially examined in this study. Accordingly, this study developed a new questionnaire to examine students’ metacognitive awareness, and initially explored students’ conceptions of science assessment through qualitative method. In sum, Table 3.1 represents the data collection of each variable in this study. As shown in Table 3.1, both the scientific epistemological beliefs and conceptions of learning science were investigated through questionnaire and interview. The metacognitive awareness was examined by questionnaire. And, the conceptions of science
49
assessment were assessed through interview.
Table 3.1. The method of data collection for quantitative part and qualitative part of study
Variables Quantitative part Qualitative part
N 240 60
Scientific epistemological beliefs Questionnaire Interview Metacognitive awareness Questionnaire Questionnaire Conceptions of learning science Questionnaire Interview Conceptions of science assessment None Interview
The general research design is shown in the Figure 3.1. To realize the research design more easily, this study describe the resign design together with the research questions raised in section I.4 as follow.
As mentioned above, the quantitative method were firstly conducted not only to explore the interrelations among students’ scientific epistemological beliefs,
metacognitive awareness, and their conceptions of learning science, but also to choose appropriate information-rich and representative subjects for the qualitative part of study. Accordingly, 240 students were asked to respond to three questionnaires to assess their scientific epistemological beliefs (referring to research question 1), metacognitive awareness (referring to research question 2), and conceptions of learning science (referring to research question 3) in the quantitative part of study.
And, those data were used to initially examine the relations among scientific
epistemological beliefs, metacognitive awareness, and conceptions of learning science (referring to research question 4). In particular, the questionnaire of scientific
epistemological beliefs was used as the base for choosing representative students for in-depth qualitative study.
For the qualitative part of study, 60 representative students were selected for
50
in-depth qualitative study. In this part of study, interview for three major components of nested ecology regarding science learning were conducted through three
fundamental questions: What is science? What is learning science? What is science assessment? These in-depth interview data yielded the research foundation for this study. Accordingly, selected students’ scientific epistemological beliefs (referring to research question 5), conceptions of learning science (referring to research question 6), and conceptions of science assessment (referring to research question 7) were
revealed. After that, the interrelations between those three variables (i.e., beliefs, learning, and assessment) obtained from interview were examined (referring to research question 8). Moreover, the data obtained from questionnaire provided the content validity for the interview data gathered in the qualitative study (i.e., scientific epistemological beliefs and conceptions of learning science).
With the data collected in the qualitative study, 60 representative students’ nested ecologies regarding science learning were constructed and investigated (referring to research question 9). In addition, those representative students’ metacognitive
awareness was assessed by quantitative questionnaire, which was used to examine the association with conceptions of science assessment. After that, the role of
metacognitive awareness on scientific epistemological beliefs and conceptions of learning science and science assessment was discussed (referring to research question 10).
51
Figure 3.1. The research design of this study 240 students
Construct students’ nested ecology of science learning
Phenomenographic method
Verify the role of metacognitive awareness on scientific epistemological beliefs, conceptions of learning science
and science assessment Interview
Conceptions of science assessment
Provide content validity for
Q1 Q2 Q3
Note: Q#: research question number
Beliefs
Learning Assessment
Beliefs
Learning Assessment
Meta
52
III.2. Subjects
The subjects of the quantitative part of study included 240 tenth graders coming from six different classes in a senior high school in Taipei County in Taiwan. As mentioned previously, learners’ scientific epistemological beliefs seem to play a core role in their nested ecology with respect to learning science. Accordingly, this study selected the students who had proper self-awareness about their scientific
epistemological beliefs as the subjects of the qualitative part of study. Thus, the 240 students, as the sampling pool, completed a questionnaire (discussed later) of assessing their scientific epistemological beliefs.
To choose appropriate information-rich and representative subjects for in-depth qualitative study, the “maximum variation sampling” method were utilized. The
“maximum variation sampling” strategy involves selecting cases that illustrate the range of variation in the phenomena to be studied (Gall, Gall, & Borg, 2003, p. 179;
Patton, 1990, p. 172). Thus, this study was conducted for subsequent selection process.
Through maximum variation sampling method, 60 students were selected from the sampling pool, based on their responses to the scientific epistemological beliefs questionnaire, to the in-depth qualitative study. Among those students, referring to sampling strategy utilized in Hashwenh’s (1996) and Tsai’s (1998) study, eighteen were randomly selected from the students who score in the top 15% of scientific epistemological beliefs questionnaire; twenty-four were randomly selected from the average group (those scoring most to the mean of the subjects of pilot study); and eighteen were randomly selected from the bottom 15% group. This sampling strategy also utilized in several studies exploring students’ scientific epistemological beliefs (e.g., Hashwenh, 1996; Tsai, 1998). Accordingly, 60 students including 30 males and
53
30 females were interviewed regarding their scientific epistemological beliefs, conceptions of learning science, and their conceptions of science assessment.
In sum, as shown in Table 3.1, 240 students participated in the quantitative part of study and complete three questionnaires (the detailed descriptions of the subjects will be presented in next section). For the qualitative part of study, 60 representative students were selected for in-depth interviewing about their scientific epistemological beliefs, conceptions of learning science, and their conceptions of science assessment.
Moreover, 60 representative students’ metacognitive awareness were assessed through questionnaire by using the data from the quantitative part of study.
54
III.3. The Instruments for Quantitative Part of Study
III.3.1. Assessing students’ scientific epistemological beliefs
There were numerous instruments developed by researchers to assess students’
scientific epistemological beliefs or their views of the nature of science. For example, Pomeroy’s questionnaire (1993), the open-ended Views of Nature of Science
questionnaire (VNOS, Lederman & O’Malley, 1990; Abd-El-Khlick et al., 1998;
Lederman et al., 2002), a multi-dimensional instrument for assessing students’
epistemological views toward science (Tsai & Liu, 2005), Conley et al. questionnaire (2004). The Conley et al. (2004) questionnaire, namely scientific epistemological beliefs questionnaire (SEB), was chosen for this study based on three reasons. Firstly, this questionnaire had relative higher internal consistency than other instruments.
Secondly, it may also have readable item statements for tenth graders than other instruments. Thirdly, perhaps the main reason, it has encompassed the four
dimensions of epistemological theories proposed by Hofer and Pintrich (1997) and mainly emphasized on the scientific epistemological beliefs. As aforementioned, the scientific epistemological beliefs seem more appropriate than the NOS to investigate students’ nested ecology regarding science learning. The questionnaires mentioned above, except for Conley et al. (2004) questionnaire, were designed on the
characteristics of the nature of science.
Therefore, a self-reported questionnaire developed by Conley et al., (2004) was utilized to gather information regarding students’ scientific epistemological beliefs in this study. The questionnaire developed by Conley et al. (2004) consists of four dimensions (a total of 26 items), presented with bipolar strongly agree/ strongly disagree in a five point Likert scale (i.e., strongly agree, agree, somewhat agree and somewhat disagree, disagree, strongly disagree). The original Conley et al. (2004)
55
questionnaire was designed for assessing elementary students’ scientific
epistemological beliefs. Accordingly, this study modified the item statements of questionnaire to fit the high school context. The revised Conley et al. (2004)
questionnaire is presented in the Appendix A, and namely Scientific Epistemological Beliefs survey (SEB) in this study. A detailed description of the four scales and sample items from each scale are presented below:
(a) Source: assessing students’ beliefs about scientific knowledge residing in external authorities. A sample item is “Whatever the teacher says in science related class is true.”
(b) Certainty: evaluating students’ beliefs in the right answer in scientific knowledge.
A sample is “The most important part of doing scientific studies is coming up with the right answer.”
(c) Development: assessing learners’ beliefs about scientific knowledge as an
evolving and changing subject. A sample item is “Some ideas in science today are different than what scientists used to think.”
(d) Justification: examining learners’ views on the role of experiments and how an individual learner justifies scientific knowledge. A sample item is “Good ideas in science can come from anybody, not just from scientists.”
When scoring the questionnaire responses, the source and certainty scales were reversed so that for each of the scales, higher scores reflected more sophisticated beliefs. Furthermore, Conley et al. (2004) had conducted a series of confirmatory factor analyses to confirm the high reliability and validity of the questionnaire. The alpha reliability of above four scales in the questionnaire used in Conley et al. (2004) study were 0.82, 0.79, 0.66, and 0.76 respectively. As this study modified the original version Conley et al. (2004) questionnaire, a series of work such as consulting with some experts in science education, the examination of both reliability and validity, a
56
series of factor analyses were conducted before and in the study. A detailed description about data analysis for this part will be discussed later.
III.3.2. Measuring students’ metacognitive awareness regarding science learning One of the most popular methods for investigating metacognition is through the use of questionnaire (Downing et al., 2007). In recent decade, several questionnaires were developed to measure students’ metacognition (Downing et al., 2007; Meyer, 2000; Sperling et al., 2002; Thomas, 2003, 2004, 2006, Vandergrift et al., 2006).
However, as aforementioned, the target of Sperling et al.’s (2002) questionnaire and Vandergrift et al.’s (2006) questionnaire does not fit the scope of present study.
And, the items of Downing et al.’s (2007) and Meyer’s (2000) questionnaires are too demanding for tenth graders to complete. Thomas’ (2003, 2004, 2006) questionnaire aims to evaluate students’ metacognition in the science classroom learning
environment. However, the scales of Thomas’ questionnaire mainly focus on students’
perceptions of what happen in the classroom environment, but not on students’
self-awareness regarding their own learning.
Accordingly, this study based on the above questionnaires to develop a new questionnaire for measuring students’ metacognitive awareness regarding science learning, namely Metacognitive Awareness regarding Science learning Inventory (MASI). To develop MASI, a pool of items was collected. Referring to above
questionnaires and aforementioned literatures, this study proposed four scales (a total of 25 items), presented with bipolar strongly agree/ strongly disagree in a five point Likert scale (i.e., strongly agree, agree, somewhat agree and somewhat disagree, disagree, strongly disagree), to measure tenth graders’ metacognitive awareness regarding science learning, that is, self-regulation, critical judgment, metastrategy, and reflection. The MASI is presented in the Appendix B. A detail description of the four
57
scales and sample items from each scale are presented below:
(a) Self-Regulation: examining learners’ self-regulation skill that exerts on their science learning processes and knowledge construction. A sample item is “I know how to commence learning science.”
(b) Critical Judgment: evaluating students’ awareness of critically evaluate information while learning science. A sample is “When discussed with my classmate, I will critically evaluate or examine the reliability of information they expressed.”
(c) Metastrategy: measuring learners’ awareness of their own strategy while learning science. A sample item is “I can motivate myself to learn when I need to.”
(d) Reflection: measuring learners’ awareness of their own cognition while learning science. A sample item is “When it comes to learning new things, I will ask myself about my present learning performance.”
Accordingly, for each of the scales, higher scores reflected higher metacognitive awareness. The examination of both validity and reliability of MASI were conducted in the pilot study. Also, a series of factor analysis (i.e., exploratory and confirmatory factor analysis) to confirm the structure of the questionnaire were conducted in the pilot study. A detailed description about data analysis for this part will be discussed later.
III.3.3. Evaluating students’ conceptions of learning science
To evaluate students’ conceptions of learning science, the Conceptions of Learning Science questionnaire (COLS), developed by Lee, Johanson, and Tsai (2008), was utilized in this study. The development of COLS was based on both the theoretical framework proposed by Tsai (2004) and the interview responses in Tsai’s related research (2004). A series of confirmatory and exploratory factor analysis were
58
further conducted by Lee, Johanson, and Tsai (2008), and confirmed the high reliability and validity of this questionnaire for assessing students’ conceptions of learning science. Thus, the original COLS, consisting of seven scales (a total of 46 items), was utilized in this study.
The original COLS utilized in this study is presented in Appendix C, and a detailed description of the seven scales with respective sample items are presented below:
(a) Memorizing: learning science is characterized as the memorization of definitions, formulae, laws, and special terms. A sample item is “Learning science means memorizing the definitions, formulae, and laws found in a science textbook.”
(b) Testing: learning science is to pass the examinations or to achieve high scores in science tests. A sample item is “Learning science means getting high scores on examinations.”
(c) Calculate and practice: science learning is viewed as a series of calculating, practicing tutorial problems, and manipulating formulae and numbers. A sample item is “I think that learning calculation or problem-solving will help me
improve my performance in science courses.”
(d) Increase of knowledge: learning science is perceived as the acquisition and accumulation of scientific knowledge. A sample item is “Learning science means acquiring knowledge that I did not know before.”
(e) Applying: science learning is for the application of received scientific knowledge.
A sample item is “Learning science means learning how to apply knowledge and skills I already know to unknown problems.”
(f) Understanding: a true understanding is characterized as the major feature of learning science; students highlight the ability to construct integrated and theoretically consistent knowledge structures in science. A sample item is
59
“Learning science means understanding the connection between scientific concepts.”
(g) Seeing in a new way: science learning is viewed in terms of getting a new perspective, and the acquisition of scientific knowledge is to obtain a new way to interpret natural phenomena. A sample item is “Learning science helps me view natural phenomena and topics related to nature in new ways.”
In this study, each factor included from 6 to 8 items, and was presented in a five-point Likert mode. Items on the scales are anchored at 5 = strongly agree, 4 = agree, 3 = no opinion, 2 = disagree, and 1 = strongly disagree. Accordingly, students gaining higher scores in a certain scale showed stronger agreement with the
In this study, each factor included from 6 to 8 items, and was presented in a five-point Likert mode. Items on the scales are anchored at 5 = strongly agree, 4 = agree, 3 = no opinion, 2 = disagree, and 1 = strongly disagree. Accordingly, students gaining higher scores in a certain scale showed stronger agreement with the