The purpose of this study was to develop and validate an MCAT system for use with the biology assessment in Indonesia. In order to achieve this objective, an item bank and scales were developed, and then the item bank was integrated into the MCAT system.
The system was then evaluated so that it had proper measurement precision and an acceptable standard error. This thesis provides a report on the processes involved in the development and validation of the MCAT system for Indonesian junior secondary school biology.
4 1.3 Significance of the Study
The study is significant because this is the first MCAT development in an Indonesian context. The design, method, and findings lay the foundation for future MCAT studies.
Whereas previous CAT studies focused on UCAT, which provides an overall score, MCAT provides sub-score information of a comprehensive achievement test result in the different domains tested. Furthermore, the results from MCAT can be used as feedback for remedial instruction. The feedback could be used by individuals as self-regulated learning, in peer groups, or in groups supported by teachers (Mok & Zhu, 2014;
Selvarajan & Vasanthagumar, 2012; Wang, 2009). Another significant contribution of the study is that the study can be used as a model for NE. This will show how to prevent items from leaking and cheating behaviors during the current NE manual P&P practices.
To summarize, this chapter has provided the background information of the study and the purposes and significance of the study. MCAT is one of the most advanced assessment systems that has been introduced since 1999. Regardless of the fact that the MCAT system has been used in some developed countries, this assessment model is still new for Indonesia. Currently, Indonesia still uses P&P mode to assess their students, including its large-scale assessment, NE. This study would be helpful particularly for students in grade 9 in their preparation for the NE. In the long-term, the system will serve as an initial contribution for future MCAT engagement in the NE of Indonesia.
5
CHAPTER TWO: LITERATURE REVIEW
This chapter reviews literature relevant to the study. It covers the biology curriculum and assessments, Indonesian national examination challenges, item bank and scale development, MCAT procedures and performance evaluation, and CAT studies in Indonesia.
2.1 Biology Curriculum and Assessment
Biology is perceived as an important subject taught and evaluated nationally by the central government of Indonesia (Ministry of National Education, 2003; Ministry of National Education, 2006a; Ministry of National Education, 2006b). Indonesia has two education systems, namely, the Ministry of National Education (MONE) and the Ministry of Religious Affairs (MORA). Both Ministries supervise schools under their umbrella from kindergarten to senior secondary school and university. They also share resources, such as teachers and school supervisors. For biology curriculum development and its national assessment, MONE plays a greater role than MORA because it develops all teaching standars and its assessment guidelines.
The subject of biology is introduced to kindergarten students through simple knowledge, such as pictures of animals and human beings (Harianti, 2007). In Primary School (PS), biology is introduced through thematic approaches in conjunction with the related subject learned, and in Junior Secondary School (JSS), the instruction is expanded to simple research (Haryanto, 2012a; Haryanto, 2012b; Karnoto & Rusdi, 2008a; 2008b;
6
2008c; Sujiranto, Widianto, & Supriyana, 2006a; 2006b; 2006c). Biology teaching in Senior Secondary School (SSS) becomes more structured based on students’ majors (Ministry of National Education 2006a), such as science and social majors. Science majors will attend schools for longer hours and take more subjects related to biology, such as chemistry, physics, and experiment/research classes. On the contrary, students in social majors will take less science-related subjects and have more study time allocated to subjects other than science (Ministry of National Education, 2003; Ministry of National Education, 2006a; Ministry of National Education, 2006b).
In terms of assessment, biology is assessed by school system, national, and international assessments (BNSP, 2013; Ministry of National Education, 2013b; Ministry of National Education, 2014). School system assessment refers to assessments conducted by teachers in the form of formative, middle-term, and summative assessments (Ministry of National Education, 2003). The national assessment is conducted by the central government (BNSP, 2013). It is administered annually at the end of the study term for Grade 6, Grade 9,and Grade 12. This assessment usually takes place around April-May every year (Ministry of National Education, 2013b; Ministry of National Education, 2014). The international assessment is an assessment conducted in cooperation with international agencies, such as The Organization for Economic Co-operation and Development (OECD), for the Trends in International Mathematics and Science Study, or TIMSS (OECD, 2012a).
7 2.1.1 Biology curriculum
As a compulsory subject imposed by the central government of Indonesia, guidelines for teaching, the form of curriculum, and the syllabus for biology are prepared centrally by the Ministry of National Education (MONE) in Jakarta. The curriculum is to be used in both Indonesian schooling systems: the Ministry of National Education (MONE) and the Ministry of Religious Affairs (MORA). Therefore, in terms of biology teaching and assessment, teachers in both types of schools are to adhere to the MONE guidelines. What they need to do is they must collaborate regarding methodologies and teaching-learning aids in order to transfer biology knowledge to their students. Teachers are free to use any textbooks approved by the national curriculum (Harianti, 2007;
Ministry of National Education, 2012; Ministry of National Education, 2013a).
MONE revises the curriculum from time to time, if required. No dramatic change has been made since 2004 when it added more materials and longer hours for study time, which changed the previously 4 hours per week to 6 hours per week (Ministry of National Education, 2003). Another revision was made in 2013 to add more hours to biology; however, this has not been approved by the national parliament since it has a significant impact on other subjects’ time allocation. Thus, all Indonesian students are expected to have the same references in learning the subject, such as the curriculum, syllabus, teaching methodology, and learning aids, as well as to be taught by teachers with similar competencies.
8
According to the regulation of the Ministry of National Education (MONE) number 23 point j (2006a, pp. 23), the standard competency to be achieved by JSS students in biology are as follows:
1. To have the ability to do observations with appropriate equipment and research with suitable procedures, to write the observations and measurement processes in appropriate tables and graphs, to conclude, and to explain the findings orally and in written form.
2. To understand the variety of creatures, the classifications based on types and characteristics, the methods of damage prevention, and the mutualism among creatures within an ecosystem.
3. To understand human anatomy functions and livings live.
Based on this regulation, which is perceived as the curriculum, MONE also supports teachers with syllabi for detailed teaching guidelines. The syllabus for grade 7, as stated by Sujiranto et al. (2006a), is as follows: research and observation of creatures and living things, variety of creatures, variety of livings and organisms (cells), mutual benefits within an ecosystem, and human population. The curriculum for grade 8 is as follows: growth and development, characteristics of human beings by age, movement systems in human beings and vertebrata, human digestion, human breathing and vertebrata, human blood system, plant structure and function, photosynthesis, and plant movement (Sujiranto et al., 2006b). The curriculum for grade 9 is as follows: human excretion system, human reproduction system, human neuron function, human eyesight, organisms (cells), heredity, and biotechnology (Sujiranto et al., 2006c).
Therefore, it can be summarized that the teaching and learning processes in JSS grade 7 focus on research/observation, creature classification of plants (botany), animals and human beings, ecosystems, and human population. In grade 8, instruction focuses on
9
types, characteristics, and functions of plants (botany), human beings, and animals. In addition, photosynthesis is part of the content. Broader instruction about human beings is provided in grade 8. In grade 9, the material learned for human anatomy, cells and their related systems, functions and problems/diseases of excretion, neurons, eyesight, and heredity is more comprehensive. Organisms’ cell and biotechnology is also taught.
A bulleted list of the curriculum for JSS is as follows:
1. Biology and research: introduction to biology, characteristics of creatures, and research.
2. Botany: types of plants, characteristics of plants, plant growth, photosynthesis, and plant cells.
3. Zoology: types of animals, characteristics of animals, animal growth and metabolism, and animal cells.
4. Human beings: anatomy functions, human environments, activities and diseases, and human cells.
5. Anatomy functions of human body: pancreas system and diseases, breathing systems and diseases, blood systems and diseases, excretion and diseases, neuron systems, and eye functions.
6. Ecosystems: human interaction, human population, prevention of ecosystem damage, biotechnology, and human industrial waste.
The biology curriculum from grade 7 to grade 9 is the basis for the item
construction. This curriculum was used as the content of this MCAT system because the system is intended to be aligned with the national examination (NE). Furthermore,
10
students at junior secondary schools need to learn the material from grade 7 to 9 if they want to score well on the national examination (NE).
2.1.2 Biology assessment
Students’ competencies in biology are measured in a number of assessments, including the school system, national, and international assessment (Ministry of National Education, 2013b; Ministry of National Education, 2014). The purposes of these assessments range from remedial purposes to curriculum adjustment and educational quality improvement (Firman & Tola, 2008; Ministry of National Education, 2010b). In fact, educational quality improvement would be nearly impossible to achieve if remedial instruction were not considered carefully. Consequently, feedback is important for any form of test.
Teachers use the school system assessment to determine student progress and how well the material has been learned by the students (Ministry of National Education, 2010b; OECD, 2013). In addition, assessments are often used to get information on who is in need of remedial instruction. Assessments are also used to identify the higher performing students so that the teacher can ask them to be peer mentors for other students. Furthermore, schools also use assessment results to rank students and to support them with different types of rewards in appreciation of their achievements.
In school assessment practices, there are at least three major assessments: pre-test, formative test, and summative test (Ministry of National Education, 2010b; Nitko, 2001).
A pretest is conducted before learning starts. This is to determine the existing
11
competencies of students. As a result, schools group students according to their level of abilities. This test is also called a placement test, and according to the test results, students are classified into core classes, classes A, B, C, or D, based on their average competencies (Smith & Ragan, 1999).
The formative test is used to identify students’ progress with the materials being learned (Nitko, 2001). This type of test allows teachers to monitor their students’
progress. Teachers may need to adjust their teaching methodology accordingly to achieve the goal of learning (Nitko, 2001). This type of test is usually called a middle-term test. A summative test is an assessment conducted at the end of a learning period. This type of assessment aims to see how students perform on all learning materials after one semester of learning. It is also called a final term test (Smith & Ragan, 1999).
For the three tests (pre-test, formative test, and summative test), teachers can use performance approaches or essay writing test approaches, depending on the topics of the learning materials to be tested. In the performance test approach, students are required to do actions based on the stimuli provided. For example, the teachers ask students to perform the proper use of laboratory tools for biology research topics. The teachers also ask students to lebel human skeletons to determine their learning progress for the topic of anatomy functions. In essay tests, students are asked to provide their opinion by writing their answers to the questions presented (Nitko, 2001).
Biology is assessed using all of the assessments in Indonesian schools. Regarding curriculum coverage, these school assessments only partially cover the national
12
curriculum for an ongoing learning period. For example, summative tests only cover one semester of curriculum studies. Students receive feedback on their performance.
On a broader scale, there is the national assessment, called ujian nasional in Indonesian (National Examination, NE). The NE aims to improve the quality of education across Indonesia (Firman & Tola, 2008). This also means that the result of this national assessment would be used by the government to improve biology teaching in Indonesia. This NE has been conducted as a form of national assessment since 1980 (Ministry of National Education, 2003). For junior secondary schools, this assessment takes place at the end of the last semester of grade 9. If students can pass the exam, they can graduate and move on to a higher schooling system, that is, senior secondary school.
Otherwise, they have to stay in the school for another year. In other words, junior secondary school students must learn all of the materials that are covered in the national curriculum in order to avoid failing this national assessment; however, students who fail the national examination receive feedback on the overall subject achievement.
Indonesia joined Trends in International Mathematics and Science Study (TIMSS) in 1999. The participation provides an overview of how Indonesian students perform in this assessment, including biology achievement, which is part of the science test. TIMSS results are also used as a consideration for curriculum adjustment and teaching-learning approaches. Indonesian students’ performance was at the 35th rank in 1999, the 55th rank in 2003, and the 63rd rank in 2011 (Mullis & Martin, 2011). Unfortunately, the results of the international assessment are announced the following year. In other words, it is not very useful for students to use it as a remedial reference (Wang, 2009). On the other
13
hand, it may be a good resource for the government for national policy making for subjects tested, including biology.
Biology is assessed using different types of tests, but students do not always get direct feedback on their achievement. Consequently, it is important to have an assessment system that not only assesses all learning materials covered in the curriculum but that also provides detailed feedback for students directly after the test in the different topics (domains) tested. The new assessment system with its feedback is critical to fill the gap and can be used for remedial instruction.
2.2 National examination (NE) challenges
As has been briefly discussed in the previous sections, Indonesia has administered the NE for more than three decades. The NE aims to be a tool for improving the quality of education in Indonesia (Firman & Tola, 2008, Ministry of National Education, 2006c;
Ministry of National Education, 2010b). Indonesia has conducted this national assessment since 1980 for grade 6, grade 9, and grade 12 (Ministry of National Education, 2010a; Coordinator Ministry for Social Affairs, 2014). This assessment takes place at end of every final semester in the different school systems: the 12th semester of primary school in grade 6, the 6th semester of junior secondary school in grade 9, and the 6th semester of senior secondary school in grade 12. Students must receive a passing grade on the NE because it is part of the requirement to graduate and move on to a higher schooling system. The passing grade for 2014 was a 5.50 average score (on 1-10 scale), and students must score above 4.00 on all NE subjects assessed (Ministry of
14
National Education, 2013b; Ministry of National Education, 2014, Nuffic, 2014). The passing grade was a 5.25 average in 2005, and students were required to score above 4.25 on all NE subjects assessed (Kasro, 2010). There were approximately six million students who participated in this assessment in 2014 (Solopos, 2014).
The subjects assessed in grade 6are Indonesian language, math, and science. The subjects assessed in grade 9 are Indonesian language, English, math, and science (Ministry of National Education, 2010b; Republic of Indonesia, 2013). The subjects assessed in grade 12are Indonesian language, English, and math. In addition, different majors will have additional subjects to be assessed: physics, chemistry, and biology for science majors; economics, geography, and sociology for social knowledge majors; and Indonesian literature, anthropology, and one international elective language (Mandarin, Japanese, German, French, or Arabic) for language majors (Ministry of National Education, 2010b; Republic of Indonesia, 2013). Through the NE results, the central government learns which group of Indonesian students is still weak and in which subjects they perform poorly (Aziez, 2011; Fahmi et al., 2011; Firman & Tola, 2008;
Sulistyo, 2009). Based on this information, the government plans quality improvement programs and supports local education offices with more funding to hire more teachers, for example.
Because of the ideal objective of the NE and its reinforcement of the passing grade, NE implementation has created anxiety for students. Students feel that the NE is a big challenge, especially for students in rural Indonesia, which has limited educational resources (Daud, 2006; Evans, 2010; Kuo & Daud, 2015; Tjalla, 2010; Yusuf, 2011).
15
Some schools can provide students with additional classes as preparation for the NE.
Some parents even spend a substantial amount of money to hire private tutors for their students to help them pass the NE. Wealthy families send their students to crammed schools; however, this is not always feasible for schools or parents in most parts of Indonesia.
NE challenges become more dynamic when third parties are involved and take advantage of this situation. Consequently, the leaking of items prior to the administration of NE has become a major issue in recent years (Kuo & Daud, 2015;
Maryono & Purnama, 2012; Rahmi, 2011; Sulistyo, 2009; Tjalla, 2010). The leaks are caused by people who want to benefit financially by selling NE items (Solopos, 2014).
Another existing challenge of the NE is related to the late arrival of the test materials to some schools in remote provinces. This happens because the Indonesian government, through its MONE offices, insists on administering the NE at the same time nationally (Ministry of National Education, 2013b; Ministry of National Education, 2014). The situation could occur because Indonesia still relies on the P&P traditional test application. To ensure that the test materials arrive on time for all schools in Indonesia, the P&P test requires substantial human resources in terms of manual work. Many of the challenges could be eliminated if a computer-based test, a computerized adaptive test in particular, were to be used for the NE (Kuo & Daud, 2014b; Kuo & Daud, 2015).
In summary, this section has discussed the biology curriculum and the different types of biology assessments. The discussion has also explained the challenges of the NE. The different types of assessments in the school systems measure the students’
16
abilities in biology. The assessments provide feedback to students, but they only cover some parts of the curriculum for a particular semester. The national assessment (NE) provides feedback for students, but the feedback is based on their overall abilities only.
As a result, the feedback is not very helpful for remedial instruction, which students really need to be successful on the NE in order to move to the next level of the Indonesian education system. The psychological pressure experienced by students to pass the NE has added challenges to the existing challenges of the NE in its P&P format.
Third parties have also worsened the challenges through item leakage for personal benefits (Kuo & Daud, 2015; Solopos, 2014).
Finally, an assessment system that covers all topics in the biology curriculum and provides feedback in all domains tested is required to adress these issues. This study will present a case for using a computerized-based test for biology assessments with the aim
Finally, an assessment system that covers all topics in the biology curriculum and provides feedback in all domains tested is required to adress these issues. This study will present a case for using a computerized-based test for biology assessments with the aim