Research Background and Motivation

Chapter 1 Introduction

This chapter is an overall introduction of the paper. It first highlights the research motivation of the paper through a discussion of the English testing history and the difficulties the English teachers are facing in Taiwan. Also, the proposed statistical analysis methods, related to the grey system theory, are introduced. The research purpose is then discussed, followed by the research questions. Next, the explanations of terms are introduced, and the significance of this paper is subsequently pointed out. Finally, an overview of the paper has been presented.

1.1 Research Background and Motivation

Traditionally, researchers have classified English language learning under two categories and they are shown in Table 1.1 (DelliCarpini, 2008; Karbalaei, 2010; Laufer and Paribakht, 1998; Nayar, 1997; Rasmussen, 2010):

Table 1.1 English language learning categories English as a Foreign Language

(EFL)

English as a Second Language (ESL)

Definition EFL refers to English instruction and learning in regions where English is not the language of everyday communication

ESL refers to English instruction and learning in regions where it is a first language or is commonly used in education or the workplace Example Taiwan or Japan England or the United States

2

The global economy is blurring these traditional classifications, and English proficiency is now of interest to countries worldwide because of its role in creating a competitive workforce (Pang, Zhou and Fu, 2002). Therefore, followed by the trend of globalization, the educational institutions in Taiwan try hard to enhance the English ability of college students, and English ability is considered to be a vital element of international competitiveness (Black, 2001; Chen and Klenowski, 2009; Gilleard and Gilleard, 2000; Liang, J. S., 2010; Rea-Dickins and Scott, 2007; Wang, 2007). So, how to evaluate students’

English performances becomes more and more important. Hence, Table 1.2 introduces the brief history of English language testing which also provides a general idea of test evolution from the 1980s to current focus (Bachman, 1990, 2000; Bachman and Palmer, 1996; Canale, 1983, 1984; Canale and Swain, 1980; McNamara, 1996; Morrow, 1979; Savignon, 1972, 1983).

When we look back at the 1980s, researchers seemed to be interested in real-life testing, and they started to design the tests which include the real-life communication (Canale, 1983, 1984; Canale and Swain 1980; Morrow, 1979;

Savignon, 1972, 1983). In the 1990s, researchers started to investigate the relationship between the test methods and test performances, and this also provoked the development of English proficiency test (Bachman, 1990;

Bachman and Palmer, 1996; McNamara, 1996). Since the year 2000 until now, the language testing still focuses on the nature of language ability, why the language assessment is important, and how to interpret the test scores (Bachman, 2000; Bachman and Palmer, 2010).

3

Table 1.2 English language testing history

Year Researchers Language Testing Development

1980 Savignon (1972, 1983), Morrow (1979), Canale &

Swain (1980), and Canale (1983, 1984).

Language was reckoned to be a set of real life encounters and experiences and tasks, a view which took real life testing so seriously that it lost both objectivity and generality.

1990 Bachman (1990), McNamara (1996) and Bachman & Palmer (1996)

Bachman proposed an interactional model of language test performance in which language ability (language

knowledge and metacognitive strategies) is seen as interacting with test method (characteristics of the environment, rubric, input, expected response, and the relationship between input and expected response) to produce a performance that can be described and reported. It has provided a principled, systematic basis for the development of language tests, such as the English proficiency test.

Current Focus

Bachman (2000) Focus on:

-Methodology -Practical advances

-Performance-affecting factors -Performance assessment -Ethical issues

4

According to Table 1.2, we notice that the trends in language testing seem to follow the trends in language teaching since language teaching methodologists have developed and used language testing models (Giri, 2003; Hu, 2004; Jia and Yang, 2005; Klenowski, 2006). Moreover, Table 1.3 summarizes some of the types of assessments used with English language learners, and these assessments can be either formal or informal (Rasmussen, 2010). According to Rasmussen (2010), the informal assessments are often used to provide formative information while the formal assessments check program success or students’ progress.

Table 1.3 Types of assessments used with English language learners

Informal Formal

Alternative, Authentic, and Classroom-based

• Student Oral Language

• Student self-assessment

5

Recently, passing the English proficiency tests in school seems to become a way to support individual English ability, and the outcomes of such tests influence students’ learning and teachers’ instructions (Abedi, 2008; Leung and Lewkowicz, 2006; Rasmussen, 2010). The systematic testing provides data that assist educators in making decisions about the initial placement of students in instructional programs or advancing them to next levels, in identifying their needs, and ensuring that they meet educational goals. (Alberta Education, 2006;

Rasmussen, 2010). Also, the teachers can understand the learning outcomes and learning difficulties among students through the tests (Rowe, 2006; Sireci, Han and Wells, 2008).

However, while the educational institutions work hard to design English proficiency tests, there is still a question: do these tests fit the students’ ability?

How should test scores be interpreted (Leung and Lewkowicz, 2006)? Do teachers believe what the test tells about the students? Besides, the educational institutions in Taiwan work hard on creating English learning environment on campus, and the main purpose is to encourage students to use the facilities, like English self-study software (i.e. Rosetta Stone, Longman English Interactive, Focus on Grammar, etc.), English corner, and English lab (please see Fig. 1.1 and Fig. 1.2). Finally, it can increase students’ English learning motivation gradually.

But how can these educational institutions understand students’ progress after the students’ use the school assigned software? The relationship between the English software and the students’ test performances is suggested to be measured in an objective and quantitative way.

6

Fig. 1.1 English self-study center in a private university

Fig. 1.2 English learning environment in a private university

7

However, Taiwanese students do not know how to use the English facilities effectively, and most of them are just doing the “suggested software” on the student manual instead of doing the English software which really meets their needs. If the educational institutions could provide the “suggested learning path”

to each individual student, it will be more helpful for students’ English learning, and the suggested process is summarized in Fig. 1.3.

Fig. 1.3 Suggested learning path process

(Adapted from Education and Manpower Bureau, 2007)

According to Fig. 1.3, we can understand that as a teacher, it is important to understand the exam results and provide remedial instructions to the students.

In order to understand the test results, we need to view the test design process first. Therefore, Bachman and Palmer (1996) mentioned the six test qualities in testdesign process, and they are shown in Fig. 1.4.

English Teacher

• Evaluation students' English performances

Advisor

• Providing suggested learning path

Student

• Do the English software provided by the learning path

8

Fig. 1.4 Six test qualities

(Adapted from Bachman and Palmer, 1996)

If the test is too easy and most of the students get high scores on the test or it is too difficult while the class average is low, the test designer has to check the reliability and content of the test (Sheu, Chen, Tzeng, Tsai and Nagai, 2012;

Tzeng, Sheu, Liang, Wang and Nagai, 2012a~2012b; Wang, Sheu, Liang, Tzeng and Nagai, 2011a~2011b, 2012a~2012c). Otherwise, it is just waste of time taking the tests because it fails to test the students’ abilities; as a result, students receive inappropriate instruction (Abedi, 2008; Leung and Lewkowicz, 2006). Therefore, the author proposes the statistical analysis method, like the grey S-P chart, to cluster students’ performances and test item difficulties. Then teachers can provide remedial instructions or assignments based on students’

clustered groups, and review test content as well (Wang, Wang, Wen, Nagai and Liang, 2011; Wang, Sheu, Liang, Tzeng and Nagai, 2011a~2011b). In this paper, statistical analysis methods related to the grey system theory are used.

Reliability

Construct Validity

Authenticity

Interactiveness Impact

Practicality

9

Wen, You, Nagai, Chang and Liang (2010) and Wang et al (2011a) point out that the grey statistical analysis method not only quantifies the relationship between elements, but also presents the results in a concrete quantitative way.

Also, regarding to the characteristics of the grey system statistical analysis method, the evaluation must be as objective as possible which increases the reliability and validity of the test results.

In addition, to enhance the teaching effectiveness in classroom, it is an important issue for teachers to find out students’ misconceptions and provide remedial instructions effectively and quickly (Sheu, Tzeng, Tsai and Chen, 2012; Thompson and Logue, 2006; Tzeng et al, 2012a~2012b). Wang et al (2011b) also uses the grey system statistical analysis methods “GSP and GSM”

to provide adaptive English teaching instructions. The detailed process between English curriculum, teaching, English assessment and test evaluation are shown in Fig. 1.5.

Fig. 1.5 The relationship between English curriculum, teaching, English assessment and test evaluation

(Adapted from Leung and Lewkowicz, 2006)

English

10

Since developing the perfect test is the ultimate goal for test designers, looking at item difficulties may help them decide what is wrong with the test items (Suen and McClellan, 2003). In the past, researchers use the Classical Test Theory (CTT) to check the item difficulty (Magno, 2009; Morales, 2009).

That is, when over 90% of the students answered correctly on the item, it may be too easy, and the revision of the item is suggested. On the other hand, if there are less than 30% of the students answered the item correctly, the item may be too difficult, and it also needs to be revised (Kelley, 1939; Wang et al, 2012a).

Later, the Item Response Theory Model (IRT) has been more popular to look into item level information (Morales, 2009). Linacre (2002) pointed out that the IRT model is based on two assumptions: uni-dimensionality and local independence. In addition, IRT expresses that the item difficulty is influencing person responses while the item difficulty estimation is influenced by individual ability (Galdin and Laurencelle, 2010; Linacre, 2002). Then Chao, Kuo, Tsai, Lin and Nagai (2010) indicate that by using IRT, there has to be a large sample size for the item difficulty estimation. Thus, they use the Grey Relational Analysis (GRA) to analyze small amount data, and the results are also in line with IRT (Chao, Kuo and Tsai, 2010).

To summarize the current situation in English teaching and testing, there are some published papers so far (Alderson, 1991, 2004, 2005, 2007, 2010;

Alderson and Banerjee, 2001, 2002; Alderson and Huhta, 2005; Bachman, 2007; Bishop, 2004; Boyd and Davies, 2002; Broadfoot, 2005;

Chalhoub-Deville, 2003; Cohen, 2007; Leung, 2004; Lumley, 2002;

McNamara, 2003; Mislevy, Steinberg and Almond, 2002; Reath, 2004;

11

Solano-Flores and Trumbull, 2003; Song and Cheng, 2006). The traditional analysis methods, such as factor analysis method, or the proposed grey system statistical analysis method can be used (Wang et al, 2011a). However, the traditional analysis methods only deal with one-dimensional space which only presents the order of latent variables. Besides, it is to describe many factors with a small number of factors, that is, summarizing related variables into a category, making every category a factor (Wang et al, 2011a).

In this paper, the author proposes to use the grey system statistical analysis method, that is, the GRA to do the structural analysis of learning concepts in English and try to find out the objective method of English coursebook chosen.

By using GRA with GSP and GSM, it is possible to draw two-dimensional space figures which not only show the relationship between elements, but also present the hierarchy of each element (Wang et al, 2011a~2011b). According to Nagai, Yamaguchi and Li (2005), Yamaguchi, Li and Nagai (2005) and Yamaguchi, Li, Mizutani, Akabane, Nagai and Kitaoka (2007), the comparison of system modeling approaches can be summarized in Table 1.4.

12

Table 1.4 Comparison of system modeling approaches (Adapted from Nagai et al, 2005, p128)

Traditional Graphical Modeling

ISM GSM

Theoretical background

OT/IRS Boolean algebra Grey system

theory Input

element causality

Directly (0,1) Directly (0,1) Directly (0,1) / Indirectly

(observed value) Causality

type

Correlation Binary relation Grey relational analysis

Graph type Diagraph Diagraph Diagraph

Adapt hierarchy

No No Yes (, )

According to Table 1.4, the propose GSM method is superior to the other two methods because it is able to analyze direct and indirect values; moreover, its hierarchy is adaptive while the traditional graphical modeling and ISM fails to do adapt hierarchy.

13