CHAPTER THREE RESEARCH DESIGN

(1)

CHAPTER THREE RESEARCH DESIGN

In this chapter, the research design is discussed in detail. Section 3.1 presents the information about the subjects. Section 3.2 introduces the research instruments, including two listening tests, a questionnaire, and an interview. Section 3.3 reports the procedure of the study. Finally, section 3.4 summarizes the main points of this chapter.

3.1 Subjects

Ninety sixth graders in Wanfang Elementary School in Taipei City participated in the study. They have had the English classes in school since Grade 2. But some students started their English learning outside school since kindergarten, as can be seen in Table 3-1:

Table 3-1 Basic Information of the Subjects

Group¹ A B C

Time Started English Learning

Before Elementary school

Grades 1-2 After Grade 3

Average Number of Years of Learning

7 years 4.5 years 2.6 years

Male 18 18 19

Gender

Female 12 12 11

Number 30 30 30

Ten subjects of Group A learned English for eight years, ten for seven years, and ten for six years, so the average number of years of learning was 7 years. Group B started to learn English from Grade 1 or 2 and half of the subjects learned English for five years and the rest for four years. Thus, their average number of years of learning was 4.5 years. Twenty subjects of Group C learned English for three years, eight for two years, and one for one year, so the average number of years of learning was 2.6 years.

1 The grouping is based on the subjects’ starting point of learning English.

(2)

The ratio of male and female students was 11: 5. They have two periods of English in school every week. Their textbooks were Darbie, Teach Me!, published by Kang-hsuan Co., Ltd. Besides the formal English courses, their English teachers read them stories in class.

The reason to choose elementary school students as the subjects in this study was because the researcher was interested in primary English education. Moreover, since one of the variables investigated in the study was the degree of topic familiarity, it would be better for the subjects to study the same set of materials to see if topics taught earlier were indeed easier for the subjects. Since Wanfang Elementary School was one of the schools which chose the textbooks designed by the same publisher, students in the school were the first choice for the researcher. Besides, since low graders have not learned much English and are not capable of expressing their ideas well in the interview, sixth graders were a better choice to be the subjects of the present study.

3.2 Instruments

Three instruments were used in the present study: two listening tests, a questionnaire, and an interview, which will be presented in section 3.2.1, section 3.2.2, and section 3.2.3 respectively.

3.2.1 Listening Tests

The listening tests were in different formats (a textual format as in Appendix 1, with the script and the answer as in Appendices 2 and 3 respectively, and a pictorial format as in Appendix 4, with the script and the answer as in Appendices 5 and 6).

The listening materials were adapted from the CDs of the textbook and recorded by

(3)

two graduate students of the English Department of NTNU² at the speed of 100 words per minute (wpm), which is a slow speech rate according to Pimsleur et al.

(1977)³. Table 3-2 presents the basic structure of the two listening tests.

Table 3-2 Basic Structure of the Listening Tests

Listening Test Test 1 Test 2

Test Format Textual Pictorial

Part Question No. Part Question No.

Section

Vocabulary Sentence Conversation

8 8 8

Vocabulary Sentence Conversation

8 8 8

Total 24 24

The major difference between Tests 1 and 2 was that Test 1 had no pictures in it while Test 2 was accompanied with visual aids⁴. The content was taken from Books 1 to 5 of Darbie,Teach Me! To investigate the effect of the hierarchy of topic familiarity, the target topics and vocabulary words chosen were strictly controlled. In the teacher’s manual, the editor states that the materials designed in the former volumes would be recycled in the later volumes. After a careful examination, the researcher found that the chosen topics were found in other volumes: the chosen topic of Book 1 (Family Members) was recycled in Book 2 (Lesson 2, Is Your Mother A Teacher?), Book 3 (Where’s Grandpa?), and Book 5 (in the Review Corner of Lesson 4, I Am Brave). Similarly, the topic taken from Book 2 (Fruit) was also recycled in Book 3 (in the Review Corner of Lesson 5, Is This Pencil Eleven Dollars?) and in Book 5 (in the

2 To see if the aural input of native or non-native speakers made a difference on the subjects’

performance, items were analyzed to compare the results. For example, in Test 1, Items 3 and 6 in Part III (the Conversation section) was taken from CD and recorded by two non-native speakers respectively.

The results showed that there was no significance between the subjects’ performance (t (89)= -1.00, p= .320). A similar situation was found in Test 2. Items 3 and 7 in the Conversation section were recorded by two non-native speakers and taken from CD respectively. However, the subjects’

performance was not different significantly (t (89)= .00, p= 1.000)

3 According to Pimsleur et al. (1977), speech rates could be categorized into five types: fast (above 220 wpm), moderate fast (between 190 and 220 wpm), average (between 160 and 190 wpm), moderately slow (between 130 and 160 wpm), and slow (below 130 wpm)

4 The pictures used in the listening tests were taken from the textbooks of Darbie, Teach Me!, and downloaded from the Internet. Special thanks go to be given to the two sources.

(4)

Review Corner of Lesson 6, What Do You Eat for Chinese New Year?) and in the later volumes as well. Therefore, the two topics selected from Books 1 and 2 (see Table 3-3 for further information) were assumed more familiar and the other two topics taken from Books 4 and 5 are less familiar (see Table 3-4 for further information), since they were not recycled as often as the previous ones.

Table 3-3 More Familiar Topics in the Tests

Topic Book Lesson Vocabulary Note

father mother brother

Family Members 1 2

sister grandma grandpa uncle Christmas

(Family Members)

1 5

aunt

Test 1: Part I: 4 & 8 Part II: 3 & 8 Part III: 1 & 7

apple orange lemon banana

Fruit 2 4

tomato

(5)

Table 3-4 Less Familiar Topics in the Tests

Topic Book Lesson Vocabulary Note

jogging fishing shopping swimming camping Hobbies; Habit

(Activity)

4 5

climbing

Test 2: Part I: 4 & 8 Part II: 1 & 3 Part III: 3 & 7 fish

chicken dumpling pork duck beef noodles Chinese New Year

(Food)

5 6

rice cake/

niangao

In both listening tests, there were three sections, each of which investigated the effect of a particular text type. There were eight questions in each section. Each question consisted of three options. Basically, the subjects were asked to check, choose, or match the correct answers after they listened to the tape.

Part I was the Vocabulary section. The subjects listened to words in isolation and they had to identify the correct answer as shown in Table 3-5, like the following example taken from Test 1:

Table 3-5 A Sample Question for Part I of Test 1 Script: banana

Question:

banana lemon apple

(6)

The subjects had to put a check (v) under ‘banana.’

Part II was the Sentence section. The subjects had to listen to the patterns and choose the correct answer, as shown in Table 3-6, an example taken from Test 2:

Table 3-6 A Sample Question for Part II of Test 2 Script: They are going camping.

Question:

( ) 1.

The last part, Part III, was the Conversation section. After listening to the dialogue between two speakers, the subjects had to infer or judge which the right answer was. Table 3-7 is an example of Test 1:

Table 3-7 A Sample Question in Part III of Test 1 Script: A: Do you eat beef for Chinese New Year?

B: No, we don’t.

Question:

fish for Chinese New Year.

beef for Chinese New Year.

The boy doesn’t eat

dumplings for Chinese New Year.

3.2.2 Questionnaire

To obtain students’ opinions and other thoughts about the two tests, a questionnaire (see Appendix 7) was used to collect their voices. Table 3-8 is a structure of the questionnaire:

(7)

Table 3-8 Basic Structure of the Questionnaire

Section Category Number Question No.

I Personal Information 2 Qs 1-2

Visual aids 2 Qs 1a-2b

Text types 2 Qs 2a-2b

II

Topic familiarity 1 Q3

The questionnaire consisted of two parts. Part I was about the subjects’ personal information such as their names and the time that they started to learn English, etc.

Part II asked their opinions about visual supports, different text types, and familiarity of certain topics. In this section, the subjects had to first choose (check actually) the best answer and then write down their reason(s). For example, Question 1a in Section II asked the subjects’ opinions about the visual support and Question 1b in Section II asked their preference for tests with or without visual aids.

3.2.3 Interview

In order to avoid the inadequacy of the above two instruments, besides the quantitative data collected from the two listening tests and the questionnaire, an interview was conducted to elicit more thorough comments from the subjects.

The interviewees were randomly selected from the three groups. Six subjects, including three high achievers and three low achievers⁵, were chosen from each group.

Therefore, there were eighteen interviewees in total. The reason that the researcher chose the interviewees was to see whether the subjects of the three groups and at different levels had different opinions. Table 3-9 summarizes the personal information of the interviewees:

5 High achievers were subjects who scored on the top in the group. On the other hand, low achievers were those who scored at the bottom in their group.

(8)

Table 3-9 Interviewees’ Scores on Listening Tests

Group Interviewee Gender Test 1 Test 2

A2 F 24 24

A3 M 22 18

A5 M 19 22

A11 M 24 24

A14 F 24 24

A

A17 M 18 22

B1 M 24 24

B2 F 24 24

B3 F 24 24

B19 F 19 20

B21 M 19 19

B

B30 M 18 17

C2 M 16 16

C8 M 17 18

C14 F 24 24

C20 F 24 24

C26 F 20 20

C

C30 M 24 24

The interview questions (see Appendix 8) aimed to collect the subjects’ perception about some variables like topic familiarity. The questions are summarized in Table 3-10:

Table 3-10 Guided Questions Used in the Interview

Section Category Number Question No.

1 Visual aids 2 Qs 1-2

2 Text types 2 Q3

3 Topic familiarity 3 Qs 4-6

4 Number of years of learning 4 Qs 7-10

There were ten interview questions in total. Related questions about the four variables were included in the lists. Generally speaking, the interview questions were of four categories. The first category was about visual aids. The subjects were asked

(9)

questions about visual aids to gather in-depth information that they might not report in the questionnaires. The second category mainly asked the subjects questions about different text types in the listening tests. The third category was about topic familiarity.

Since in the questionnaire the subjects only had to circle the level of familiarity according to their perception, the three questions could elicit the underlying meaning of their choices. As to the last category (i.e., the number of years of learning), the researcher wanted to see whether the subjects’ English learning habit differed because of their exposure. Thus, four questions were asked about their learning habits.

3.3 Procedures

In this section, details of a pilot study and the formal test will be presented in sections 3.3.1 and 3.3.2 respectively.

3.3.1 Pilot Study

Before the formal test, a pilot study was administered to detect the difficulty level of the listening tests, to elicit the subjects’ feedback on the test design and questionnaire, and to understand the problems they encountered while they took the listening tests. At Yi-fang Elementary School in Taipei County, the researcher conducted a pilot study with 26 fifth graders. The subjects took the listening tests and finished the questionnaire on June 27, 2005. The collected data were analyzed by SPSS.

The results showed that the subjects performed differently on two different formats of listening tests (textual and pictorial) and on different parts of listening tests (vocabulary, sentence, and conversation). They performed significantly better (p=.003) when they took Test 2 (i.e., the pictorial format). As to the influence of text types, the results of one-way ANOVA showed that there was a significant difference at the .01

(10)

level between the three text types in both tests (Test 1, p=.003; Test 2, p=.006). After a further analysis of Tukey HSD for post hoc, the contrast between the Vocabulary and Conversation sections and between the Sentence and Conversation sections in Test 1 achieved a significant difference at .05 level (Vocabulary vs. Dialogue: p=.002;

Sentence vs. Dialogue: p=.04). However, in Test 2, only the contrast between the Vocabulary and Sentence sections achieved a significant difference at .01 level (p=.006). Therefore, in non-picture format, the students performed significantly better on the Vocabulary section than on the sentence patterns and dialogues. Yet in pictorial format, vocabulary became the most difficult section compared to the sentences and dialogues. As to the hierarchy of topic familiarity, the students performed significantly better on the more familiar topics than the less familiar topics. Finally, though the number of students in each group was not the same, the results of one-way ANOVA showed that time of exposure did not make a significant difference in both tests.

The questionnaire provided a qualitative view of students’ opinions about the two listening tests. Question 1a (helpfulness of pictures) and Question 1b in Second II (preference for test formats) were significant after the Chi-Square analysis. The subjects (85 %) thought pictures were helpful, and they also preferred listening tests with pictures (92.31 %). From Question 2 in Second II, the results of Chi-square showed that the subjects thought Conversation of Test 2 (in pictorial format) was more difficult than the other two while their opinions about the three parts in Test 1 were not different significantly (in textual format).

However, since the subjects did not use Darbie, Teach Me! as their textbook, we cannot infer from the results whether the degree of topic familiarity was crucial in deciding the students’ performance. Besides, the inconsistent results of test types also require further investigation to see whether text types really make a difference in

(11)

cannot address the fourth research question. Besides, the pool of subjects is too small to be generalized to all elementary students. More participants are needed for the formal study.

3.3.2 Formal Testing

The formal test was conducted at the end of September, 2005. The subjects took both listening tests in class. They first finished Test 1 and then Test 2 to avoid the effect brought by the visual supports. It took them about 15 minutes to complete each test. After finishing the tests, they needed to fill in the questionnaire. The whole process took them approximately 30 minutes.

After scoring the test sheets, the researcher grouped the subjects into three according to the number of years of learning English. Then one week after the formal test, the researcher visited the school again for interviewing some subjects. Six interviewees, 3 high achievers and 3 low achievers, were randomly chosen from each group. Totally, eighteen subjects were interviewed. The interview was conducted during the subjects’ lunch break. Each subject was interviewed for 10 minutes. The interviews were recorded via an MP3 player and later transcribed and translated by the researcher.

3.3.3 Scoring and Data Analysis

Since the tests were in multiple-choice format, each correct response was given one point. The subjects could get 8 points in total for one section, and 24 points for one test.

Later, after grading the test sheets, the scores were keyed into an SPSS file to find out the effect of each variable. Paired t-test was used to analyze the effect of visual aids. One-way ANOVA was applied to examine the differences between three

(12)

different text types, the hierarchy of topic familiarity, and the number of years of learning. For the post hoc, the Turkey HSD method was used. The questionnaire results were presented with frequency and calculated by Chi-Square and one-way ANOVA.

3.4 Summary of Chapter Three

In this chapter, the information of the subjects and the design of this study have been reported in details. Instruments used in the present study and the procedure have also been introduced. The results of the present study and the discussions about the results will be presented in Chapter Four.