This chapter presents the methodology of this study. It comprises five sections.

CHAPTER THREE METHODOLOGY

This chapter presents the methodology of this study. It comprises five sections.

First, a description of the participants is presented. The second and third sections illustrate the framework and development of the treatment and instruments implemented in this study. Finally, the procedures of data collection and the methods of data analysis are described.

3.1 Participants

The participants in this study were students enrolled in two second-year remedial classes in a vocational industrial high School in Taipei City. Compared with their peers in the school, the participants had lower language proficiency and a lower level of achievement, for their academic performance in English at the end of the previous semester was less satisfactory, and they had to retake the course during summer recess.

The Experimental Group, Class A, was composed of thirty-four male students.

Class B, serving as the Control Group, was composed of thirty-five males and two

females. Most of the participants were male, since this school recruited industrial

technology majors and male students were more interested in this field than females

in Taiwan. In addition, the participants’ average age was 17 and the average length of learning English was seven to eight years. Moreover, according to the collected data from the questionnaire, 50% of the participants in the Experimental Group reported that they had received test-taking strategy instruction on cloze tests in class before, whereas 64.9% of those in the Control Group had. In general, the two groups were comparable in terms of the characteristics mentioned above.

3.2 Treatment

The six-session test-taking strategy instruction was designed based on O’Malley

and Chamot’s (1988) scope and sequence framework for second language learning

strategy instruction. The sequence consisting of general guidelines for learning

strategy instruction were developed by O’Malley and Chamot as part of a content-

based elementary and secondary ESL program. This framework was modified by the

researcher of the present study with the same sequence, namely, preparation,

presentation, practice, evaluation, and expansion, but different activities in order to be

consistent with the traits of test-taking strategies, which also focused on learner

strategies as well as test management (Cohen & Upton, 2006). Table 3.1 illustrates the

modified framework for test-taking strategy instruction in the present study.

Table 3.1

Scope and Sequence Framework for Test-taking Strategy Instruction on Cloze Tests in the Present Study

1. Preparation: Develop student awareness of different test-taking strategies through:

- Identifying current test-taking strategy use

- Small group and class discussions on the benefit of using proper strategies 2. Presentation: Develop student knowledge about test-taking strategies by:

- Providing rationale for test-taking strategy use - Describing and naming strategies

- Modeling strategies

- Additional instruction on transitional words/phrases and cohesive ties

3. Practice: Develop student skills in using test-taking strategies through:

- Individual learning tasks

- Peer tutoring while solving a problem - Group discussions

4. Evaluation: Develop student ability to evaluate own test-taking strategy use through:

- Writing down strategies used immediately after a task - Discussing strategy use and sharing experiences in class

5. Expansion: Develop transfer of test-taking strategies to new tasks through:

- Discussions on metacognitive and motivational aspects of strategy use - Additional practice on similar tasks

1

The additional instruction was designed on the premise in the literature that intersentential

comprehension, textual sensitivity, and recognition of cohesive devices are targeted as overall language

proficiency and that cloze tests are a valid technique to measure this proficiency.

On the basis of the framework, the test-taking strategy instruction and practice took six 90-minute sessions. The syllabus is listed as follows.

Table 3.2

Syllabus of Test-taking Strategy Instruction on Cloze Tests in the Present Study Session One: Orientation and awareness-raising

- discuss and check the answers to the questions in the pre-test

- discuss the difficulties and benefits of using proper test-taking strategies - identify your own test-taking strategy use

Session Two: Presentation, practice, and evaluation (Part I)

- local/clause-level strategies to decompose individual sentences: look for key vocabulary or phrases, use syntax, and recognize collocation in the text Session Three: Presentation, practice, and evaluation (Part II)

- transitional words/phrases and cohesive ties

Session Four: Presentation, practice, and evaluation (Part III)

- global/text-level strategies to achieve overall comprehension: use discourse structure, read previous and subsequent text, and make an inference from context Session Five: Presentation, practice, and evaluation (Part IV)

- test-management strategies: reread, predict answers, use background knowledge, eliminate unreasonable options, make an educated guess

, and self-monitoring Session Six: Wrap-up and expansion

- review and share experiences with test-taking strategy use on cloze tasks - integrate test-taking strategy use into knowledge and make applications

2

All of the cloze tasks for practice were adapted from the reading passages of the participants’ textbook, Far-East New English Reader for Vocational High Schools, Book IV (see Appendix A-1, A-3, and A-4).

3

The material concerning lists and practice of transitional words/phrases was adapted from the teacher’s manual of Far-East English Reader for Vocational High Schools, Book V, and the material regarding cohesive ties was adapted from Chen’s (2003) teaching material for instruction on text structure (see Appendix A-2).

4

These strategies were adopted from Cohen and Upton (2006, p.36-37).

3.3 Instruments

Three instruments were used in this study to collect data: (1) the pre-test and the post-test of multiple-choice rational cloze tests, (2) a retrospective questionnaire on participants’ use of test-taking strategies for cloze tests prior to and after the treatment of test-taking strategy instruction, modeling and practice, and (3) a questionnaire on participants’ responses to the test-taking strategy instruction.

3.3.1 Pre-test and Post-test of Multiple-choice Rational Cloze

Both the pre-test and the post-test of cloze comprised four passages with five multiple-choice questions for each, which the researcher of the present study adapted from JCEE, SAT, EEFTC, and her own work (see Appendix B and Appendix C). In each test, three of the passages were appropriate for the participants’ proficiency level, and the other was moderately above their proficiency level in order to elicit their higher-order mental processing while taking the test. The topics of these passages in the tests were familiar ones for the participants. The difficulty level of each passage was measured based on the criteria of the computer software of Flesch-Kincaid Grade Level Score

, the number of words beyond the 1000 frequency level, the number of

5

The Flesch-Kincaid Grade Level rates a text on a U. S. grade-school level. The score means the

number of years of education generally required to understand this text. For example, a score of 8.2

would indicate that the text is expected to be understandable by an average student in 8th grade.

sentences, and the number of words in the passage. The topics and readability of each passage are listed in Table 3.3.

Table 3.3

Topics, Difficulty Level and Length of the Passages Used for the Pre-test and Post-test

Topic Flesch-Kincaid

Grade Level Score

No. of words beyond the 1000 frequency level

No. of sentences

No. of words

Pre-test Passage 1 Life:

Movie time

7.1 12 11 149

Passage 2 American Holiday:

Thanksgiving

7.5 18 13 146

Passage 3 Language: An interesting idiom

7.7 19 11 154

Passage 4 Famous People:

J. K. Rowling

8.3 20 10 147

Post-test

Passage 1 Travel:

Postcards

7.1 10 11 145

Passage 2 Life:

Online shopping

7.4 17 12 149

Passage 3 Culture: Different New Year customs

7.6 17 9 142

Passage 4 Taiwanese Holiday :

Lantern Festival

8.6 21 10 155

Additionally, the missing words were deleted using rational deletion procedure along with Bachman’s (1985) four deletion types: (1) within clause; (2) across clause, within sentence; (3) across sentences, within text; and (4) extra-textual. In the pre-test and post-test, the total number of items belonging to Type 1 was 11, 11 items belonged to Type 2, 17 items belonged to Type 3, and one out of the forty items was categorized as Type 4. The deletion procedure was consistent with Bachman’s argument that Type 2 and 3 should be maximized in order to measure the participants’

intersentential knowledge of textual and discourse structure. The categorization of these four deletion types followed Jonz’ (1990) interpretation, as illustrated in Figure 3.1.

Figure 3.1 Jonz’ (1990) Interpretation of Bachman’s (1985) Categories

Can closure be affected if the clause is isolated from text?

Can closure be resolved with support of textual clues outside the clause?

Is the clue within the same orthographic sentence?

Type 3

“across sentences, within text”

Type 1

“within clause”

Type 4

“extra-textual”

Type 2

“across clause, within sentence”

NO

NO

NO YES

YES

YES

The comparable distribution of rational deletion types of the pre- and post- cloze tests is presented in Table 3.4. Moreover, three distractors, considered plausible for primary-intermediate level students and of the same part of the speech as the correct answer, were provided for each blank.

Table 3.4

Distribution of Deletion Types of the Pre-test and Post-test No. of Type 1

(within clause)

No. of Type 2 (across clauses, within sentence)

No. of Type 3 (across sentences,

within text)

No. of Type 4 (extra-textual)

Pre-test

Passage 1 1 1 3 0

Passage 2 2 2 1 0

Passage 3 1 0 4 0

Passage 4 2 3 0 0

Total 6 5 9 0

Post-test

Passage 1 2 1 1 1

Passage 2 1 1 3 0

Passage 3 0 3 2 0

Passage 4 2 1 2 0

Total 5 6 8 1

3.3.2 Questionnaire on Participants’ Use of Test-taking Strategies for Cloze Tests

The same questionnaire was used to probe the participants’ cognitive and metacognitive awareness of their strategy use in taking cloze tests before and after the test-taking strategy instruction to see if there would be any change caused by the treatment (see Appendix D). The questionnaire employed in this study made reference to Joe’s (1993, 1994) research results of Taiwanese EFL students’ reading strategies on cloze tests, Phakiti’s (2003) questionnaire measuring learners’ cognitive and metacognitive strategies, and Cohen and Upton’s (2006) data analysis of test takers’

strategies used with different item types on the reading section of the LanguEdge Courseware material. These analyses and descriptions on strategies were then adapted by the researcher of the present study to suit the traits of a cloze test. All the answers in the questionnaire were constructed in a five-point Likert scale ranging from

“never,” to “always.” The statements in the questionnaire were presented to the

participants in their first language, Mandarin Chinese, for better understanding and

ease of answering. Table 3.5 presents the English version of the questionnaire on

participants’ test-taking strategies employed in this study.

Table 3.5

Statements of the Questionnaire on Test-taking Strategies for Cloze Tests (English) When I am taking a cloze test,

1. I read the whole passage first.

2. I read every word and sentence carefully.

3. I look for key words or phrases.

4. I make use of syntax or punctuation.

5. I read words, phrases, or sentences silently.

6. I use known phrases or collocation.

7. I translate words, phrases, or sentences into Chinese.

8. I look for the relationship of sentences (e.g., conjunction or transition).

9. I go back in the passage to review or understand information.

10. I read ahead to look for information.

11. I make an inference based on the context.

12. I use my prior knowledge.

13. I predict my own answer before looking at the options.

14. I choose an option that is familiar to me.

15. I choose an option that is different from others.

16. I eliminate unreasonable options.

17. I postpone answering a question when encountering difficulty.

18. I reread the passage to look for clearer clues to the answer.

19. I resort to my intuition to decide an answer.

20. I make a wild guess.

21. I reconsider or double-check my answer.

After I take a cloze test,

22. I know the main idea of the passage.

23. I am aware of the strategies I use.

24. Do you use any strategies other than the above statements?

Please describe it/them here.

3.3.3 Questionnaire on Participants’ Responses regarding the Test-taking Strategy

Instruction

The third instrument was a questionnaire designed by the researcher of the present study to explore the participants’ responses to the six-session test-taking strategy instruction (see Appendix E). It was made up of two parts and all the statements were presented in Chinese. The first part contained ten items constructed in a five-point Likert scale ranging from “strongly disagree,” to “strongly agree.” The second part consisted of two open-ended questions intended to engage the participants to make a self evaluation of improvement and difficulty they have experienced during the instruction session. All the statements and questions in this questionnaire, as presented in Table 3.6, were designed to tap the participants’ retrospection on the experience in the test-taking instruction program.

Table 3.6

Statements of the Questionnaire on the Responses regarding the Test-taking Strategy Instruction (English)

Part One

Learning test-taking strategies helps me

1. increase my reading speed when taking a cloze test.

2. enhance my cloze test performance.

3. understand the main idea of the passage effectively.

4. reduce the anxiety when I take a cloze test.

5. increase my interest in learning English.

Table 3.6 (continued)

6. Learning test-taking strategies may help me solve English tasks in addition to cloze tests.

7. I will benefit in taking cloze tests if I do more test-taking strategy practice.

8. I will suggest that teachers provide us with test-taking strategies for cloze tests in class.

9. I am more confident in solving cloze tasks after practicing test-taking strategies in this class.

10. I am willing to continue using test-taking strategies taught in class when encountering cloze tasks.

Part Two

11. Did you encounter any difficulties during the activities of learning and practicing test-taking strategies? Please describe it/them.

12. What benefited you most during the activities of learning and practicing test-taking strategies?

3.4 Procedures of the Instruction and Data Collection

The detailed steps of procedures of the instruction and data collection are described as follows in three stages: before, during, and after the implementation of the study.

3.4.1 Before the Implementation of the Study

Before the main study, six students, excluded from the pool of the participants

but of the same grade, were recruited to complete the cloze tasks for the pre- and post-

tests and also the questionnaire concerning their test-taking strategy use. In order to

get a result comparable with the main study, the students were chosen from those who got a score only a little higher than the requirement. After examining the performance of these six students, several distractors of the cloze tasks were revised or changed, and a couple of words in the passage were replaced by those within the 1000 frequency level. In addition, the students’ responses to the questionnaire were taken into consideration. Three students mentioned that they didn’t understand one of the statements in the questionnaire, “I use paragraph structure to arrive at my answer.” In order to avoid possible misunderstandings, this statement was discarded. Finally, two experienced English teachers were invited to proofread all the instruments used in the present study. The purpose of this procedure was to ensure a comparable difficulty level of the two cloze tests, and to ensure that the participants in the main study would have no difficulty understanding the statements in the questionnaire.

3.4.2 During the Implementation of the Study

Since the participants came from two classes, the data were collected separately,

and the interval for the collection of equivalent data from the two groups was two

days for some practical reasons. The participants of both groups were not informed of

the aims of the study for the purpose of preventing any bias. After the cloze pre-test

and the questionnaire on test-taking strategies were administered, the Experimental

Group received the test-taking strategy instruction conducted by the researcher herself, while the Control Group received normal instruction for remedial classes given by another teacher.

During the implementation of the study, there were six 90-minute instruction sessions over a three-week span. The teaching procedures followed the syllabus illustrated in Table 3.2. The researcher presented the strategies in a direct and informed way. In each session, four to six target strategies were first explicitly introduced and modeled for the participants in the Experimental Group. For instance, in order to demonstrate how to arrive at the correct answer to a question by using the strategy of predicting, the researcher employed the think-aloud method. Take Question 8 in the practice of global/text level strategies for example (see Appendix

A-3):

In North America, the custom is to take the other person’s hand tightly. But in

the Middle East, the custom is to take the other person’s hand 8 .

8. (A) gently (B) likely (C) hardly (D) safely

The researcher read the sentences out, and said, “The two sentences are opposite in

meaning because of the conjunction ‘but’. So the answer should be the word with the

opposite meaning of ‘tightly’, such as ‘loosely’, ‘freely’ or ‘gently’.” Moreover, this

teacher modeling was followed by detailed explanation of word meaning and

sentence structure, usually in Chinese for ease of understanding.

After teacher introduction and modeling, the students engaged in practice either individually or in small groups and were encouraged to apply these strategies appropriately. This was followed by students’ self-evaluation of their use of the strategies. In addition, classroom activities such as brainstorming, collaborative learning, and group competition were used to increase the students’ motivation.

3.4.3 After the Implementation of the Instruction

Immediately after the instruction sessions, the cloze post-test was given to the Experimental Group and Control Group. Furthermore, the participants in the Experimental Group were asked to fill out two questionnaires. The first one was the same questionnaire on test-taking strategies for cloze tests, which was aimed to investigate the change caused by the instruction. The other one was the questionnaire on the participants’ responses for the purpose of revealing their attitudes toward the instruction.

In sum, the procedures for data collection in the main study were as follows: (1)

the pre-test and the questionnaire on participants’ use of test-taking strategies for

cloze tests before the treatment, (2) the six-session treatment of test-taking strategy

instruction and practice, (3) the cloze post-test and, (4) the questionnaire on

participants’ use of test-taking strategies for cloze tests after the treatment and the

questionnaire on participants’ responses regarding the test-taking strategy instruction.

3.5 Data Analysis

Quantitatively, by operating in the SPSS program for Windows, t-test was

applied to analyze the results of the cloze pre-test and post-test to compare intra- and

inter-group differences. Besides, the questionnaires on the participants’ test-taking

strategy use were analyzed by employing descriptive statistical methods. In order to

find out the change caused by the treatment, a Paired Samples T-test was utilized to

determine whether participants’ use of test-taking strategies was significantly

different after the treatment. Furthermore, data collected from participants’ response

questionnaire were sorted with frequency to investigate participants’ responses to the

experience of the test-taking strategy instruction program. Finally, based on all the

participants’ answers to the questions of pre- and post- cloze tests, item quality was

assessed on the basis of discrimination index for the purpose of analyzing the

participants’ performance on different types of cloze deletions.