Research Design

(1)

CHAPTER THREE METHOD

In this chapter, the method employed for the present study will be described. In the first section is the research design, which gives an account of the general

description of the participants as well as the procedure and the setting of the study. An account of the classroom procedure is provided in the second section. The data

collection procedures are then described. The instruments employed for the study are described in the last section.

Research Design

This study was designed to investigate the effect of multi-draft writing procedure on students’ writing ability and the change of their attitudes toward revision and writing. To achieve this purpose, the participants were divided into two groups: the control group (the single-draft group), who were not required to do any revision on their writing and the experimental group (the multi-draft group), who were required to revise twice on each composition topic they wrote.

At the beginning of the study (in early September, 2003), a writing pre-test was administered. The participants were given 60 minutes to write on the topic “How I Relieve My Pressure.” They were also asked to fill in an attitude questionnaire, which was aimed to investigate their attitudes toward revision as well as writing in general (See Appendix A for questionnaire items). At the end of the study (in mid-December, 2003), a writing post-test was given and the participants were given 60 minutes to finish their writing. The topic was “How I Obtain Knowledge.” After the post-test, a second attitude questionnaire was administered. This second questionnaire contained

(2)

all the items in the first questionnaire and some extra questions probing students’

attitudes toward treatment.

Between the pre-test and the post-test, the single-draft group and the multi-draft group received different writing treatments. The participants in the single-draft group were required to write nine expository compositions—each time they wrote on a new topic. The participants in the multi-draft group were required to write on only three topics, and for each topic they had to produce three drafts. The design of the study is illustrated in Figure 1.

Figure 1. The Procedure of the Study

September, 2003 Writing pre-test: How I Relieve My Pressure Pre-test questionnaire

September through December Treatment

(Different Writing Procedures)

Multi-draft group:

Instruction + Writing + Revision Single-draft group:

Instruction + Writing

December, 2003 Writing post-test: How I Obtain Knowledge Post-test questionnaire

Setting

The present study was conducted in National Taishan Senior High School in Taipei County. The research site was chosen because the researcher decided to recruit members from her own English classes. This decision was made following Zamel’s (1983) and Perl’s (1980) suggestion that recruiting one’s own students enables one to

(3)

know firsthand what the students are learning and to develop with them the rapport and trust needed for the study.

Participant

The participants included 77 high school seniors from two intact classes, one class containing 45 female students, the other class containing 45 male students.

These students were averagely aged seventeen and they all had learned English for at least 5 years, three years in junior high school and two years in senior high school.

Since the two intact classes were of different genders, to eliminate any potential influence of gender, the experimental group and the control group were designed to include both female students and male students. The students were grouped using the following procedures (See Table 1): Students in each class were randomly assigned to two groups. That is, the female students were divided into group A and group B; the male students were divided into group C and group D. Group A (22 female students) and group C (23 male students) formed the multi-draft group; group B (23 female students) and group D (22 male students) formed the single-draft group. In this way, both the experimental group and the control group consisted of female students and male students.

(4)

Table 1

The Number and Gender of Participants

Group A 22 Class I

Female students N = 45

Group B 23

Multi-draft group (Experimental group)

N :

42

= 45 - 3

Group C 23 Class II

Male students N = 45

Group D 22

Single-draft group (Control group)

N:

35

= 45 -10

As can be seen in Table 1, the two recruited intact classes actually consisted of 90 students in total. However, three students in the multi-draft group and ten students in the single-draft group were excluded from the final analysis because they either didn’t take the pre-test or the post-test or didn’t hand in all the nine writing

assignments. As a result, in the study, the multi-draft group consisted of 42 students and the single-draft group consisted of 35 students.

Writing Procedures

From early September to mid-December, 2003, the students in both the

multi-draft group and the single-draft group received a total of nine class meetings of writing instruction and writing practice. Each meeting was composed of two class periods, each lasting 50 minutes. In the first period, the students were given instruction on writing; in the second period, the students in the single-draft group wrote a composition and students in the multi-draft group revised their previous compositions. The instruction, writing practice, and the teacher feedback together formed the writing procedure in the study.

(5)

Classroom Instruction

For the nine writing instruction periods, a variety of activities and exercises were designed and used, including peer feedback, student composition sharing and analysis, brainstorming, revision exercises and exercises on the topic sentence, coherence and transitional words. These activities and exercises were intended to improve students’

composition in organization, content, grammar, vocabulary and mechanics. The two groups (the single-draft group and the multi-draft group) in a class together

participated in the instruction.

Peer Feedback

Peer feedback was conducted as the most important activity in the writing instruction sessions. All students were required to exchange their compositions with their peer writers. They shared their compositions and gave feedback to each other.

While the activity was proceeding, the teacher walked among the students as a facilitator to enhance more enthusiastic discussion.

The length of time for peer feedback was flexible, ranging from 20 minutes to 40 minutes, depending on how much time was left after the instruction activities and exercises for the week. Peer feedback was included as one of the instruction activities in the present study because researchers claim it makes students more aware of the audience perspective and thus enables the students to improve their content and organization (Keh, 1990; Mendonca & Johnson, 1994; Mittan, 1989; Zamel, 1982).

Yet, different from the previous researchers, who suggested the teacher pair up the students for the peer review sessions (See for example, Mittan, 1989; Paulus, 1999), the students in this study were allowed to share their writing with any students they felt comfortable with and they could even change partners. There were two reasons

(6)

why peer feedback was conducted this way. As Mangelsdorf (1992) indicated, when students’ ability varied a great deal, it was not easy to pair up students fairly. Usually better students gave good feedback to the weaker students but got little feedback in return. Another reason was that students in the study were beginning writers and one student was not able to offer much feedback. Students in the study were encouraged to exchange their writing with more peers so as to get richer comments and see various perspectives. Given the freedom to choose their partners, students could work with different people and feel more relaxed. During these peer feedback activities, the students did not have to fill out any evaluation form; instead, they simply enjoyed reading others’ works, having others read their work and giving and getting oral feedback.

Student Composition Sharing and Analysis

In the writing instruction periods, the teacher used well-written compositions by the students as model compositions. The students were guided to appreciate the strengths in their peers’ compositions, such as rich content, clear organization,

sophisticated sentence structures, precise vocabulary or correct grammar. At the same time, the teacher showed the students how to further improve these compositions if there were any problems. For each instruction periods students usually read five to six compositions but discussed only three to four in detail due to time limit.

Assorted Exercises

I adopted some exercises from Far East English Composition for Senior High School by Maosung Lin (2000) to help students get familiar with the concept and usage of topic sentence and transitional words (See Appendix I). Also, I used students’

compositions and a symbol list to explain how to make revision (See Appendix J).

(7)

Brainstorming

Each time, at the end of the instruction, the teacher guided the whole class to brainstorm to get ideas for the new topic for roughly five minutes. First, the teacher explained the topic. Next, the teacher asked the students to contribute their ideas and examples and then asked them how these ideas and examples could be organized.

Writing Practice

The second period of the nine meetings was reserved for writing practice. Both groups (the multi-draft group and the single-draft group) in a class together received the instruction in the first period, but in the second period, the two groups were required to go through different writing procedures. A comparison of different writing procedures for the two groups were presented in Table 2.

(8)

Table 2

The Two Different Writing Procedures for Two Groups

Meeting Single-draft Group Multi-draft Group

1 Topic 1

How I Relieve My Pressure

Topic 1 Draft 1 How I Relieve My Pressure

2 Topic 2

My Ideal Husband

Draft 2

Revise the organization and content

3

Topic 3

How I Am Going to Spend My College Life

Cycle I

Draft 3

Revise the grammar, vocabulary and mechanics

4 Topic 4

The Joy in My Life

Topic 2 Draft 1 The Joy in My Life

5 Topic 5

If I Can Travel Abroad

Draft 2

6 Topic 6

Difficulties I Have in Learning English

Cycle II

Draft 3

7

Topic 7

The Strengths and Weaknesses in My Personality

Topic 3 Draft 1

The Strengths and Weaknesses in My Personality

8 Topic 8

The Season I Like Best

Draft 2

9

Topic 9

The Importance of ____in Our Society (Students decide the missing word)

Cycle III

Draft 3

As shown in Table 2, the students in the single-draft group wrote on a new topic each week so they wrote nine different compositions in total. But the students in the multi-draft group only wrote on three different topics and for each topic, they

produced three drafts. If the writing and rewriting on one topic constituted one writing cycle, then the multi-draft group had gone three three writing cycles. For the first writing practice in each cycle, the students in this group wrote on a new topic. For the second practice, they revised the organization and content; for the third, they revised

(9)

the grammar, vocabulary and mechanics. During their writing, students, when confronting with difficulties, were encouraged to consult any source of information available for suggestion: dictionaries, their classmates and the teacher.

Teacher Feedback

The teacher was the only reviewer who wrote comments on students’

compositions. The rationale of having the teacher as the major reviewer was based on the assumption that students benefited more from teachers’ feedback than from their peers’ (Chen, 1997; Yagelski, 1995).

For compositions written by the students in the single-draft group, the teacher marked them with symbols, questions and explanations--all grammatical, vocabulary and mechanical errors. The teacher also gave comments on the organization and content. Students in this group were not required to make any revision though they were encouraged to correct their own errors in their composition.

By comparison, for compositions produced by the multi-draft group, the teacher gave comments on content and form in different drafts. In draft one, the teacher provided comments only on global aspects -- organization and content. In draft two, the teacher focused on local errors --grammatical, vocabulary and mechanical errors.

In draft three, only general encouragement, praises and criticism were given. Students in this group had to revise based on the teacher feedback.

Data Collection Procedures

The present study was to probe into the effects of different writing procedures on students’ writing ability and attitudes towards revision and writing. A writing pre-test and a writing post-test were administered to examine the change in student’s writing

(10)

performance. Two attitude questionnaires were administered at the beginning and the end of the study to investigate students’ attitudes toward revision and writing.

Writing Pre-test and Post-test

The writing pre-test was administered in the first of the nine meetings and lasted for 60 minutes. All the participants were required to write on the topic: How I Relieve My Pressure. When the pre-test was done, all the compositions were collected and photocopied. The post-test, also lasting for 60 minutes, was administered after the participants had finished the nine writing tasks. Likewise, after the post-test was done, all the compositions were collected and photocopied.

During the composition pre-test and post-test, the students could neither discuss with anyone nor ask the teacher any questions. They could, however, refer to their dictionaries. When time was up, all the students, whether they had finished writing or not, had to turn in their composition.

The intact compositions, rather than typed duplications, from the pre-test and the post-test were mixed and coded to disguise the writer’s identity. These compositions were then graded by two experienced raters.

Attitude Questionnaires

Two attitude questionnaires were developed by the researcher (See Appendix A and C for questionnaire items). The two questionnaires ( the pre-test questionnaire and the post-test questionnaire) were intended to explore students’ attitudes toward writing and revision before the treatment and after the treatment. The pre-test questionnaire, containing four items, was administered to all the participants after the writing pre-test.

(11)

The post-test questionnaire, containing 15 items, was administered after the writing post-test.

Instruments

The instruments employed for the present study included a rating scale and two attitude questionnaires—a pre-test attitude questionnaire and a post-test attitude questionnaire.

The two attitude questionnaires were designed to investigate the students’

attitudes toward writing and revision. In order to avoid confusion, the researcher used the Chinese version of questionnaires.

The Rating Scale

The rating scale was devised to evaluate the quality of students’ composition writing in the writing pre-test and the post-test.

The grading of the compositions in the present study was based on a rating scale adapted from the one used in Joint College Entrance Examination (JCEE) (See Appendix F). The scale includes five components: content (5 points), organization (5 points), grammar (4 points), vocabulary (4 points) and mechanics (2 points). The full score is 20. According to the JCEE rating scale, compositions are divided into four levels of proficiency: Excellent (19-20), Good (15-18), Fair (10-14), Weak (5-9), Poor (0-4).

The rating scale in the study was different form the original scale in two respects.

First, the original rating scale was intended for holistic grading, but in the present study, in order to compare students’ performance in each of the five components, the rater were asked to grade the compositions analytically by giving each of the

(12)

components a score. The five scores were then added and the resulted score will represent the overall writing ability. Besides, some notes were added beneath the original rating scale to facilitate rating after discussing with the raters (See Appendix J).

The Pre-test Attitude Questionnaire

The pre-test questionnaire was composed of two parts with four items in total.

The first part of the questionnaire contained three 5-point Likert scale items (items 1 to 3) and the second part was an open-ended question (Item 4). The four items were classified into two categories according to the types of attitudes they investigated:

writing apprehension (Items 1-2) and revision (3-4) (See Appendix B).

The Post-test Attitude Questionnaire

The post-test questionnaire was composed of two parts with 15 items: eleven 5-point Likert scale items (Items 1-11) and one multiple response item (Item 12) in part one; four open-ended questions (Items 13-15) in part two. All the four questions in the pre-test questionnaire were used in the post-test questionnaire (Items 1, 2, 3 and 13). Eleven extra items (Items 4-12, 14-15) were added in the post-test questionnaire so as to elicit information concerning the participants’ attitudes toward the writing instruction and the writing procedures of the study.

The 15 items were classified into four categories according to the types of attitudes they investigated: writing apprehension (Items 1-2, 7); feedback & sharing (Items 4-6, 14-15); revision (Items 3, 12-13); the influence of the writing course (Items 8-11) (See Appendix D).

(13)

Data Sources & Analysis Procedures

The data collected for this study came from the following two sources: (1) the writing pre-test and the writing post-test, (2) the questionnaire pre-test and the

questionnaire post-test. In the following subsections, I will first describe data sources and then the analysis procedure.

The Writing Pre-test and Post-test

The writing pre-test and the post-test constituted the first data source. The compositions obtained from the pre-test and the post-test were graded by two

Chinese-speaking English teachers. Both of them have been teaching English for three years in high school. They were familiar with the JCEE rating scale and experienced in correcting English compositions written by high school students.

Rater Training Session

Before the raters began to grade the compositions, they were invited to participate in a training session to ensure the reliability between their evaluations.

First, I explained the scale in detail so that the raters could fully understand it. Next, I gave each of the raters four sample compositions written by the participants during the treatment and graded by the researcher (see Appendix H). The four sample

compositions, ranging from excellent to poor, were chosen with the purpose that raters could have an idea of the types of compositions they were going to grade. I explained the analytic scores given for each of the five components of the compositions in reference to the rating scale. After both of the raters agreed on the criteria and understood how the rating scale was used, they were asked to grade four sample compositions of different quality levels. When they finished grading, scores for each

(14)

feature given by the two raters were compared. Finally, the raters discussed the discrepancies and adjusted their differences.

At the end of the training session, each of the raters was given a package which contained a copy of all the compositions from the writing pre-test and post-test, the rating scale and the five graded sample compositions. The compositions were

numbered to disguise the students’ identity, the pre-test compositions were mixed with post-test compositions and the compositions by the two groups were also mixed.

Besides, all the compositions were arranged randomly. That is, they were not arranged from the best to the poorest or the poorest to the best. Finally, before the raters were dismissed, they were reminded to keep their own scoring stable.

Reliability Testing

Each draft was read by two different raters and the average of the two scores was used as the final score. If the total scores given by the two raters for a composition varied by less than three points, the average scores for the discrete components were accepted in the analysis. If the total score given by the two raters for a composition varied by three points or more, the two raters discussed and adjusted their scores for the discrete components. The Pearson product-moment correlation coefficients were employed to calculate the inter-rater reliability between the two raters using the final scores given to the compositions. The inter-rater reliability was 0.875 at 0.01

significant level.

Comparison of Writing Pre-test and Post-test

To determine if there was significant improvement between these two

compositions, paired-sample t-test was performed. Also, independent sample t-test was performed to compare the scores of the two groups both in their writing pre-test

(15)

and writing post-test.

Pre-test and Post-test Attitude Questionnaires

To answer the third and fourth research questions, data collected from the pre-test questionnaire and the post-test questionnaire were compared. For the

Likert-scale items and the multiple response item, a chi-square test was performed to examine the percentage differences, if any, in responses between the two groups. As for the open-ended questions, all responses were sorted, categorized and compared.

Summary of the Method

In this chapter I first cover discussion of the research design. Then, I discuss the classroom procedure and the data collection procedure. Finally, I discuss the

instruments as well as the data sources and the analysis procedure.