國
立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
CHAPTER THREE METHOD
This study aims to examine the correlations between scores on the
multiple-choice writing test of the English Professional Subject (EPS) and English performance. This study also determines which subtest of the EPS objective test is more related to the writing ability of the students from Department of Applied Foreign Languages (DAFL). In addition, this study investigates the examinees’ opinions on the EPS writing test. This chapter is thus divided into the following four sections: a brief description of the participants, the instruments employed, the procedure of data collection, and the methods of data analysis.
Participants
The sample of the participants in this study was randomly selected from a population of 1,143 third-year DAFL students in 13 commercial vocational high schools in the great Taichung area (Ministry of Education, Department of Statistics [MEDS], 2011). Three schools were at first randomly drawn from the 13 schools.
Then, one class within each of the three schools was randomly selected for inclusion in the study. A total of 124 third-year DAFL students were first sampled. Nevertheless, five participants were dropped from the samples because they had taken the EPS multiple-choice writing test before. Hence, as indicated in Table 3.1, a total of 119 students, composed of 17 males and 102 females, participated in the present study.
Table 3.1 Demography of Participants
No. of Class No. of Students
Gender
Male Female
School 1 1 50 7 43
School 2 1 28 1 27
School 3 1 41 9 32
Total 3 119 17 102
‧ 國
立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
As can been seen in Table 3.2, most of the participants (60.5%) have learned English for 7 to 10 years. Some students (29.4%) have learned English for more than 11 years, approximately from preschool age. Few students (10.1%) have received 5 to 6 years of formal English instruction.
Table 3.2 English Learning Experience of Participants
Years No. Percent
5-6 years 12 10.1%
7-8 years 23 19.3%
9-10 years 49 41.2%
More than 11 years 35 29.4%
However, in terms of English writing, the students do not have too many learning experiences (see Table 3.3).
Table 3.3 English Writing Learning Experience of Participants
Years No. Percent
None 31 26.1%
Less than 1 year 47 39.5%
1-2 years 28 23.5%
3-4 years 9 7.6%
More than 5 years 4 3.4%
Most of the students (65.6%) have none or less than one year of learning experiences in English writing. The reasons why the participants do not have enough instruction in English writing are that one of the participating schools does not have courses of English Writing and that the other two with English Writing courses do not pay too much attention to teaching writing due to their tight teaching schedule.
Table 3.4 then illustrates the certificates of the General English Proficiency Test (GEPT) the participants acquired.
‧ 國
立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
Table 3.4 Acquired GEPT Certificates of Participants
GEPT Certificates No. Percent
None 13 10.9%
Elementary (First) 2 1.7%
Elementary (Second) 13 10.9%
Intermediate (First) 55 46.2%
Intermediate (Second) 16 13.4%
High-Intermediate (First) 20 16.8%
High-Intermediate (Second) 0 0%
Over 76% of the participants have acquired the certificate of or higher than the first stage of the intermediate level. Because the participants major in English, their English proficiency levels are higher than those of the students from other departments of the vocational high schools.
Instruments
The instruments employed in the present study included the EPS multiple-choice writing test, the English picture writing task, indirect writing questionnaire (IWQ), and direct writing questionnaire (DWQ). The questionnaires were written both in English and in Chinese. However, the Chinese versions were used to guarantee the participants’ full understanding of the questions.
EPS Multiple-choice Writing Test
The EPS multiple-choice writing test from the 2011 academic year was used for the present study. The 2011 edition contains 40 multiple-choice items, divided into five parts: topic sentence selection (TSS), sentence insertion (SI), sentence deletion (SD), sentence rearrangement (SR) and cloze. The examinees need to finish the test within a 100-minute time limit. According to the conference held by National
Teachers’ Association to comment on the EPS writing test in the 2011 academic year, the test items conform to the objective of the writing test (National Teachers’
Association [NTA], 2011). Since the emphasis of the first four sections of the writing
‧ 國
立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
test is to assess students’ basic writing ability (You et al., 2002), the first four parts of the 2011 edition were directly employed as the multiple-choice writing test of the current study.
In order to inspect whether the testing constructs of the EPS subtests could represent students’ writing ability, the test items and the number of items for each of the four parts remained the same as those of the 2011 EPS multiple-choice writing test.
The objective test in the current study was thus composed of 30 multiple-choice items, divided into four subtests: TSS, SI, SD and SR. The number of items was 10, 7, 7, and 6 for each of the four subtests respectively. The study allotted 75 minutes for the participants to complete the test (for the writing test and its answer keys, see Appendix A and B).
English Picture Writing Task
The present study employed picture writing as the direct writing task because DAFL teachers have recommended using picture writing to test their students’ English writing proficiency in the EPS writing assessment (You, et al., 2002). For unskilled student writers, it would be easier to write about stories because the pictures can provide a reference point and create a context for examinees to develop their compositions (X. Lin, 2006; Wright, 1989; Wright, 1996).
The English picture writing task used for the current study was directly taken from the 2006 academic year of the writing exam of the Scholastic Aptitude Test (SAT), which requires test takers to describe a series of three related pictures. The 2006 edition has received rave reviews from teachers and students because the
creative and detailed pictorial stimulus can evaluate examinees’ various writing ability (X. Lin, 2006). The writing task in the current study, in light of the testing instruction of the 2006 edition, required the participants to compose a 100-word passage about the prompts in 30 minutes. Directions were given to call for the students to describe
‧ 國
立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
what was happening in the three-frame picture sequence (see Appendix C).
The present study employed holistic scoring technique in the assessment of the picture writing task in that this scoring technique has won praise from writing
assessment researchers. For example, Perkins (1983) noted that if overall writing competence was the aim of direct measure of writing ability, holistic scoring possessed the strongest construct validity among all of the scoring techniques.
Moreover, Cooper (1984) also pointed out that holistic scoring was the most valid and direct way to rank-order and select candidates in terms of writing ability.
Since the picture writing was taken from the SAT writing exam, the scoring rubric developed by Joint College Entrance Examination Center and used as the standard to score SAT picture writing was adopted for the holistic scoring for the present study (see Appendix D for Chinese version, and Appendix E for English version). As X. Lin (2009) reported, the holistic scoring was classified into five ranks:
very good (19-20 points), good (15-18 points), fair (10-14 points), poor (5-9 points), and very poor (0-4 points). Using the analytic scoring rubric, the two raters awarded a comprehensive score that reflected their overall impression of text quality. They then verified whether the holistic score they had just given corresponded to the scoring scheme. The holistic score was briefly checked by the following five components:
content (five points), organization (five points), grammar (four points), vocabulary (four points), and mechanics (two points).
According to Stephenson and Giacoboni (1988), this scoring procedure,
“combining the elements of the analytical and holistic scoring methods, provides results that are less subject to the relativistic criticism that pure holistic scoring might elicit” (p.10).
Indirect Writing Questionnaire
IWQ consisted of 13 questions and was divided into two parts: (a) basic
‧ 國
立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
information; (b) questionnaire questions (see Appendix F for English version, and Appendix G for Chinese version). The first part aimed to collect the participants’
demographic information, including their sex, their learning experience of English and English Writing, their English proficiency levels, and their previous experiences in taking the 2011 edition of the EPS objective writing test.
As for the second part of the questionnaire, it was intended to elicit the DAFL students’ perceptions of the EPS writing test. Questions 1 to 4 asked the students the abilities they applied while doing the four subtests of the EPS writing test respectively.
Nine ability items, listed in each of the four questions, were given based on Grabe and Kaplan’s (1996) taxonomy of language knowledge of writing. Ability items 1-4 belonged to knowledge at the word and sentence levels, and ability items 6-9 to knowledge at the above-sentence level. Since multiple-choice writing tests are referred to as recognition measures (Breland & Gaynor, 1979; Cooper, 1984), ability item 5 was used to examine whether the participants applied reading comprehension in the assessment of the writing test1.
Then, the participants were required to number the four question types on a scale of the most difficult to the easiest in Question 5. The students checked the difficulties they ran into while answering the most difficult question type in Question 6. Eight statements of the difficulties the participants might face were listed in this question based on Ackerman and Smith’s (1988) indirect writing model. The items all fitted into the reviewing process of the writing framework. Items 1-3 fell into
difficulties at the word and sentence levels and items 4-7 into difficulties at the discourse level. The final item was used to investigate whether the participants had difficulty finishing reading the items of the test within time limit.
1 The nine items listed in each of Questions 1 to 4 were not numbered in IWQ. Nevertheless, they were specifically referred to in numbers in the presentation of results in Chapter Four. The same situation also happens to Question 6 of IWQ, and to Questions 1 and 2 of DWQ.
‧ 國
立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
The final two questions were meant to collect the participants’ views on the indirect writing test. Question 7 brought forth the students’ recommended question types for assessing writing indirectly other than the four item types of the EPS writing test. With regard to Question 8, the participants were requested to express their
opinions on the appropriateness of the indirect format to examine writing ability.
Direct Writing Questionnaire
In the beginning of DWQ, the participants were asked whether they took the 2006 edition of the SAT picture writing exam before in order to exclude any practice effect.
The main part of the questionnaire included eight questions, aiming to explore the participants’ perceptions of the EPS indirect writing test and the picture writing task (see Appendix H for English version, and Appendix I for Chinese version).
Question 1 included nine ability items, building on Grabe and Kaplan’s (1996) detailed list of writing constructs, to draw forth the abilities undergone by the participants while performing the picture writing task. Items 1-4 were meant to examine whether the students utilized knowledge at the word and sentence levels, whereas items 5-9 were used to inspect knowledge at the above-sentence level.
In Question 2, in light of Ackerman and Smith’s (1988) direct writing model, 10 statements of the difficulties the participants might encounter were listed to elicit the difficulties the participants faced in the assessment of the writing test. Items 1, 8, and 9 fell into the planning stage of the cognitive writing process, items 2-7 into the translating stage, and item 10 into the revising stage.
Questions 3 and 4 then aimed to reflect the participants’ attitude toward the direct writing task. The students were required to describe whether they liked producing picture writing in Question 3 and to recommend direct writing question types apart from picture writing in Question 4.
‧ 國
立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
The participants’ opinions on the comparison between indirect and direct writing tasks were put forward in Questions 5 and 6. Question 5 queried the
participants about the writing assessment format they preferred and inquired into the reasons why they liked it. In Question 6, the students were asked to decide which question type was more difficult and to describe the reasons why they thought so.
Questions 7 and 8 were intended to ask the participants their opinions on the EPS writing test of the TVE Joint College Entrance Exam. Question 7 requested the participants to answer whether the EPS writing test of the entrance exam should include English writing. In Question 8, the participants were required to determine which testing format they thought the writing test should adopt.
Procedure
The research procedure of the present study comprised the following five steps, as illustrated in Figure 3.1.
‧ 國
立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
Figure 3.1 Research Procedure of the Study
First of all, the instruments used for the study were prepared. The EPS
multiple-choice objective test was organized and printed. As for the picture writing task and the questionnaires, they were also prepared and then subjected to expert judgment. One professor specializing in testing and three experienced vocational high school English teachers ensured the appropriateness of the writing task for the
proficiency levels of the participants. The questionnaires were reviewed and modified by a professor, teaching at a university in northern Taiwan and well-versed in theories
Preparing picture writing task and questionnaires
Expert validity Preparing EPS
multiple-choice writing test
Training of two raters
Machine-scoring indirect writing
Scoring direct writing
Data analysis
Contacting teachers of the three classes from the three different schools & making arrangement for indirect and direct writing tests, and questionnaires
Consent form + indirect writing + indirect writing questionnaire school 1→ school 2→ school 3
Direct writing + direct writing questionnaire school 1→ school 2→ school 3
Keying-in
questionnaire data
‧ 國
立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
of testing. Secondly, the three schools that participated in the study were contacted in order to set the testing schedule. Thirdly, the writing tests and the questionnaires were administered to the participants school by school as scheduled. Fourthly, after the completion of all the tests and questionnaires, all of the data were processed. The multiple-choice writing test was machine-scored. The data of the questionnaires were keyed in. With regard to the direct writing task, two experienced vocational high school English teachers were invited to score the writing samples. One rater has 9-year and the other 16-year teaching experiences. Both have graduate qualifications in Teachers of English to Speakers of Other Languages (TESOL). Training sessions of the two raters were first held. Then, all of the writing samples were scored
independently and holistically by them. At last, the resulting raw scores of the direct and indirect writing tests and the data of the questionnaires were used for analysis.
The administration of the indirect and direct tests and the questionnaires was conducted in December in 2011 and in the same manner for each of the three schools.
Three periods of English Writing classes were used to implement the study. The first two class periods were devoted to the indirect writing test and IWQ. Before
measuring the students, the participants signed consent forms to the research after informed of the purpose of the present study. Then, they took the EPS multiple-choice writing test in 75 minutes. Finally, they filled in IWQ in 10 minutes. One day later, the third class period was used to administer the direct writing task and DWQ. A brief introduction of the task was first provided. The participants were then required to write a passage based on the picture prompts on the picture writing task in 30 minutes.
Afterwards, DWQ was administered to the students, who filled it out in 10 minutes.
After the administration of the direct writing task, four training sessions, in light of the thorough training processes reported in Shohamy, Gordon and Kraemer’s (1992) and Weigle’s (2002) studies, were held in January in 2012. At the beginning of the
‧ 國
立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
training, the purposes and demands of the writing task were introduced. The scoring rubric was then demonstrated with five randomly-selected samples that exemplified the five different ranks in the scoring rubric in the first session. Once the readers got familiarized with the rating scale, another five random selections of the writing samples were given to them in the second session. In this session, they were given a chance to practice scoring the samples. Following this scoring practice was a detailed discussion and negotiation of the ratings they had assigned. The discussion was conducted until they reached an agreement in the assessment of the samples. The third and fourth sessions were undergone with the same procedures using different
randomly-selected writing samples.
Right after all of these training, four scoring sessions, building on Breland and Jones’ (1982) study, were held. The handwritten samples were randomly divided into four sets. During each scoring session, the two raters scored the sets of the samples independently and holistically without knowing which school the writing came from.
Since readers tended to rate lower near the end of a scoring period (Godshalk et al., 1966), the same set of papers was read in reverse order by the second rater. Rest breaks were taken during each scoring session to reduce fatigue and increase
reliability (Breland et al., 1987). Moreover, to enhance the reliability of the evaluation, regular monitors were held to ensure graders’ consistency in applying the scoring rubric (Breland et al., 1987; Jacobs, et al., 1981; White, 1984). For the few cases in which independent ratings differed by more than five points, a third reader was brought in to resolve discrepancies (X. Lin, 2009).
Data Analysis
The data collected in the study included the scores of the indirect writing test and those of the direct writing task, and the students’ opinions generated from the questionnaires.
‧ 國
立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
The first two sets of data were subjected to correlation tests. Since the grading scale for the indirect and direct writing test was different, raw scores were used to do the statistical analysis (The participants’ raw scores of the two writing tasks can be found in Appendix J). The total score of the objective test of the present study (30 items) was 75. In other words, the score for each correct item was 2.5, the same as that in the 2011 edition of the EPS writing test of the TVE Joint College Entrance Exam. Therefore, the total scores of each of the four subtests were 25, 17.5, 17.5, and 15, respectively. Raw scores were obtained by counting the number of correct answers on all four subtests.
With the picture writing task, the total score was 20. The scores of the picture writing test were calculated by averaging the scores assigned by both raters to each written product. For the writing samples called for the third reading, the final ratings were the averages based on the two scores in closest agreement. After grading the written texts, the inter-rater reliability was computed to check the degree of agreement between the two independent scorers. Inter-rater reliability coefficient, estimated by Spearman correlation between the independent ratings assigned to the writing samples, was .75 in the current study. The resulting reliability coefficient corresponded to Shohamy’s (1985) requirement that two readers involved in scoring writing ability should reach a reliability of .70.
With regard to the last data obtained from IWQ and DWQ, they were categorized and analyzed. The data drawn from the first part of IWQ were used to supplement the description of the participants. The results of the second part of IWQ were to describe the students’ perceptions of the EPS indirect writing test. The participants’ opinions on the EPS writing test and the picture writing task were
represented in DWQ. Both data of the questionnaires were analyzed for frequency and percentage, displayed in tables.
‧ 國
立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
To answer the first research question, Pearson product-moment correlations
To answer the first research question, Pearson product-moment correlations