Chapter Three
Methodology
In this chapter, we will first present the background information of the subjects,
followed by a brief discussion of the reading materials used in the study. Then we will
describe the procedure of the present study, the pretest and posttest administered to
the subjects, and the research instruments. The scoring of the students’ writing
samples, including the scorers and the assessment criteria, is discussed at the end of
the chapter.
3.1 Subjects
171 senior subjects in An Kang Senior High School participated in the study.
They came from 4 classes-- Class 601, Class 602, Class 603, and Class 604. Subjects
in Classes 601 and 602 were natural sciences majors, with the former consisting of 32
students and the latter, 39. Subjects in Classes 603 and 604 were social sciences
majors, with a total of 50 students in each.
The average score of these subjects when they entered An Kang Senior High
School was approximately 190; therefore, they were not highly-proficient students in
terms of their academic performance. In their first year of senior high school, they
were randomly arranged into four classes and in their second year they were
regrouped based on their own choice and aptitude. They did not have much formal
training in composition writing in school until they became eleventh graders.
These 171 subjects did not take any test on English summary writing in school
and had never received any formal instruction on how to write an English summary.
Classes 602 and 603 formed the experimental group and were taught by the researcher.
Classes 601 and 604 constituted the control group, which underwent no treatment at
all during the study, and the instructor was a colleague of the researcher. An Kang
Senior High School is a small school which has only four classes for the twelfth
graders. Furthermore, as it is against the school policy for the same teacher to teach
all these classes, it was impossible for the researcher to teach the students in both the
experimental group and the control group.
With respect to these subjects’ academic performance, their scores in the first
mock exam were as follows. In the first mock exam held on September 4th and 5th,
2003, the average score of the natural sciences group was only 19.9 (out of a total of
100), while that of the social sciences group was only slightly better (26.12). This
clearly shows that these subjects in An Kang Senior High School still have great room
for improvement in their general English proficiency.
3.2 Materials
Before the research project on summary writing instruction began, the
researcher spent quite a long time looking for appropriate materials that not only
would go well with these subjects’ proficiency levels but also would be suitable for
the instruction of summative skills. After some active search, the researcher finally
decided on the reading series from the Far East Company as the major source of
reference for the present study on summary writing instruction.
Two articles were selected from Far East English Extensive Reading Series as
the texts for summary writing in the pretest and posttest. The pretest article chosen
was entitled “Interesting Ways to Learn A New Language” from Lesson One, Book
One, and the posttest text selected was entitled “Bathroom Signs” from Lesson Nine,
Book One. These two articles (see Appendices I and II) were of similar length (350
words) and the same difficulty index (7.9)1.
Other passages of shorter length and lower difficulty level were selected from
Far East English Extensive Reading Series to serve as model texts or practice
___________________________________
1 The length and the difficulty index of each reading article can be found at the bottom of each article in the Far East English Extensive Reading Series. There are 168 articles in the series and the highest difficulty index amounts to 12.0 while the lowest one, 2.6. According to the introduction by the authors, the article with the length of about 400 words and its difficulty index of below 7.0 is suitable to the seventh graders in the United States. While the article with the length of between 500 and 700 words and its difficulty index of below 9.0 is suitable to the ninth graders in the United States. Therefore, in
passages for the participating students to practice applying the rules in the Rule
Governed Approach when they were learning to write English summaries. Some
articles or practice materials in the workbook of Far East Book Five and in Far East
Composition were also used in the present study in order to help the students enhance
their ability in selecting topic sentences or even inventing their own topic sentences.
The origins of the above-mentioned practice materials can be seen in Appendix III.
3.3 Procedures
The treatment time lasted for three and a half months, from September to
December 2003. After the pretest, the researcher did extensive preparatory work
before she initiated the instruction of summative rules. She started by teaching the
subjects in the experimental group the techniques of brainstorming with six
wh-questions. Then the researcher gave explicit instruction to the experimental
subjects regarding the important notion of the “main idea” in a text, owing to the fact
that several researchers have also highlighted that a summary must contain the gist of
the original passage (Friend, 2001; Fitzgerald & McBeth, 1991; Donn, 1988; Smith &
Dahl, 1984; Seale, 1978). What's more, as Friend (2001) and Fitzgerald & McBeth
(1991) point out in their respective studies, a summary should be written in
paraphrase; the experimental subjects received the instruction on how to paraphrase a
short paragraph one week later.
After introducing the basic notions and concepts, the researcher started the
formal instruction of the six rules of the Rule Governed Approach. The experimental
group was given instruction of the rule of selecting topic sentences first, followed by
the rule of inventing the topic sentences. After the rule of inventing topic sentences
came the super-ordination rules, including super-ordination of nouns, following by
super-ordination of action. Finally, the deletion rules were presented and practiced,
including deletion of unimportant or trivial materials, subsequently followed by
deletion of important but repeated materials.
In the formal treatment process, the researcher provided a step-by-step
modeling of these rules, activating discussions among the subjects, analyzing samples
in class, and requiring the subjects to complete a summary-writing task with the skills
they had learned in class.
After the treatment of summary writing instruction, both the experimental and
control group took part in the posttest.
3.4 Pretest
Before the researcher asked subjects in both the experimental and the control
groups to write their first English summary, two of the researcher’s colleagues who
taught both groups Mandarin Chinese had given these one hundred and seventy-one
subjects a test on Chinese summary writing ability as shown in Appendix IV. These
two Chinese teachers collected samples of the subjects’ Chinese summary writing and
assessed them according to the same criteria used in assessing English summaries as
discussed in Section 3.8.2 below. The scores of the subjects on the Chinese summary
writing task will be employed as a tool of comparison when the researcher discusses
the results of the study in Chapter 5.
The pretest of the study mainly consisted of three tests, along with a
questionnaire. Both groups were asked to write an English summary within fifty
minutes and they were explicitly told not to copy any sentence from the original text.
If it was necessary to cite from the original text, subjects were told to use double
quotation marks. Immediately after the completion of the summary writing task, both
groups were asked to complete a questionnaire (see Appendix V).
Both groups were also required to take Intermediate Level Tests on reading and
writing abilities from GEPT (see Appendices VI and VII) to identify their respective
proficiency levels. The grades the students obtained from the reading and writing tests
were used as a reference for assessing how good their overall English language
proficiency was and how they performed in the first English summary writing task.
3.5 Posttest
The posttest of the study also included three tests and a questionnaire.
Following the same procedure, both groups were asked to write a summary of another
passage with similar difficulty index, also within fifty minutes. Immediately after the
completion of the task, subjects in both groups were required to complete the same
questionnaire as they did in the pretest. By doing so, the researcher would be able to
investigate whether or not the instruction of summative skills had affected their
attitude towards and interest in learning summary writing.
As in the pretest, both groups were required to take two other forms of
Intermediate Level Tests on reading and writing abilities from GEPT (see Appendix
VIII and IX). The results of the two tests were used for the purpose of comparison
with those in the pretest, allowing the researcher to make judgments as to whether the
instruction of summary writing also helped the subjects to improve their overall
English language proficiency.
3.6 Instruments
The instruments that were employed in the present study consisted of the
statistical analysis, the questionnaire, and the performance data.
3.6.1 Statistical Analysis
Statistical instruments were employed in this study in order to investigate the
effects of the treatment. The data collected for analysis to examine the effects of
summary writing instruction included (1) the scores of Chinese summary writing task
(2) the scores of the pretest/posttest summary writing tasks (3) the scores of the
pretest/posttest GEPT reading comprehension tasks (4) the scores of the
pretest/posttest GEPT writing tasks, and (5) the scores of the JCEE composition tasks.
For the measurement of the subjects’ linguistic competence, the above-mentioned
scores were computed using the Statistical Package for Social Science (SPSS) version
11.0 for Windows. The results of ANOVA, Chi-square test, correlation coefficients,
and Frequency Table were used to compare and analyze the results of the study.
3.6.2 Questionnaire
The questionnaire for the study was designed by the researcher to serve the
purpose of investigating the students’ attitude toward summary writing. Besides, to
ensure that each student understood the description completely, the questionnaire was
written in Chinese. It was delivered to both groups immediately after the pretest and
the posttest, and the main purpose was to investigate the subjects’ background
information, their experience with summary writing, their attitude toward summary
writing, as well as the methods the subjects adopted in summarizing an article.
Totally, there were twenty question items in the questionnaire. A five-point
Likert rating scale was used to signify the subjects’ conformity with the statement as
the data collection scheme. The subjects scored their answers on the basis of their
opinions from strongly agree, agree, no comments, disagree, to strongly disagree.
Then, these items were counted as 5 points, 4 points, 3 points, 2 points, and 1 point,
respectively. Since this inventory utilized a bipolar semantic differential format, some
of the items involving negative statements were counted in reverse order (items 3, 4, 7,
8, 11, 15, and 19). These items with negative descriptions were designed for the
purpose of probing into the real feelings of the subjects when they were asked to write
an English summary.
Below is the English version of the attitude questionnaire and the questions are
classified based on four factors: Confidence, Anxiety, Usefulness, and Preference.
Factor I: Confidence (1, 5, 9, 13, 17)
1. I consider it easy to write a summary in English.
5. I would like to spend my free time practicing summary writing outside the classroom.
9. English summary writing is a joyful writing process.
13. I think I have a special talent in English summary writing.
Factor II: Anxiety: (3, 7, 11, 15, 19)
3. I don’t know how to start writing an English summary when I am asked to.
7. If summary writing takes the place of guided writing in the JCEE, I am worried that I won’t be able to get good grades.
11. If I have a choice, I would prefer not to write an English summary.
15. When I think of English summary writing, I have a headache.
19. I feel uneasy when I am asked to share my summary with my classmates.
Factor III: Usefulness (4, 8, 12, 16, 20)
4. If college entrance exams do not include the summary writing task, I don’t feel the need to learn how to write an English summary.
8. I do not think English summary writing is useful to me in the English learning process.
12. Learning how to write an English summary is helpful to my learning of English.
16. It is worthwhile to learn to write an English summary.
20. To satisfy future academic needs, I have to be proficient in English summative skills.
Factor IV: Preference (2, 6, 10, 14, 18)
2. I will make the best use of every opportunity to learn to write English summaries.
6. I will first locate the topic sentence of the whole passage when I write an English summary.
10. I will delete the repeated or unimportant sentences of the passage first,
when I write an English summary.
14. I will categorize the same groups of verbs or nouns, and substitute them with more general terms first, when I write an English summary.
18. Before I start writing an English summary, I will look through the text carefully from the very beginning.
After examining the answers that the subjects marked on the question items, the
researcher found that there were 15 invalid responses in the experimental group, and
13 in the control group. Therefore, the total number of valid questionnaires collected
and analyzed was 74 in the experimental group and 69 in the control group.
3.6.3 Performance data
Both the experimental and the control group were asked to write two English
summaries and complete two GEPT writing and reading proficiency tests. Students’
samples of writing proficiency tests in the pretest and posttest (see Appendix X) were
evaluated according to the criteria set up by LTTC as shown in 3.8.1 below, while
their summaries in both the pretest and posttest (see Appendix XI) were scored based
on the criteria discussed in section 3.8.2 below.
In addition, the researcher took the scores of their JCEE compositions into
consideration when she investigated the effects of the instruction on summary writing.
This was done for the researcher to more accurately assess whether the instruction of
summary writing helped the subjects to perform better in writing the JCEE
compositions.
3.7 Graders
For the scoring of students’ writing samples gathered in this study, two Chinese
teachers in An Kang Senior High School, Ms. Lai and Ms. Lin, were recruited for the
assessment of the Chinese summary writing samples. The assessment was done based
on the same criteria as those used for evaluating the English summaries. This allowed
the researcher to have a fair basis for comparing the summary ability of the subjects in
both Chinese and English.
As for the English writing samples, there were totally three graders, including
the researcher herself. Like the researcher, Ms. Chang, who is now teaching at a
national senior high school in Taipei County, and Ms. Wu, a colleague of the
researcher at An Kang Senior High, are both experienced English teachers and are
familiar with the criteria for assessing a summary and the standards of the GEPT
writing proficiency test. In fact, all the three graders for the English samples had
attended grader-training sessions and are qualified graders for GEPT intermediate
level English writing proficiency tests.
3.8 Grading Criteria
Samples of the subjects’ GEPT compositions were assessed based on the
criteria set up by LTTC. These criteria were closely followed by the three graders
throughout the scoring process. In addition, the grading criteria for the summary
writing samples were thoroughly discussed and the key points to be included in the
summary were decided by these graders beforehand. The same procedure was
followed by the two graders of the Chinese summaries before the actual scoring, and
in their grading, the criteria in 3.8.2 were closely observed.
3.8.1 Criteria for Assessing the GEPT Writing Samples
The pretest and posttest GEPT writing tasks were assessed based on the criteria
set up by the LTTC for evaluating students’ GEPT intermediate level writing samples.
The English version of the criteria set up by LTTC is stated as follows:
Level Criteria
5 The content is properly expressed and well-organized; the vocabulary and sentence patterns are used intelligibly; very few errors in the grammar, spelling and punctuation.
4
(pass)
Most of the content and organization is good; the vocabulary and sentence patterns are correctly used; some errors in grammar, spelling and punctuation.
3 The content is unintelligible and unsuitable to the topic; its organization is bad; the usage of vocabulary is limited; many errors in grammar, spelling, and punctuation.
2 The content is unintelligible and unsuitable to the topic; its organization is bad; the usage of vocabulary is limited; substantial amount of errors in grammar, spelling, and punctuation.
1 The content is unintelligible and unsuitable to the topic; it has no organization at all; the usage of vocabulary is limited; errors in grammar, spelling, and punctuation are abundant and serious.
0 Nothing/virtually nothing is written in the composition.
3.8.2 Criteria for Assessing both English and Chinese Summary Writing Samples
The Chinese and English summaries were assessed on the basis of the following
criteria, set up by the CEEC specifically for evaluating students’ performance in
summary writing tasks.
The following is the English version of the Criteria set up by the CEEC.
Level Criteria
5 States all the main points and expresses them coherently and appropriately.
4 States (over) three-fourths of the main points and expresses them coherently and appropriately. Or,
3 (Pass)
States (over) half of the main points and expresses them appropriately.
States (over) three-fourths of the main points but the expression is not very appropriate.
2 States (over) one-fourth of the main points and expresses them appropriately.
States (over) half of the main points but the expression is not appropriate.
1 Unable to state the main points.
Unable to make the ideas clearly understood.
0 Writes nothing.
The content is irrelevant to the topic.
Note: If students copied more than seven words consecutively without using double quotation marks, his/her score would be demoted to a level lower.
3.8.3 Scoring of GEPT Reading Comprehension Tests
The reading comprehension test of GEPT contains forty questions in the
multiple-choice format. The test is divided into three main parts: vocabulary and
structure, cloze passages, and reading comprehension. The total score of this test is
one hundred and twenty. To pass the test, candidates must obtain a score of eighty or
higher—that is, the students must answer two-thirds of the questions correctly. The
scoring of the students’ performance on the test was done by the researcher, following
closely the scoring keys from the LTTC. Besides, samples of the students’
performances on GEPT reading comprehension tests can be seen in Appendix XII.
3.8.4 Key Points of the Pretest Summary Writing
In the pretest, the summary writing task was based on the article: “Interesting
Ways to Learn a New Language.” Before the grading procedure started, the three
graders went through the whole text carefully and then discussed with one another in
great length. It was finally decided that the following key points must be taken into
account when they graded the subjects’ summaries.
Interesting Ways to Learn a New Language 1. There are 4 ways to learn a new/ second language.
(1) write out new words or sentences on the paper and stick them all over their homes
(2) record their voice on the tapes and listen to their pronunciation later (3) talk to themselves
(4) teach themselves to dream by thinking only in the new language for an hour before they go to bed
2. Students can try out different methods and decide which is the most effective or talk to someone else to ask for advice.
3.8.5 Key Points of the Posttest Summary Writing
In the posttest, the text that was given to the subjects to summarize was
“Bathroom Signs,” and these three graders scored subjects’ summaries based on the
Bathroom Signs
1. Looking for a bathroom is as hard as searching for hidden treasure.
2. Sometimes even if you know where the bathroom is, you still have problem finding the one for men or women.
3. If the bathroom signs are marked in Chinese characters, it is much easier than if they are marked in English.
4. In Chinese, the characters for men and women are unique.
5. In English, however, there are many different words for men and women and using a dictionary is not always helpful.
6. In some special places, slang terms are used.
7. In other modern places, pictures or science symbols are also used.
8. To understand these pictures, you have to understand the local culture.
3.9 Summary
In this chapter, the research methodology was presented, including the
background information of the subjects, the materials used in the study and the data
collection procedures. In addition, it was explicitly pointed out that the items for the
analysis in this study included pretest/posttest summary writing tasks, pretest/posttest
GEPT reading comprehension tasks, pretest/posttest GEPT writing tasks, and the
responses to the questionnaire in the pretest and posttest, along with the subjects’
performance on a Chinese summary writing task.
In this chapter we also pointed out that a quantitative analysis was conducted on
the data collected using the SPSS for Windows 11.0. What is more, three raters,
including the researcher herself, followed the grading criteria closely in their
assessment of the writing samples, checking the key points previously agreed upon
and examining carefully the subjects’ performances on summary writing tasks and the
GEPT writing tasks in the pretest and the posttest.
