CHAPTER THREE: METHOD
This chapter begins with an account of the research design, explaining the selection of the three textbook series and the use of the interview technique. Then it moves on to delineate the instrument, data collection, and analysis procedures.
Research Design
The overall research procedure of the study unfolds as follows. First, a coding system was utilized to analyze all the writing sections in the three textbook series.
These writing sections were identified, extracted, and categorized based on the coding scheme developed for this study. Following the coding and analysis of the three material sets, semi-structured interviews were conducted with six senior high school English teachers. The interviews attempted to gain insights into the teachers’ beliefs, their teaching practice, and their evaluation of the three series of writing sections.
Selection of Textbooks for Analysis
To advance our understanding of the writing materials in senior high school English textbooks in Taiwan, three sets of textbooks were selected for analysis. These three sets were Shih et al.’s (2001) Far East English Reader for Senior High Schools, Chen’s (2002-2004) San Min English Reader for Senior High Schools, and Lin et al.’s (2004-2006) Lung Teng English Reader for Senior High Schools, referred to hereafter as FE, SM, and LT.
1The three series were selected as a result of their widespread adoption by senior high schools in Taiwan.
Each textbook series consisted of six volumes, and every volume, twelve units (except the LT series, which had only ten units in its volumes five and six). A writing section was usually presented at the end of every unit. Unlike the other two series
1 To achieve a thorough evaluation of the three textbook series with specific regard to their writing instructional materials, it was fundamental to have available all their six volumes. The latest editions of the three textbook series, however, did not yet have all six volumes published at the time of the study.
For this reason, they were deemed inappropriate for the current investigation. The three series of textbooks actually selected to be analyzed were hence their most recent and complete editions.
which had writing sections throughout all six volumes, the FE series contained writing sections in the last two volumes.
Instrument
The only instrument used in the present study was a coding scheme containing five categories, Grammar and Lexis, Rhetorical Organization, Composing Strategies, Genres, and Writing Tasks. This coding scheme was adapted from Chan’s (2001)
coding scheme, which had been derived from the analysis of her own textbook data.
Chan’s original coding scheme (including the categories of Grammar, Textual Organization, Writer Strategies and Exercises) was problematic in that it failed to
include Genres as a distinct major category. This defect in design obscured the distinction between genre and text type, two seemingly interchangeable terms. As a result, Chan’s coding scheme might prove inadequate for analyzing certain writing instructional materials. With the inclusion of Genres as a major category, the resulting five categories of the current scheme exhibit greater explanatory power. It hence will provide a more solid and thorough analysis of composition materials. To effectively classify and subsume the writing materials in the three textbook series, subcategories of each of the five categories were developed. The entire coding scheme, composed of four instructional categories—Grammar and Lexis, Rhetorical Organization,
Composing Strategies, Genres—and one task category—Writing Tasks—is displayed
in Figure 1. The coding of the study, divided into two major phases, involved the four
instructional categories in the first phase, and the task category in the second (see
Appendix A for an elaborated version of the scheme categories and subcategories).
I. Grammar and Lexis 1. Lexical items 2. Punctuation 3. Pronouns 4. Articles
5. Countable & uncountable 6. Quantifiers
7. Modifiers 8. Comparatives 9. Verb forms 10. Infinitives 11. Gerunds
12. Prepositional phrases &
phrasal verbs
13. Participial constructions 14. Subject-verb typology 15. Agreement
16. Modal auxiliaries 17. Sentence patterns 18. Tense & aspect 19. Voice & Speech 20. Conditionals 21. Clauses
II. Rhetorical Organization
A.
Paragraph/essaystructure
1. Overall paragraph structure 2. Controlling ideas 3. Topic sentences 4. Support sentences 5. Concluding
sentences 6. Overall essay
structure 7. Thesis statements 8. Introduction 9. Body 10. Conclusion 11. Transition &
coherence 12. Formal features 13. Other devices
B. Text types 1. Descriptive 2. Narrative 3. Expository 4. Argumentative
III. Composing Strategies 1. Overall writing process 2. Invention techniques
a. Brainstorming b. Clustering c. Free writing d. Looping e. Cubing
f. Heuristics 3. Planning 4. Multiple drafting 5. Peer/teacher feedback 6. Revising
7. Editing
IV. Genres 1. Form 2. Resume 3. Advisement 4. Poem 5. Notice 6. Message 7. Notes 8. Card 9. Letter 10. Summary
11. Personal expression 12. Autobiography
VI. Tasks
A. Word-level tasks 1. Fill-in-the-blanks 2. Form-filling 3. Listing
B. Sentence-level tasks 1. Translation 2. Error correction 3. Answering
questions 4. Sentence writing 5. Picture description
C. Paragraph-level tasks 1. Note-taking 2. Free writing 3. Journal writing 4. Outlining 5. Arranging
sentences 6. Paragraph
elements 7. Paragraph rewriting 8. Paragraph writing 9. Picture writing 10. Giving feedback
11. Summary
12. Essay writing 13. Writing project
D. Non-writing tasks 1. Group work 2. Reading 3. Identification 4. Ordering 5. Punctuation 6. Self-monitoring 7. Checklists
Figure 1. The Coding Scheme
The four instructional and one task categories along with their subcategories are briefly discussed and explained in the following sections.
Grammar and Lexis
Grammar and Lexis, the first instructional category, contains 21 subcategories, covering lexical items and an array of linguistic entities, ranging from phrases through particular sentence patterns to clauses of various lengths and types. This inventory was developed from Chan’s (2001) variable of Grammar in her scheme that contained seventeen subcategories. Four extra subcategories—quantifiers, comparatives,
agreement, and sentence patterns—were added subsequent to an examination of the
writing sections in the three series. These 21 subcategories can be seen under
Grammar and Lexis in Figure 1.
Rhetorical Organization
Consisting of 17 subcategories, Rhetorical Organization was further divided into two main groups: paragraph/essay structure and text types. The 17 subcategories were procured from extensive discussions by numerous researchers in the field of L2 composition (e.g. Chan, 2001; Ferris, 1998; Hyland, 2003; Raimes, 1991; Reid, 1993;
Richards & Schmidt, 2002; Silva, 1990). The first group, paragraph/essay structure, includes thirteen subcategories of techniques, chiefly for opening, developing, concluding, smoothing or adding variety to a paragraph/essay. Formal paragraph features also fall to this group of subcategories. The second group of subcategories contains four key text types, namely, descriptive, narrative, expository, and
argumentative. These text types represent rhetorical possibilities for patterns of organization, i.e. for structuring and developing a paragraph for different rhetorical purposes.
Composing Strategies
Composing Strategies, the third instructional category, covers 7 subcategories and addresses a variety of techniques for approaching and managing a given writing task. Briefly, they include techniques for generating ideas, engaging in multiple drafts and revisions, receiving teacher or peer feedback on intermediate drafts, and
ultimately, editing composed products. These subcategories were obtained from reviews of recent research work (e.g. Chan, 2001; Ferris, 1998; Hyland, 2003; Johns, 1990; Raimes, 1991; Reid, 1993; Reimer, 2001; Silva, 1990).
Genres
Genres is comprised of a total of 12 subcategories, dealing with functional writing in a range of situational contexts. An assortment of 7 everyday genres
suggested by Hyland (2003) were incorporated: advertisement, poem, notice, message,
notes, letter, and autobiography. Following an examination of the three material sets,
five more genres were identified and incorporated, including form, resume, card, summary, and personal expression.
Tasks
While the four instructional categories seek to identify the instructional focuses of writing materials, the task category attempts to analyze their accompanying tasks.
It covers a total of 28 subcategories, constructed essentially on the basis of Chan’s (2001) scheme variable of Exercises. These subcategories, after trialed on the three series, were rearranged, adapted, and classified into four broad types: word-level tasks, sentence-level tasks, paragraph-level tasks, and non-writing tasks.
As can be seen in Figure 1, word-level and sentence level tasks contain three and five subcategories respectively. These two task types subsume tasks that involve ascending degrees of written production within the sentence level.
Paragraph-level tasks, covering a total of thirteen subcategories, stretch beyond the sentence level and require learners to construct multiple sentences or even paragraph/essay length texts. This set of tasks offer opportunities to either reinforce pre-writing techniques, deepen familiarity with paragraph structure, or provide practice with extended writing.
Non-writing tasks, the last major type of tasks, refer to those with no actual
production of writing required. Such tasks, unlike the first three types of tasks, do not
gear learners toward written output. They mainly seek to strengthen certain areas of
writing knowledge through, for instance, multiple choice, punctuation, identification,
and ordering exercises. There are a total of seven non-writing task subcategories,
listed under non-writing tasks in Figure 1.
Data Collection Procedures
In the present study, two sources of data were collected for analysis: all writing sections from the three textbook series and semi-structured interviews with six senior high school English teachers. The procedures for data collection are detailed below.
Collection of Textbook Writing Materials
All the writing sections in the three textbook series were identified and extracted.
A total of eighteen volumes, i.e. six volumes from each series, were scanned. All textbook pages containing writing sections were identified, photocopied, and
compiled into three separate data sets, comprising the core source of data for analysis.
Collection of Teacher Interview Data
The interview data were the second data source in this study. Semi-structured interviews were utilized to elicit relevant information from six senior high school English teachers. They were conducted mainly to investigate the teachers’ philosophy of composition instruction, actual use of the writing materials, and evaluation of these materials. The interviews were structured around a series of nine questions (see Appendix B) designed to procure information unavailable through the textbook
analysis. The questions were guided essentially by three major themes, i.e. philosophy, practice, and evaluation.
For each textbook series, two teachers were interviewed. This resulted in a total of six interview data sets. All interviews were lasted for approximately sixty minutes.
They were audio-taped and thereafter transcribed verbatim for analysis.
Data Sources and Data Analysis Procedures
This section presents the two major data sources—the three series of writing
materials and the six interview data sets—and the procedures for analyzing them.
Analysis of Textbook Writing Materials
The first data source was the textbook writing sections. As stated earlier, the writing sections from all three textbook series were identified and arranged into three data sets for coding and analysis. The coding procedure entailed having all extracted instructional materials appropriately labeled and categorized. Subsequent analysis of the three resultant coded data sets focused on calculating their frequencies for instructional units and task units. The actual coding procedure is reported in the
following section.
Coding Procedures
To start with, a set of coding sheets was designed for categorizing and recording instructional units and task units. Listed on the coding sheets were the four major instructional and one task categories with their subcategories arranged underneath.
2The coding procedure proceeded in two major phases. The objective of phase one was to identify instructional units and determine their corresponding major instructional categories as well as subcategories. Then the sources of the identified instructional units were noted under the matching subcategories on the coding sheets.
For instance, in Volume 3, Unit 9 of the SM series, “participial phrases” was identified as an instructional unit belonging to the major instructional category of Grammar and Lexis. Its specific volume and unit (noted as 3-9) were then recorded on the coding sheets under the linguistic subcategory of participial constructions. In phase one, only instructional units were coded, leaving all task units to be dealt with in phase two. The procedures for coding instructional units are graphically presented in Figure 2.
2 The design of the coding sheets followed the form of the coding scheme, as shown in Figure 1.
Figure 2. Procedures for Coding Instructional Units
Upon finishing the coding of instructional units, the researcher moved on to phase two. The objective of phase two was to categorize and label all tasks. Similarly, to label task units, the specific source volumes and units of the tasks identified were recorded under their matching task subcategories. For example, the writing section in Volume 3, Unit 3 of the SM series included two tasks, with one asking learners to rewrite sentences and the other requiring them to make sentences based on words provided. These tasks were coded below the task type of sentence-level tasks, with their specific source volumes and units (3-3) recorded under the subcategory of sentence writing. Occasionally, a given writing task could be coded multiple times.
For example, the writing task in Volume 6, Unit 9 of the LT series involved reading, summarizing, and then responding to a text, and thus was coded three times. Figure 3 illustrates the procedures for coding task units.
Identify an instructional unit
Place the unit under a matching subcategory Determine its corresponding
major category
“Participial Phrases” in V3 U9 of the SM series identified as an instructional unit
Determined as belonging to the major category of Grammar and Lexis
With “3-9” written under the
subcategory of participial
constructions
Figure 3. Procedures for Coding Task Units
Inter-Coder Reliability
To ensure the validity and reliability of coding results, a reliability check with a second coder was carried out prior to the entire coding procedure. The second coder, an MA student in the TESOL program at National Taiwan Normal University, and the researcher coded a randomly selected 10% of all data, that is, 10% from each data set.
Previous to the reliability coding, the researcher provided the second rater with thorough explanations of the coding system. Once familiarized with the coding scheme, the second coder was given opportunities to code a small amount of the data for the sake of practice and confidence building. Upon the completion of this training session, the researcher and the second coder independently coded the randomly selected 10%. No further discussions were exchanged during this process.
Upon finishing coding the designated materials, the two sets of coded data were compared and the inter-rater reliability was calculated. The results of the reliability check are as follows: For the three sets of textbooks—the FE series, the SM series, and the LT series—the agreement rates were 95.45%, 95.24%, and 91.67%
respectively. The overall agreement rate was a high 94.03%.
Identify a task unit
Place the unit under a matching task subcategory
Determine its belonging task type
“Write a paragraph about the greatest discovery” in V6 U3 of the SM series identified as a task unit
Determined as belonging to the task type of paragraph-level tasks
With 6-3 written under the
subcategory of paragraph writing
Procedures for Analyzing the Writing Sections in the Three Series
This section delineates the procedures for the content analysis of the three writing sections in the three series. The analysis of instructional units and task units will be described below respectively.
Analysis of instructional units. The analysis of instructional units mainly
involved calculating the frequencies and percentages of the four major categories and those of their subcategories. All data were entered into SPSS 14.0 for Windows. Then a chi-square test was run to determine whether the observed distributional proportions of the four major categories were homogeneous or heterogeneous across the three series.
Analysis of task units. All tasks in each material set were classified into four
major categories according to the level of writing required, ranging from non-writing through word-level and sentence-level to paragraph-level tasks. The frequencies and percentages of task units under these four types of tasks were reported across the three series to identify and compare their task focuses. Once again, chi-square was
performed to establish the homogeneity or heterogeneity of the distributions of the four task type categories in the three series. After the coding of task units, efforts were also made to identify the most frequent task subcategory and the extent of extended writing practice
3included in the three series.
Analysis of Interview Data
As stated previously, the procedure for analyzing the interview data was based on three major themes: philosophy, practice, and evaluation. The six interview data sets were analyzed and compared, between the teachers and across the material sets.
3 Extended writing tasks simply refer to those that required text production beyond the sentence-level and that provided at least one or more of the following: topics, information to include, useful words, expressions, sentence patterns, models, outlines, and styles.
