CHAPTER THREE RESEARCH DESIGN
This chapter describes the research design of the present study. The participants are introduced in section 3.1, and the instruments in section 3.2. The procedures of data collection and analysis are presented in section 3.3. Finally, the main points of this chapter will be summarized in section 3.4.
3.1 Participants
A total of 94 participants (90 English teachers; 4 textbook writers) participated in the present study. Originally there were one hundred and 128 questionnaires were distributed to English teachers. After screening, 38 were excluded because they consisted of invalid answers and insufficient personal information. Still some were randomly eliminated after grouping due to the purpose of keeping the balanced number in each group. Therefore, only 90 questionnaires of the teachers were chosen as valid data.
The 90 English teachers were recruited from different senior high schools at the end of August, 2005. According to the information they provided in the personal background sheet (see Appendix B), they were divided into three groups on the basis of the textbook sets adopted in the previous school year (i.e., 2004/8~2005/8): 30 of them used Textbook A, another 30 Textbook B, and the other 30 Textbook C
35. More details are given in Table 3-1.
35 The reason behind the selection is that Textbook A and Textbook B have been frequently compared in the previous studies (Chang, 2002; Chen, 2002; Chen, 2003; Hsu, 2004; Ko, 2005; Yeh, 2003). It is evident that they have been widely adopted in recent years. Textbook C is included because more and more schools adopt it, but only Cheng (2002) and Ko (2005) discussed it.
Table 3-1: Background Information of the Teacher Participants Recruited in the Present Study
Background Information Textbook A (n = 30)
Textbook B (n = 30)
Textbook C (n = 30)
Total (n = 90)
1. Service year 10.25 10.87 10.87 10.67
2. Places of teaching
Taipei city Taipei county Out of Taipei
7 9 14
7 9 14
15 2 13
29 20 41 3. Use of NICT version
Yes No
14 16
16 14
8 22
38 52
Note. X stands for the year of teaching in schools. The boldfaced “38” indicates the number of teachers that used the NICT version before and their opinions would be analyzed later.
The 4 textbook writers (1 editor; 3 teacher writers) were recruited from the compiling committee in Publisher A only
36. They have compiled the whole set of Textbook A, from Book 1 to Book 6. The editor is a professor with bountiful experiences of language teaching and research. The other three writers are experienced senior high school teachers, knowing both learners’ and teachers’ needs well.
3.2 Instruments
Multiple instruments were adopted to collect data for the present study. The primary data were collected through a questionnaire designed for English teachers (see section 3.2.1) and an in-depth interview with teachers, teacher writers, and editors (see section 3.2.2). In addition, supplementary data were collected through an analysis of the textbooks (see section 3.2.3).
3.2.1 Questionnaire
The questionnaire (see Appendix C) used in the present study was adapted from
36 Originally, the researcher invited Publishers A, B, and C to share their opinions in compiling Textbooks A, B, and C, respectively. However, Publishers B and C, in fear of disclosing their confidential information, refused the invitation. Only Publisher A was interviewed.
Shih’s (2000) checklist for the evaluation of elementary English textbooks
37, and Shih’s (2002a) as well as Chen’s (2002) evaluation criteria for senior high English textbooks. It consisted of two parts: post-use evaluation of two versions (the currently-published one and the National Institute for Compilation and Translation (NICT) version) and suggestions for future compilation. The detailed information is summarized in Table 3-2:
Table 3-2: An Outline of the Questionnaire
Category n Contents
Part One: Textbook features (4-point Likert scale and an open-ended question) General features 10 overall organization, topics/text types, linkage
management, language appropriateness, etc.
Language skills 5 language activity design for each language skill (e.g. reading, writing), integrated skills, etc.
Language components 5 vocabulary selection, explanation for grammar points, etc.
Related resources 5 teachers’ manual, students’ workbook, etc.
Others (open-ended Q) 1 other strengths or weaknesses Part Two: Textbook functions (multiple choice)
Achievement 12 elevation of teachers’ power, increase of learners’ language abilities and learning motivation, etc.
Suggestions 16 practicability of teachers’ manual, readability of reading texts, variety of topics/text types, etc.
Others 1 needs for the standard version
Note. N stands for the total number of question items for each subcategory.
Part One focused on the teachers’ perceptions about the features of the specific version after they had used the textbook for a year or more. This part also elicited senior teachers’ evaluations of the NICT version. Thus, two columns were provided to
37 As mentioned in the previous chapters, the main reason for adopting Shih’s (2000) scheme is that her checklist is applicable to our local situation. According to her study, teachers and textbook
reviewers prefer a concise checklist, which can pinpoint important considerations in textbook selection and evaluation “within two pages” (p.218).
generate a comparison between the newly-published set and the NICT version.
Twenty-six items were included (25 questions on a 4-point Likert scale
38and 1 open-ended question). The 25 questions were further divided into five categories. The first category consisted of 10 items, which were used to elicit the teachers’ general evaluations of the textbook set used before. Questions such as overall organization, topic quality and arrangement, linkage management, language appropriateness of the textbook were asked. There were 5 items for the second category, which were used to tackle with the features of activity design for language skills. The third category included 5 items, which highlighted the presentation of language components such as vocabulary and grammar. The fourth category consisted of 5 items for teaching resources such as the teachers’ manual and students’ workbook. The last category was an open question about additional comments and suggestions.
Part Two was designed in a multiple blank-checking form, aiming to gather teachers’ viewpoints on current and future textbook compilation. The first question consisted of 12 items about textbook functions in the open editorial system. The second question provided 14 items about suggestions for future compilation. The last question was about the role of the NICT version if it would rejoin the market.
3.2.2 Interview
An in-depth semi-structured interview was designed to elicit more perceptions about the textbooks from textbook producers’ (i.e., editors and writers) and users’ (i.e., teachers). The questions were mainly taken from Ko’s (2005) and Pan’s (2004) interview question lists. The interview structure also followed the questionnaire outline in order to generalize the reasons behind textbook compilation and evaluation,
38 In the present study, 5 points are included on the scale: “4,” “3,” “2,” “1,” “0,” representing
“superior,” “good,” “fair,” “poor,” and “no offer,” respectively.
The interview questions for teacher users (see Appendix D), editors-in-chief (see Appendix E), and teacher writers (see Appendix F) all consist of four parts, as shown in Table 3-3:
Table 3-3: An Outline and Example Questions of the Interviews for Teacher Users and Textbook Writers
Textbook writers Core Questions Teacher users
Chief editors Teacher writers I. Textbook
features
Questions in total: 5 e.g., Why do you think
this textbook set have those good features?
Questions in total: 6 e.g., What features did
you set for your textbook set?
Questions in total: 5 e.g., What are the most
unique features does your textbook set have?
II. Compiling procedures
Questions in total: 6 e.g., Did the writers conduct a need analysis or a pilot study before textbooks are produced?
Questions in total: 5 e.g., Did you follow
any specific procedures in the compiling process?
e.g., What kind of difficulties you had in compiling?
Questions in total: 5 e.g., Did you follow any
specific procedures in the compiling process?
e.g., What kind of difficulties you had in compiling?
III. Textbook functions
Questions in total: 4 e.g., Why does this
textbook set provides the function you just checked?
Questions in total: 4 e.g., What functions
does your textbook already have? What are not yet
achieved?
Questions in total: 4 e.g., What functions does
your textbook already have? What are not yet
achieved?
IV. Future suggestions
Questions in total: 2 e.g., Do you have any
other suggestions for future textbooks?
Questions in total: 2 e.g., Do you have any
other suggestions for future textbooks?
Questions in total: 2 e.g., Do you have any
other suggestions for future textbooks?
Part One investigates the participants’ opinions about the features of the textbooks they used or compiled last school year. Part Two explores the compiling principles and procedures of the textbooks. Part Three generates the interviewees’
concerns about the functions of the current textbook sets, and the final part elicits
further suggestions for future textbook compilation. For more details, please see
Appendices D, E, and F.
3.2.3 Textbooks
In order to identify more features and principles of the textbook compilation process, three sets of senior high school English textbooks in use and the NICT version were analyzed. The three current sets are Chen’s (2002~2005) San Min English Reader for Senior High Schools, Lin, et al.’s (2001~2002) Lung Teng Reader for Senior High Schools, and Shih, et al.’s (2001) Far East English Reader for Senior High Schools. They were randomly given the pseudo-names as Textbooks A, B, and C.
3.3 Procedures
This part describes the pilot study in section 3.3.1 and the procedures of the formal study in section 3.3.2.
3.3.1 Pilot Study
To obtain comments for questionnaire revision, a draft of the questionnaire was given to four senior high English teachers from different schools. Two of them have taught for more than ten years, while the other two less than ten. The textbooks they used in the past school year were also different. Their personal information is shown in Table 3-4:
Table 3-4: Personal Information of the Four Participants in the Pilot Study
Participants T1 T2 T1 T2
Service year 2 3 23 23
School location Changhua Taipei Taipei Hsinchu
Textbook used in the past school year
Textbook B Textbook A Textbook C Textbook A
Use the NICT books before No No Yes Yes
After the participants finishing the questionnaire, an interview was further conducted. Based on their suggestions, some revisions were made. First, the 4-point scoring scale was expanded to include the point of “zero” for the option “no offer” of certain components. Second, in the part about “language components,” “the design of phrases” was added, since the teachers all considered it to be evaluated. Finally, the wording of some items was changed to avoid misconceptions.
3.3.2 Formal Study
In this section, the procedures of the formal study are described. Section 3.3.2.1 unfolds the steps of data collection, and section 3.3.2.2 introduces the way to analyze the data, obtained from the questionnaire, interview, and textbooks.
3.3.2.1 Data Collection
First of all, three sets of textbooks (Textbook A, Textbook B, and Textbook C) (i.e., eighteen textbooks in total) were collected from the market. Then, the questionnaire for textbook evaluation was distributed to voluntary teachers recruited from different senior high schools. It took them about twenty minutes to complete the questionnaire. Meanwhile, those who had used the NICT textbooks before were asked to fill out the columns for the evaluation of the NICT version as well. After the questionnaire was completed, a small token was given to the teachers to show the researcher’s appreciation. Based on the different versions of the textbooks the teachers had used in the last school year, the participants were divided into three groups: users of Textbook A, users of Textbook B, and users of Textbook C.
Thirty-eight teachers who also evaluated the NICT textbooks were extracted.
Thereafter, nine teachers (i.e., three from each set) were interviewed individually.
Those who evaluated the NICT set were also involved. For the sake of convenience,
the interview questions were given to the interviewees beforehand and the interview was conducted in Chinese. The entire interview took the participants one hour long to complete, either face to face or over the phone.
In the meantime, four textbook writers (i.e., one editor and three teacher writers) for Textbook A were invited to share their viewpoints further in a follow-up interview.
The interview questions were also given to the interviewees beforehand, and each entire interview took the participants about one hour. All the interviews were recorded, and the interviewees were told that their personal information would be confidentially concealed and removed till it was no more needed.
3.3.2.2 Data Analysis
The results of the questionnaire were keyed into the computer and analyzed through the statistical package SPSS. In Part One, the teachers’ evaluation scores for the first 25 items were summed up and averaged. Then, One-way ANOVA (Analysis of Variance) was adopted to compare the mean scores of the three versions item by item and to see how different the teacher users’ opinions were. Furthermore, to determine the distinct features of the three textbook sets, the 25 items were reclassified into five categories. They were syllabus organization (Items 1 to 5), content arrangement (Items 6 to 10), design for language skills (Items 11 to 15), presentation of language components (Items 16 to 20), and related resources (Items 21 to 25). The mean score of each category was also compared through One-way ANOVA. In addition, Paired-samples T-test was employed to compare the mean scores of the textbooks in use with those of the NICT version evaluated by the 38 participants. In Part Two, where more than one answer was allowed, the frequency counts were conducted to see to what extent and how many teachers would be satisfied with the given items.
The interview data were transcribed and then translated into English, and finally
qualitatively analyzed according to the procedures as shown in Table 3-5. The data obtained from the interviews with the teacher writers were coded with the framework provided by Jolly and Bolitho (1998: 98), for the identification of the process of textbook writing:
Table 3-5: Main Steps in the Production Path of New or Adapted Materials (Jolly &
Bolitho, 1998: 98)
Steps Description
1. Identification of need Identification by teacher or learner(s) of a need to fulfill or a problem to solve by the creation of materials
2. Exploration of language Exploration of the area of need/problem in terms of what language, what meanings, what functions, what skills etc.
3. Contextual realization Contextual realization of the proposed new materials by the finding of suitable ideas, contexts or texts with which to work
4. Pedagogical realization Pedagogical realization of materials by the finding of appropriate exercises and activities and the writing of appropriate instructions for use
5. Physical production Physical production of materials, involving consideration of layout, type size, visuals, reproduction, tape length etc.
6. Student use of materials Pilot on student use of materials
7. Evaluation of materials Evaluation of materials against or agreed objectives
In order to see if the teacher writers followed any principles in textbook writing,
the interview data were further coded with Bell and Gower’s (1988) framework (see
Table 3-6).
Table 3-6: Key Principles for Materials Writing (Summarized from Bell and Gower (1998)) Principles Explanation
1. Flexibility Design a pedagogically-working sequence for activities, but still with flexibility to move them around or cut them out if necessary.
2. From text to language
Provide authentic texts which contained examples of the focus language, rather than construct texts of your own.
3. Engaging content
Choose generative texts that would motivate students to talk or write. Choose texts which focused on many old, favorite topics (relationships, clothes, money etc.), but find new angles.
4. Natural language
Expose students to real and unscripted language (i.e., language that native speakers use to communicate naturally) in order to motivate them and help get them off the learning plateau.
5. Analytic approaches
Practice students’ conscious mind in analyzing the grammatical structures that they have been familiar with but they need to work more on—the difference between the Present Perfect and Past Simple, or will and going to for example.
6. Emphasis on review
Re-present the grammar points but in general fluent and accurate use, rather than get across the ‘meaning’ and use of the structure.
7. Personalized practice
Provide practice activities for learners to draw on their own experience.
8. Integrated skills
Both language work and the productive skills should come out of work on listening and reading texts.
9. Balance of
approaches
Balance in approaches to grammar (inductive and deductive), fluency-accuracy work, and both controlled and creative practice.
10.Learner development
Integrate learner development work throughout rather than make it
‘up-front’ training. Nevertheless, provide up-front work on vocabulary skills or grammar work at the end of each unit.
11.Professional respect
Produce something that can give professional satisfaction and is academically credible to the colleagues.
Note. The statements are taken from Bell and Gower (1998: 121-125) but summarized by the researcher.
