Pilot Study
The pilot study was conducted to see whether the treatment material and procedure, as well as the test material and procedure were appropriate for the data collection. Participants for the pilot study were composed of forty 12th graders of senior high who were not included in the main study. Results of the pilot study are presented in the following two parts. The first part is about the treatment. The second part regards the pre- and post-test.
Pilot Study on Teaching Material and Procedure
The appropriateness of QtA treatment material and procedure would be the first main concern in the pilot study. Within two class periods, students were instructed to read a narrative story in QtA approach followed by a 10 multiple-choice reading comprehension questions.
The treatment story, “The Blanket,” in pilot study is one of the six treatment stories to be adopted in the main study. The selection procedure of the treatment material is illustrated with detail in the Main Study section. The reason why “The Blanket” was used in pilot study is that its number of words and readability level are closest to the average of treatment materials of the main study.
The story was divided into 15 segments and was meant to be covered within two class periods. The first period began with 10 minutes of QtA orientation and 40 minutes to cover the first half of the story with 8 segments. The remaining 7 segments constituting the second half of the story were read and discussed in 35 minutes in the second class period. The last 15 minutes in the second period were left for students to do the 10 multiple choice questions (five factual questions and five interpretive questions). The detailed steps of QtA instruction are elaborated in the Main Study section.
In the first period of the pilot treatment, students were found to be not motivated in participation. Therefore, three changes were made in the second period, which led to improved classroom participation. As a result, these changes were implemented in the main study.. In the first change, to boost students’ initiations, students were arranged into small groups and rewards were given to the group volunteering to respond. Secondly, more Follow-up Queries like “What do you think about other peers’ comment?” would be adopted to prompt more elaborated interaction. By applying this kind of query, students will attend to topics of current discussion more. They would listen more attentively to others’ response and co-construct the meaning of the text together. Thirdly, some students responded that they needed more time for discussion. The number of segments would therefore be trimmed to 12 segments, instead of 15 for each story, to allow more extended discussion in each segment.
In the end of the treatment, a tally from the in-class poll was given and up to 93% of the students considered the text appropriate in terms of difficulty level. What’s more, around 80
% of the participants thought their comprehension of the story was improved with the implementation of QtA.
Pilot Study on Test Material and Procedure
The other purpose of the pilot study is to examine the suitability of test material and procedure.
In one 50-minute class period, the same participants were instructed to finish four tasks (reading, response writing, short-answer questions, and perception questionnaire). For the first ten minutes, students were instructed to read a fable story closely. After 10 minutes, the fable story was collected and students were required to write a response in about 120 words regarding anything they want to say about the fable. When the response writing was finished in 20 minutes, they were asked to do the third task of answering 9 short-answer comprehension questions (three factual questions; three interpretive questions; three responsive questions) in 10 minutes. The last 10 minutes was reserved for the students to
complete perception questionnaires for the QtA process.
After the test, two major changes on the test procedure and test material were made for the main study. For the procedure, the time for each task was lengthened. The time allotted for response writing was extended to 25 minutes instead of 20 minutes. Fifteen minutes instead of 10 minutes were reserved for answering the short-answer comprehension questions. And the questionnaire was shifted to the second class period to prevent students from rushing their response writing. For the material, an informal in-class survey showed that students had a hard time comprehending the fables because the English animal species were strange to them.
To assist students’ comprehension of the fables, the Chinese translation of the animal species was given in the margin of the reading sheet to reduce the perception of difficulty.
Main Study Participants
Two classes of 11th graders (94 students) in a girls’ senior high school in Taipei city were recruited for the experiment. One class of forty-seven students served as the Experimental Group. The other class with forty-eight students was the Control Group. After data collection, one student from the Experimental Group was excluded from data analyses because she did not participate in the pretest, thus leaving forty-six students in the Experimental Group.
The participants had received formal English education at school for at least six years.
They began to learn English formally when they were in the fifth grade in the year 2006.
In the first year of their senior high, students had six periods of English and one period of English Conversation per week. The experiment was executed in the second term of their second year in the applied English program. For students in the second year in applied English program, they have various kinds of English courses per week, including one period of English, one period of English Conversation, two periods of English Speech, two periods of English Novel, four periods of English Listening, and four periods of English Reading. The
researcher taught English Reading to both groups in their second year. The treatments and tests were administered as part of the course content. The students of these two intact classes were of a homogenous level of English reading proficiency, for they obtained similar average scores of the English Reading Periodical Exam in the previous semester: one class is 72.3; the other is 73.1.
Materials Selection
The selection of the materials is described in three stages. The first part illustrates the selection of text genre. The second part depicts the selection of stories for the treatment sessions. The third part is about the selection of text for the pretest and the posttest.
Deciding text genre for treatment and for pre and posttest. Questioning the Author technique, according to Beck and McKeown (2006), has been successfully used in both expository and narrative texts. Narrative, short story genre was chosen in the current study.
The following are three reasons why narrative texts rather than expository texts were adopted in the study. Firstly, while reading stories, readers often identify with certain characters and therefore are motivated to read along (Collie & Stephen, 1987). In addition, narrative texts are often based on common events containing more dialogues and less abstract terminology than expository texts, so they could better engage readers. Thirdly, with intriguing plots and imbedded themes, stories allow for more inferences and multiple interpretations than plain factual description (Carter & Long, 1991). This suits the purpose of discussion in QtA and response writing in assessment, which requires readers to respond to, make connection and generate personal meaning. In addition, such extra efforts correspond to the text criteria for developing language and comprehension ability proposed by Beck and McKeown (2001b), who contended that a text should be “conceptually challenging enough to require grappling with ideas and taking an active stance to construct meaning” (p.10). With the above reasons, short stories, instead of other genres, were opted as the main type of assessment and treatment material.
Teaching materials. The second stage was to determine the stories for treatment sessions. Six short stories from Discovering Fiction, reader series of North American Short Stories, by Cambridge University Press (Kay, 2008) were chosen (see Appendix A). This reader series include American classic short stories by diverse authors in the 19th to 20th century. The reason why American short stories are chosen rather than stories from other countries is that American English is still the dominant style in Taiwanese English education (Lai, 2008). Therefore, it is assumed that the participants would be more familiar with its language style.
Stories chosen for the current study not only contain intriguing plot but also follow the event structure with the elements of setting, characters, plot, conflict and theme. The average readability level of the stories was 3.6, ranging from 2.9 to 4.6 and their average number of words was 1070, ranging from 979 to 1216 (see Table 1). A survey of English textbook (Far East, Book 3) used by participants in the semester showed that its average readability was 6.3, ranging between 3.7 and 8.6 and the average number of words was 440, ranging between 260-540.
Although the treatment stories were easier and longer than the textbook articles used by the participants in the previous semester, they were still perceived suitable based on the following two reasons. First, one treatment story, “The Blanket,” with 1051 words and readability level 3.5, which was closest to the average, was tried out in the pilot. Participants were able to be guided through this 1051-word story under QtA within two periods of classes.
In addition, around ninety percent of students considered the text appropriate in terms of difficulty. Secondly, each article form the textbook is meant for intensive language study within eight class periods. In QtA lessons, we have longer passage read in two class periods with an aim of understanding story message not of close language study. As a result, the other five stories with similar difficulty level, length, and event structure are also considered appropriate for treatment in the study.
The order of the six stories in the treatment session was arranged according to the number of words in each passage from the shortest to the longest (see Table 1).
Table 1
Treatment Materials
Title Word Count Readability
One Thousand Dollars (O Henry) 979 4.1
The Pace of Youth (Stephen Crane) 1004 3.2
Snake Dance (Zona Gale) 1042 2.9
The Blanket (Floyd Dell) 1051 3.5
The Californian’s Tale (Mark Twain) 1132 3.0
You Go Your Way, I’ll Go Mine (William Saroyan) 1216 4.6
average 1070 3.6
Test materials selection and modification. Two modern fables by James Thurber (1983a; 1983b) were selected for the reasons below. To begin with, fables are brief tales depicting animals acting like humans and containing a moral lesson about human behaviors (Abrahamsen & Sprouse, 1995). Fables are less straightforward in delivering its meaning, thus need readers make more effort to figure out the meaning and allow for the readers’
various interpretations about the theme. These characteristics made fables suitable to serve as the test material. Among types of fables, modern fables were chosen because they were unfamiliar to the participants compared to classics such as Aesop’s fables. In addition, the two modern fables were written by the same author, James Thurber, which made the two texts more comparable. They are both about an animal or an insect pursuing his dream with dramatic ending and a climax at the last paragraph. Table 2 shows the numbers of words and readability of the test materials.
Since the readability of the two passages (5.4 for The Moth and the Star and 6.8 for The Seal Who Became Famous) are not equal, three modifications were done to ensure the
comparability and the suitability to the target readers. Firstly, the moral lessons of the two fables, stated in the end of the story, were deleted so that students would not be restricted to the author’s interpretation of the stories and can have their own. Secondly, to make sure at least 90 percent of the vocabulary in the texts was within the Senior High Basic English Vocabulary List (CEEC, 2002), the researcher and her English teaching colleague read through the two fables and worked together to replace the difficult words with easier ones.
For example, in the first passage, the vocabulary, “singe” and “scorch,” are replaced by
“burn.” In the second passage, difficult terms such as “a gardenis in his lapel” were substituted by easier ones, “a flower in his collar.” Thirdly, complex sentence patterns were also simplified by breaking one lengthy sentence into two shorter sentences. For example, “A seal who lay basking on a large, smooth rock said to himself …” was revised into “A seal lay in sunlight on a large, smooth rock. He said to himself …”
Table 2 The Seal Who Became Famous Original
Modified
328 315
6.8 4.5
The resultant readability of the two fables reaches 4.5 (see Table 2). The two passages were piloted with forty 12th graders and an in-class informal inquiry indicated that around 70% students considered the length and difficulty of the two passages appropriate. Therefore, materials for pretest and posttest were decided (see Appendix B). Pilot test confirmed the suitability but the Chinese translation of animal names, moth and seal, were provided since students reported difficulty in comprehension.
Segmentation and generation of queries for treatment material. Each treatment stories were divided into 12 segments where students pause to discuss. The segmentation of the stories was based on the researcher’s major understanding related to the story theme. The range of one segment could be one sentence or several paragraphs, as long as there is an essential concept within (Beck & McKeown, 2006).
For the six treatment stories, queries were prepared for each segment in each story. After reading each segment, teachers would direct queries to students to prompt their discussion.
The queries used in the current study can be categorized into Initiating Queries, Follow-up Queries and Narrative Queries, according to the suggestions by Beck and McKeown (2006).
Initiating Queries are designed to open the discussion among readers, such as
”What is the author trying to say here?” “What is the most important message in this section?” After readers start to talk about the text, Follow-Up Queries will be applied to help students focus on the content, encourage deeper thought, connect ideas and elaborate response, such as “That’s what the author said, but what did the author mean?” “Why do you think the author tell us that now?” “How does this connect with the previous concepts?” Furthermore, Narrative Queries will also be adopted to foster readers’ understanding and appreciation of the story by having students pay attention to authors’ manipulation of narrative elements ( Beck et al., 1996), such as ”What has the author told us about the character?” “What happened here?”
“How did the author settle that for us?” “How has the author let you know that something has changed in the story?” “Given what the author has already told us, how do you think (character’s name) will handle this situation? “The above three types of queries are essential to lead readers to have a focused and productive discussion.
Comprehension check for treatment material. For each of the six treatment materials, multiple choice questions for comprehension check were developed. The comprehension check was used to motivate students to concentrate more on the stories, the data of which was not analyzed in the study. It included five factual questions and five interpretive questions.
Instruments
Three instruments were adopted to answer the research questions. The first one was two sets of short-answer comprehension questions for the pretest and posttest reading respectively (Appendix C). The second one was response writing prompts for the pretest and posttest reading (Appendix D). The third one was a perception questionnaire for QtA (Appendix E).
Short-answer comprehension questions. In the pretest and posttest, nine open-ended short answer questions were designed for each of the reading passage to assess understanding after reading (see Appendix C). They consisted of three factual questions, three interpretive questions and three responsive questions. According to Applegate et al. (2002), factual question requires the readers to recall what they have read directly from the text. Interpretive question goes further to ask the readers to use their background knowledge, the schema, to draw a conclusion about the text or to fill up textual gaps, such as the theme or inferences.
Responsive question has the reader express an idea freely related to the behavior of the characters or the story plot.
Though students were mostly encouraged to answer the short answer questions in English, they were allowed to supplement with Chinese if they had trouble expressing ideas.
The participants could use either L1 or L2 in this section since it was the idea not the language that would be evaluated.
Written response instruction sheet. To authentically tap into how readers generate ides after reading the story, one written response sheet was designed for pretest and posttest (see appendix D). Readers were asked to write a 120-word1 response with the prompt of “Write anything you want to say about the story you just read” (Many, 1992). Participants were encouraged to use English to write their response and Chinese was allowed when they have trouble expressing in English. The purpose of the response writing was to assess students’
1 120 words are the limit of word number for English writing in College Entrance Examination.
interpretation and reflection of the story.
Perception questionnaire. The questionnaire was designed to investigate students’
perception toward QtA lesson and to understand students’ self-evaluation of learning with QtA (see Appendix E).
The first section asks the participants to choose some of their favorite features of QtA approach. The second section encompasses students’ self-assessment of growth in general English ability, reading ability and thinking ability. The third section is for free sharing about QtA. In this section, five questions are asked. The first two questions included their affective response to the text in QtA lessons and the intention to take more QtA classes in the future.
The next two were open-ended questions, with one asking students the reasons why they like or dislike QtA approach, and the other having participants state whether they had applied QtA technique while reading on their own. The last question, again open-ended, was to confirm whether the participants had received other similar reader-based instruction.
Study Design
The design for the treatment and the pre and posttest is demonstrated in Table 3. The participants had four periods of English Reading course in one week. In the first and second period of the first week, the pretest was implemented. Then from the third and fourth period of the first week, six interventions were implemented consecutively. One intervention took two class periods. The posttest was carried out in the third and fourth period of the fourth week.
Table 3
Data Collection and Treatment Schedule
period 1-2 period 3-4
week 1 pretest intervention 1
week 2 intervention 2 intervention 3
week 3 intervention 4 intervention 5
week 4 intervention 6 posttest
Treatment procedure. The experiment was carried out at the beginning of the second semester in semester year of 2012. The experimental and control group received instruction of one treatment story for two consecutive periods in six sessions within four weeks. The treatment procedure for each group is stated in the following sections.
Treatment procedure. The experiment was carried out at the beginning of the second semester in semester year of 2012. The experimental and control group received instruction of one treatment story for two consecutive periods in six sessions within four weeks. The treatment procedure for each group is stated in the following sections.