Chapter Three Research Design
In this chapter, the overall design of the present study will be reported. Section 3.1 describes the participants of the study. Section 3.2 presents the instrument, including listening lectures, immediate retrospective verbal reports, a coding scheme, and semi-structured interviews. Section 3.3 describes the procedures of the present investigation. Finally, section 3.4 summarizes the main points of this chapter.
3.1 Participants
The participants consisted of 12 Chinese students from National Taiwan Normal University (NTNU), as can be seen in Table 3-1:
Table 3-1: The Information of the Participants
Groups Subject Gender Number of Years of English Learning
LC Score
G1S1 10 11
G1S2
M
9 8
G1S3 6 6
Group 1
G1S4
F
6 11
G2S1 10 8
G2S2
M
12 12
G2S3 13 15
Group 2
G2S4
F
14 14
G3S1 14 13
G3S2
M
14 13
G3S3 13 11
Group 3
G3S4
F
13 12
Note: The score of the listening comprehension test ranged from 0~15.
They were four non-English major freshman students from the higher-intermediate level of Freshman English, four sophomore students, and four
graduate school students from the English Department. Their average number of years of learning English was 7.75 for Group1, 12.25 for Group 2, and 13.5 for Group 3.
Therefore, they were assumed to be more appropriate for investigating metacognitive listening strategies. As for the score of their listening comprehension (LC) test1, the average for Group1 was 9, that for Group 2 12.25, and that for Group 3 12.25 and they all passed the standardized placement test designed by the English Department with regard to listening comprehension. Therefore, with their rich learning experience and over 9 points of listening comprehension test averagely, the participants were regarded as successful learners. In addition to plenty of English learning experience, Group 3 had one more English teaching experience in language institutes or cram schools.
3.2 Instruments
Three major research instruments were used in the present study, including listening lectures, immediate retrospective verbal reports, and semi-structured interviews.
3.2.1 Listening Lectures
The listening passages used in the present study were three lectures (one for the training session and the other two for the data collection session) selected from Intermediate Listening Comprehension2. Since lectures in this textbook were organized in rhetorical patterns, such as chronology, process, classification/definition,
1 The standardized listening comprehension test was designed by the English Department at NTNU for the Freshman English placement test (Fall, 2006) and NTNU English Proficiency test (Spring, 2006).
The level of the test was between GEPT intermediate and GEPT higher-intermediate. There were three parts of text types in the listening comprehension test, including answering questions, conversations, and short talks. Only the scores of short talks were used in the present study and there were 15 multiple choice questions in the section of short talks. One point was given when the test-taker chose the correct answer.
2 The book was written by Patricia Dunkel & Phyllis Lim in 1994 and published by Heinle & Heinle.
comparison/contrast, and causal analysis, three lectures used in the present study were taken from the same category, the category of chronology3. The lecture of Pompeii:
Destroyed, Forgotten, and Found was used in the training session. The other two
passages were used in the data collection session. The lecture of Napoleon: From Schoolboy to Emperor described the well-known figure Napoleon’s school and
military life throughout his life; the lecture of Harriet Tubman: A Conductor on the Underground Railroad introduced Harriet Tubman as the remarkable woman in the
United States history (See Appendix A for the script). In order to avoid any procedural problems that might influence the result of the present study, the lecture Pompeii and Harriet Tubman were shortened to be similar in length with Napoleon4. In addition, the three lectures were similar in the complexity and difficulty. The criteria for choosing these listening passages were based on participants’ background knowledge and familiarity with the historical figures. The Napoleon passage was assumed to be historical familiar since he was introduced in the participants’ history textbooks in high school and also widely known to them in Taiwan. The Harriet Tubman passage was assumed to be historical unfamiliar because she was not discussed in high school textbooks and rarely known to students in Taiwan. Table 3-2 provides more detailed description of the lectures.
3 There are three lectures for each genre. And there are 15 lectures in total in the textbook. All the lectures are authentic, conducted by native speakers in natural speech. And these lectures differ in topics and lengths.
4 The three lectures in the genre of chronology are of different length. The passage about Napoleon is the shortest of the three; hence, the researcher shortened the other two passages. And in order to keep the completeness of idea units and coherence of the lecture, the three lectures were made similar in duration and number of words in total.
Table 3-2: Description of listening lectures Listening
Lectures
Duration min, sec.
Number of words (complete)
Number of words (chunked) Pompeii:
Destroyed, Forgotten, and Found
1’50’’ 263 131
Napoleon:
From Schoolboy to Emperor
1’38’’ 212 157
Harriet Tubman:
A Conductor on the Underground Railroad
2’07’’ 324 227
3.2.2 Immediate Retrospective Verbal Reports 3.2.2.1 Introduction of Verbal Reports
Introspection has become increasingly popular in listening strategy research (Goh, 1998). According to Ericsson and Simon (1987), introspection means a process by which individuals “think aloud” the mental steps they take or “talk aloud” the mental images they see when completing a task. Namely, it is the “observation of the contents of one’s mind” (p. 43). In listening research, participants are typically asked to listen to a selected text and then verbalize the process they undertake and the strategies they use to help them comprehend.5 Previous studies employing this method include Fong, Lin, Shih, & Tseng (1998), Goh (1998, 2002), James (1987), O’Malley, Chamot, Kupper (1989), Vandergrift (1997, 2003a), and Young (1996).
There are two types of verbalization6 (Ericsson & Simon, 1980, 1993), as shown in Figure 3-1.
5 When using introspective procedures, it is important to adopt a robust framework for eliciting and interpreting the data. On the basis of Ericsson and Simon’s (1980, 1993) framework, there are some conditions for accessing the information and reporting it verbally:
(i) Information recently heeded by the central processor is kept in short-term memory (STM) and is directly accessible for producing verbal reports. Since STM has limited capacity, only the most recently heeded information is accessible directly.
(ii) A portion of the contents in STM is fixed in long-term memory (LTM) before being permanently lost from STM, and this portion can, at later points in time, be retrieved from LTM. After retrieval, this
“old” information must be transferred to STM before it can be reported. (Ericsson & Simon, 1993:11)
6 The terns verbalization and verbal report are interchangeable in this study.
Figure 3-1: Description of verbal report7
The first type is “concurrent verbal reports—‘talk aloud’ and ‘think aloud’
reports—where the cognitive process, described as successive states of heeded information, are verbalized directly” (1993:16). The second type is retrospective report, accessing from short-term memory (STM), from the durable memory trace
laid down of the information heeded successively while individuals complete a task after the task is finished. In the present study, all the verbal reports are retrospective.
3.2.2.2 Rationale for the Use of Verbal Report
There are different kinds of instrument for the research of listening strategies, including questionnaires, interviews, and verbal reports. Each instrument has its own drawbacks and limitations. Questionnaires can indicate participants’ perceived usefulness of listening strategies, instead of actual use of strategies. Furthermore, it may remind participants of potential listening strategies they can use in the process of completing questionnaires. Although questionnaires can be administrated widely in scope and in quantity, they cannot provide in-depth information about participants’
actual use of strategies and when, why, and how to use these strategies. In other words, questionnaires cannot elicit the desired data for the present study. However, verbal report (think-aloud), both concurrent and retrospective, can provide on-site strategy
7 Introspective verbal reports are similar to think-aloud procedures.
Introspection
Introspective verbal report
Concurrent verbal report Retrospective verbal report
use of participants. Therefore, immediate retrospective verbal reports are chosen to be the instrument for the study.
According to Branch (2001), the verbal protocol analysis method is an excellent choice for qualitative research to provide a rich source of data. More reasons are given by Goh (2002:189), as follows:
(1) Verbal data on listening processes are predominantly retrospective.
Because of the rapid flow of information, the working memory has to be freed for processing continuous input. What listeners will typically do is to process the heeded input first before reporting through retrospective verbalization.
(2) No extra demands are made on processing capacities during listening because retrospective verbalizations do not interference with the processing of input. What may be expected, however, are incomplete verbalizations because learners may have problems expressing some things in the target language8. Although this may render some information inaccessible, it does not invalidate the information reported.
(3) Probes9 that do not require informants to consult their memories about actual cognitive process that take place should not be used. They should be asked only to describe how they try to understand what they hear.
The method of verbal reports as data, however, has been criticized ever since it was first introduced in the late 19th century in psychological experiments. To validate the research procedure, Ericsson and Simon (1980, 1993) have made a strong condition for the use of verbalization by specifying the situation where data should be collected and interpreted. As an SLA researcher who criticized against the use of
8 Participants’ L1 would be used in the present study to reduce the incompletenesss.
9 Probe questions are provided in verbal reports to facilitate the participants’ thinking aloud when they are not sure how to continue.
verbalization, Seliger (1983) argued that “language learners cannot take on the role of linguists in producing information about learning process” (cited from Goh, 1998, p.
128). White (1980) further asserted that consciousness “is limited to the products of mental processes and the processes themselves are beyond the reach of introspection”
(p. 105). Therefore, he concluded that “the conscious introspections of language learners are really the product of underlying processing and not the process itself” (p.
187), adding that there are some other problems with introspection such as participant’s short memory span in another language and the risk of participants to make inference about what they did.
Despite the criticism, verbal reports have been widely used in the field of SLA (Faerch & Kasper, 1987; Matsumoto, 1993). Goh (1998) elaborated the reasons for the increasing popularity of verbalization. First of all, learner strategy research has created a demand for procedures which can elicit data about what learners know and do. This information can normally be available only when learners are required to introspect and think aloud (Wenden & Rubin, 1987). More importantly, arguments supporting the role of consciousness in language learning imply that learners can be aware of their own learning process (Johnson, 1996; McLaughlin, 1990; Schmidt, 1990). Another reason for gaining respectability of verbal reports as a research method is the improvement which has been made to avoid the related pitfalls (Cohen, 1996; Matsumoto, 1993). Grotjahn (1987) claimed that improved research design for verbalization, such as methodological triangulation, can be used in qualitative, exploratory and quantitative, hypothesis-testing studies. For example, Van and Abraham (1990) combined an introspective think-aloud task to examine the unsuccessful language learners’ strategy use.
3.2.2.3 Probe Questions
Probe questions were provided with caution when listeners did not verbalize the desired information. With regard to probe questions used in verbal reports, Ericsson and Simon (1980, 1993) recommended avoiding two types of probes. The first type is not to require participants to explain the reason for doing something that has been observed because it may generate answers based on their knowledge instead of their cognitive process. The second type is a question about a hypothetical state. Questions should concentrate on participants’ beliefs and perceptions, not on what happened.
This has critical implications for interpreting data. In the present research, only reports on actual cognitive process were considered in the identification of metacognitive strategies. Comments about listening in general or in hypothetical states were excluded.
As Goh (2002) declared, “For the immediate verbalization sessions, the informants listened to a pre-chunked text with pauses. At each pause, they reported how they had tried to understand the proceeding segment” (p. 189). Listeners were asked to reflect on how they made sense of what they had heard, whether there were unfamiliar words or phrases, what they did not understand, how they figured it out, and what memories or images occurred to them while listening. Probe questions were provided when the participants seemed to be uncertain about how to continue, for example, “What are you thinking right now?” “What is going on in your mind?”
“What did you not understand?” “How did you figure that out?” “Can you be more specific?” (See Appendix B).
As the literature suggested (cf. Brown & Day, 1083; Flottum, 1986), use of L1 can avoid encoding difficulties that might obscure the intention of the subjects. Thus, to reduce the incompleteness of verbal reports, the use of the participants’ native language, Mandarin Chinese, during the verbalization was allowed in the present
study.
3.2.2.4 Design of Immediate Retrospective Verbal Report
Based on the modal of O’Malley, Chamot, and Kupper (1989) and Rankin (1988), a verbalization procedure has two separate phases: a training phase and a data collection phase (Vandergrift, 1997). Listeners were pre-trained on thinking aloud in the training session. They might speak out a summary of what they heard and any other thought “running though their minds” (Anderson & Vandergrift, 1996) during the pre-chunked intervals. The breaks were determined by natural discourse boundaries. The points were pre-selected and identified on the tape scripts and the sequence was rigorously followed for each participant.
3.2.2.5 Coding Scheme
Protocols were analyzed according to a predefined taxonomy of listening comprehension strategy. Although coding was guided by the classification scheme, it was open to add new categories. According to Mann (1982), any strategy that does not fit the categories of analysis would be formally defined with an example for reference in the following coding process.
In the vain of Vandergrift’s (1997) taxonomy, there are four major categories of metacognitive listening strategies: Planning, Monitoring, Evaluation, and Problem Identification. The Planning strategy encompasses Advanced Organization, Directed Attention, Selective Attention, and Self-management. The Monitoring strategy includes Comprehension Monitoring, Auditory Monitoring, and Double-check Monitoring. The Evaluation strategy refers to Performance Evaluation and Strategy Evaluation. And the last category is Problem Identification. Table 3-3 shows Vandergrift’s taxonomy.
Table 3-3: Vandergrift’s (1997) Taxonomy for Metacognitive Strategies10
1. Planning: Developing an awareness of what needs to be done to accomplish a listening task, developing an appropriate action plan and/or appropriate contingency plans to overcome difficulties that may interfere with successful completion of the task.
Type Definition Example
1a. Advance Organization Clarifying the objectives of an anticipated listening task and/or proposing strategies for handling it.
I read over what we have to do.
I try to think of questions the teacher is going to ask.
1b. Directed Attention Deciding in advance to attend in general to the listening task and to ignore irrelevant
distractors; maintaining attention while listening.
I listen really hard.
I pick out the words that are familiar so that … (in combination with
inferencing) 1c. Selective Attention Deciding to attend to
specific aspects of language input or situational details that assist in understanding and/or task completion.
I listen for the key words.
1d. Self-management Understanding the conditions that help one successfully accomplish listening tasks and
arranging for the presence of these conditions.
I try to get in the frame of mind to understand French.
I put everything aside and concentrate on what she is saying.
2. Monitoring: Checking, verifying, or correcting one’s comprehension or performance in the course of a listening task.
Type Definition Example
2a. Comprehension Monitoring
Checking, verifying, or correcting one’s
understanding at the local level.
I translate and see if it sounds right (in combination with transfer).
I just try to put everything together, understanding
10 This table is taken from Vandergrift (1997: 391-393).
one thing leads to understanding another.
2b. Auditory Monitoring Use one’s “ear” for the language (how something sounds) to make
decisions.
I use my knowledge of Portuguese, primarily sound (in combination with transfer).
I use the sound of words to relate to other words I know.
2c. Double-check Monitoring
Checking, verifying, or correcting one’s
understanding across the task or during the second time through the oral text,
I might catch it at the end and then I’d go back.
Sunny in the morning, that’s not making sense…(earlier) it
sounded like a cold front, something doesn’t make sense to me any more.
3. Evaluation: Checking the outcomes of one’s listening comprehension against an internal measure of completeness and accuracy
Type Definition Example
3a. Performance Evaluation
Judging one’s overall execution of the task
How close was I? (at the end of a think-aloud report).
3b. Strategy Evaluation Judging one’s strategy use I don’t concentrate too much to the point of translation of individual words because then you just have a whole lot of words and not how they are strung into some kind of meaning.
4. Problem Identification Explicitly identifying the central point needing resolution in a task or identifying an aspect of the task that hinders its successful completion
I’m not sure but
“partager” and I’m not really sure what that means.
3.2.3 Semi-structured Interview
Due to the criticism of verbal report data, Garner (1988) proposed that adopting a multimethod rather than a single method in identifying learners’ strategic repertoires can avoid unnecessary ambiguity and criticism. Therefore, a semi-structured interview was conducted to elicit more information about the participants’ metacognitive strategy use as well as their attitudes. Well-constructed interviews as a research tool have gained credence in recent years (Chamot, 1987; Cohen, 1987; Rubin, 1987;
Wenden, 1987). As Chamot (1987) noted, “what emerges is the fact that the best way to get at what strategies learners actually use…is to ask them (p. 87).” The use of interviews “allows for greater depth” than other methods of data collection (Cohen &
Manion, 1994: 272). Previous studies have adopted semi-structured interviews to investigate learners’ strategies (Chamot & Kupper, 1989; Vandergrift, 1996;
Zimmerman & Pons, 1986). Instead of a multiple choice format, an interview can avoid reminding the participants of any metacognitive strategies. Individual interviews can avoid the prompt of one listener’s suggestion of strategy use to another.
Semi-structured interviews feature the use of an interview guide (Bernard, 2000). Its advantage lies in that the interviewer can carefully decide how best to use the limited time available in the interview situation and focus on the relevant topics. Although the present interview was guided by prepared questions, it was open to the researcher to pursue other avenues prompted by the participants’ responses and to probe into more information for the less verbal participants.
The main purpose of the interview guide was to know more about the participants’ metacognitive strategy use, attitudes towards strategy use, beliefs about language learning and listening, and learning history (see Appendix C). The interview guide used in the present study was derived from the following sources: (1) Oxford’s study (1990) and Teng’s (1996) revision in Chinese, and (2) Beliefs About Language
Learning Inventory questionnaire (BALLI, EFL version, Horwitz, 1987). The interview guide adapted from Teng’s study aimed at exploring the participants’
attitudes towards their metacognitive strategy use. In order to further investigate the participants’ beliefs about listening and the influence of metacognitive strategy use, several guidelines were adapted from the BALLI for listening. Finally, interview questions about the participants’ learning experience and history were asked to examine what contributed to their metacognitive strategy use.
As evidenced in Section 3.2.2.3., using subjects’ L1 to conduct the interview can avoid the incompleteness of interview responses. Therefore, the interview was conduct in Mandarin Chinese.
3.3 Procedures
A pilot study was conducted before the formal experiment of immediate retrospective verbal reports and semi-structured interviews. It was conducted on an individual basis and audiorecorded thoroughly for later verbatim transcription and coding. After the coding of verbal reports, few new categories were added to the coding scheme (i.e., Vandergrift’s classification scheme) because some strategies did not fit the categories of analysis. After some revisions were made to the coding scheme based on the result of the pilot study, participants were chosen. Immediate retrospective verbal reports and semi-structured interviews were conducted individually and recorded thoroughly.
3.3.1 Pilot Study
Two participants (one undergraduate student in his fourth year, labeled as PU, and one graduate student in his third year, labeled as PG) took part in the pilot study
in September, 200611. Before entering the training phase of immediate verbal reports, they were informed of the purpose of the study and the overall procedure. After the training phrase (see Appendix D for the training dialogue), in which the participants practiced thinking aloud their metacognitive strategy use, they entered the data collection phrase, listening to two lectures and verbalizing what they did to achieve comprehension. Each lecture was played twice, a natural-speed version for the first time, a chuck-by-message-unit version for the second time. During the second listening in the pauses, they reported on what they had done and how they had tried to understand the proceeding aural input. Probe questions were provided when they did not know how to proceed, when they merely translated what they had heard into Chinese, and when they commented on the content of the lectures instead of what they manipulated to comprehend. After the verbal report, they were interviewed with the interview guide.
The results of the pilot study (see Table 3-4) indicated that Selective Attention, Double-check Monitoring, and Problem Identification were the metacognitive strategies most prevalent. For PG, Problem Identification and Double-check Monitoring were the metacognitive strategies most commonly used; for PU, Selective Attention and Double-check Monitoring were of the highest frequency.
11 The pilot study was conducted during the summer vacation. And it was difficult to recruit the participants who met the criteria of the present study. For the research convenience, these two participants joined the study voluntarily.
Table 3-4: The Results of the Pilot Study
PG PU Category
Familiar Un-familiar Familiar Un-familiar
Selective Attention 11 3 11 15
Comprehension Monitoring 5 8 4 10
Directed Attention 3 3 3 11
Double-check Monitoring 10 8 7 9
Problem Identification 6 13 4 11
Anticipation 2 1 9 6
Comprehension Confirmation 3 4 3 7
Helper Identification 2 1 1 0
Real-time Assessment 2 5 2 2
Advance Organization 1 0 1 1
Auditory Monitoring 0 1 0 0
Motivation Adjustment 0 1 2 3
Strategy Evaluation 0 3 1 1
Performance Evaluation 0 0 0 1
The two participants differed in that PU used more metacognitive strategies in types and frequency. However, they both used more metacognitive strategies when listening to the lecture on the unfamiliar topic, employing more Problem Identification, Selective Attention, and Comprehension Monitoring. They employed fewer metacognitive strategies when listening to the lecture on the familiar topic, mainly including Selective Attention and Double-check Monitoring.
The interview results indicated that the number of years of learning English, teachers’ instruction, and listening activities also influenced their use of metacognitive strategies in listening. They both had learned English for more than 11 years and reported plenty of related learning experience. In their English listening class, their teachers might ask them to practice listening for the key words and provided them with various listening passages. After class, PG believed that watching English DVDs for several times did improve his English listening and sometimes listening to English
radio programs before listening comprehension tests might help. For PU, some video tapes from the library, such as BBC and ABC, served as the access to different accented English; by watching them several times, his English listening was improved gradually. And he developed the habit of listening to ICRT and FTV English news, checking his comprehension against his knowledge about the current news.
3.3.2 Revision of the Coding Scheme
Some new metacognitive strategies were identified in the pilot study because Vandergrift’s coding scheme could not identify the metacognitive listening strategies used by the participants. The first new category was Helper Identification. After listening, the participants could clearly and concretely state what helped them understand or the key constituents in understanding the lecture. For example, PG reported, “I think ‘military’ is the key. If the listener does not know the word, s/he may not comprehend the lecture well. And prior knowledge is also important. If you know nothing about Napoleon, you may have difficulty understanding the lecture.”
Motivation Adjustment was another new category. During the listening process, the two participants demonstrated their motivation or showed their interest toward the listening lecture or the topic. For instance, PG reported, “I’m interested in the statement ‘The Underground Railroad is not a real railroad.’ Why? I’m curious about it.”
Some metacognitive listening strategies were reported in the interview. Attention Holding was a strategy mentioned in the interview by the two participants. They suggested that attention played a critical role in the listening process and it would be necessary for listeners to refocus on the aural input when they were lost in the stream of sounds. As PU said, “You have to refocus on the spoken text and try to find out some familiar words or something like that; otherwise, I think most people will just
give up listening.”
Pre-listening Relaxation was the other metacognitive listening strategy reported in the interview. As PU suggested, “feeling nervous will prevent me from listening comprehension because I cannot concentrate. One of my classmates gets nervous so easily that she can perform well in listening when practicing but fail when taking listening tests.” And PG further reported that it would be better to remind him to relax before listening.
In addition to the four new categories of metacognitive strategies, another strategy may be better described in terms of Goh’s (2002) taxonomy, Real-time Assessment of Input. The strategy of Real-time Assessment was quite common in the pilot study, meaning that listeners assessed the importance of the input they had listened to and then decided what to do next. For example, PG reported, “I think it’s not important so I just let it go, even though I did not catch it.”
After re-examining the strategies identified in the pilot study and previous literature, two strategies were incorporated. Anticipation was combined with Advanced Organization since Vandergrift (1997: 396) claimed that “on-line”
prediction was a common planning strategy when the participants began their task
“cold.” And Comprehension Confirmation was put into the category Comprehension Monitoring since the listener interacted with their background knowledge, not with the researcher, who could not give them any response concerning with their questions.
Table 3-5 shows the revised coding scheme reorganized by the researcher on the basis of Vandergrift’s and Goh’s taxonomies and the result of the pilot study.
Table 3-5: Revised Coding Scheme for the Present Study Metacognitive
Strategy
Sub-strategy Definition Example
Advanced Organization
Clarifying the objectives of an anticipated listening task and/or proposing strategies for handling it
I try to think of
questions the teacher is going to ask.
Planning
Pre-listening Relaxation
Preparing emotionally for a listening task
I encourage myself to relax.
Directed Attention
Deciding in advance to attend in general to the listening task and to ignore irrelevant distracters
I pick up the words that are familiar to me.
Selective Attention
Deciding to attend to specific aspects of language input or situational details that assist in understanding and/or task completion
I notice intonation features.
Attention
Attention Holding
Refocusing attention and avoiding
distraction
I ask myself to concentrate on the listening when I am distracted.
Manipulation Understanding the conditions that help one successfully accomplish listening tasks and arranging for the presence of these conditions
I try to get in the frame of mind to understand English.
Self-
Management
Motivation Adjustment
Demonstrating and adjusting their
motivation or interest toward the listening lecture or topic
The Underground Railroad is not a real railroad? Why? I’m curious about it.
Comprehension Monitoring
Checking, verifying, or correcting one’s understanding at the local level
I just try to put everything together, understanding one thing leads to
understanding another.
Auditory Monitoring
Using one’s “ear” for the language (how something sounds) to make decisions
I use the sound of words to relate to other words I know.
Monitoring
Double-check Monitoring
Checking, verifying, or correcting one’s understanding across the task or during the second time through the oral text
I might catch it at the end and then I’d go back.
Performance Evaluation
Judging one’s overall execution of the task
How close was I? (at the end of a think-aloud report).
Strategy Evaluation
Judging one’s strategy use
I think looking for the keys words is quite helpful.
Evaluation
Real-time Assessment of Input
Determining the value of specific parts of the input
Although I did not catch it, I think it is not important. Then I just let it go.
Problem Identification
Explicitly identifying the central point needing resolution in a task or identifying an aspect of the task that hinders its successful completion
I heard “he was bla bla bla….” I did not catch it. I guess he is trying to talk about his job.
Identification
Helper Identification
Explicitly identifying what helped
understanding
To me, prior knowledge is important. I know Napoleon so it’s easy for me to comprehend the lecture.
The first category is Planning, which includes Advanced Organization and Pre-listening Relaxation. The second category, Attention, consists of: (1) Directed Attention helps listeners pick up the main ideas or the overall meaning of the utterance when they cannot comprehend the whole; (2) Selective Attention prepares listeners to focus on key words, information structures, repetition, and intonation features in the utterance so that they can understand better; (3) Attention Holding facilitates listeners refocus when they lose their train of thought. The third category includes Manipulation and Motivation Adjustment. Three sub-strategies (Comprehension Monitoring, Auditory Monitoring, and Double-check Monitoring) are under the category of Monitoring. The fifth category includes Vandergrift’s Performance Evaluation and Strategy Evaluation and Goh’s Real-time Assessment of Input. The last category consists of Vandergrift’s Problem Identification and Helper Identification.
3.3.3 Selecting Participants
For the purpose of this study, 12 participants were recruited from National Taiwan Normal University. They were told the nature and purpose of the present study and asked to sign on the consent forms to show their willingness to join the experiment.
3.3.4 Conducting Immediate Retrospective Verbal Reports
Data was collected through the participants’ immediate retrospective verbal reports about their listening. Following Vandergrift’s (1997) modal, the data collection session included three stages: warm-up, transition, and verbal reports.
The warm-up stage, consisting of greetings and humor, aimed at reducing the participants’ uneasiness and establishing a good working relationship. Then the
detailed instruction of the procedure was offered to help the participants understand the experiment.
At the transition stage, the participants practiced thinking aloud with the material provided in the training session (about 5 minutes). The lecture for the training session consisted of three paragraphs. Therefore, there were three stages in the training session. At Stage 1, the participants were required to practice how to verbalize their mental process while listening. At Stages 2 and 3, the participants might practice more if they were not familiar with the verbal reports or not satisfied with their own performance. After they felt comfortable with their reports, they moved on to the verbal report session. In order to focus on the metal process of listening without inference from the psychomotor processing of writing, the participants were not allowed to take notes while listening.
In completing their verbal reports, the tape was stopped at the pre-chunked point and the participants needed to verbalize what they were thinking. When they were unsure of what to say or how to proceed, probe questions would be given. At this moment, a second tape recorder would record the whole process. In order to reduce the activation of schemata before listening and to assess all thought processes leading to comprehension, the participants were not told anything related to the text they were going to listen to.
Verbal reports were transcribed verbatim. All protocols were coded independently on the basis of the revised coding scheme (see Table 3-5) by three raters12. They met regularly to conduct reliability checks and discussed the coding difficulties to resolve questions and discrepancies. The quantitative analysis was performed to examine the categories and frequency of metacognitive strategy use with
12 Two of the three raters were graduate students who received the training of coding at NTNU. And the researcher served as the third rater when a disagreement appeared. They discussed the definitions and the verbal reports to reach the consensus.
different variables like topic familiarity. Therefore, the data was keyed in and processed by Excel for its frequency.
With regard to the immediate retrospective verbal reports, the coding was based on the participants’ verbal reports during the pre-chunked pause. Every single idea the participants tried to comprehend in the chunked utterance was counted and categorized. When the coding was completed, a metacognitive listening strategy profile was made for each participant. Each coded report of a metacognitive strategy was tabulated. The profiles were grouped and analyzed later according to the groups and familiarity with topics of the lectures.
3.3.5 Conducting Semi-structured Interviews
A semi-structured interview took place a week after the participants complete the immediate retrospective verbal reports and after the coding of their metacognitive strategies was finished. The data from the coded metacognitive listening strategies provided the researcher with some stimulus in conducting the semi-structured interview with the participants. In the semi-structured interview, the participants were asked to express their attitudes towards their metacognitive listening strategy use, their beliefs about language learning and listening, and their previous learning experience and history. In order to avoid the influence of time pressure on their performance, there were no time limitations for the listening tasks and the interview.
With the help of the interview guide (see Appendix C), which served as a basic checklist to ensure the inclusion of all relevant topics (Patton, 1990), the participants’
attitudes towards their metacognitive listening strategies, beliefs about listening, and previous learning experience and history was elicited. The interview was recorded, transcribed, and analyzed later.
A descriptive analysis was employed to examine the participants’ beliefs, metacognitive awareness, and motivation in listening. A qualitative analysis was performed to illustrate what contributed to the participants’ use of metacognitive strategies and to explain why they employed certain strategies in listening. It was also used to clarify the descriptive findings from the immediate retrospective verbal reports. The overall procedure of the present study is shown in the following figure:
Figure 3-2: The Procedure of the Overall Research Choosing a coding scheme
Selecting listening lectures
Designing interview guides
Conducting a pilot study
1. Conducting immediate retrospective verbal reports 2. Analyzing the protocol with the revised coding scheme
1. Conducting a semi-structured interview 2. Categorizing the interview responses
Selecting participants Revising the coding scheme
3.4 Summary of Chapter Three
In this chapter, the research design was reported. The information of the participants, the instruments (i.e., listening lectures, immediate retrospective verbal reports, and semi-structured interviews), and procedure were also introduced. The findings will be reported and discussed in Chapter Four.
