Chapter 3 Research Design
The research questions of the present study are as follows:
1. What kind of notes can contribute to a better comprehension?
2. Among all the criteria of note contents, which ones are the best predictors of EFL learners’ listening performance?
3. What is the relationship between note contents and global and local questions?
4. Can the subjects benefit from note reviewing?
5. What kind of notes can contribute to a longer retention?
In this chapter, section 3.1 describes the background information of the participants. The materials and methods of the present study together with the procedures are introduced in sections 3.2 and 3.3. Finally, the man points of this chapter are summarized in section 3.4.
3.1 Participants
In this study, the participants included a college professor of the English Department of National Taiwan Normal University and two classes of her students enrolled in the class, Intermediate Aural-Oral Training. The instructor was in charge of conducting and monitoring the whole experimental procedures during the regular meeting hours.
The course, Intermediate Aural-Oral Training, was set up as a successive course after Basic Aural-Oral Training, which mainly exposed students to daily English conversations. As a required course for all English majors, it was designed as a regular two-hour class per week and the required textbook was Culture Watch, whose
source was ABC news related to cultural issues. In class, the instructor asked the subjects to take notes while watching the news and checked their listening comprehension. In addition to the above requirements, the instructor gave the subjects assignments of working on different listening projects at the interval of 2-3 weeks.
The two classes of the English majors formed the pool of forty-one subjects, who were randomly assigned into the reviewing and non-reviewing groups.
Twenty-one students were labeled as the non-reviewing group and the remaining twenty as the reviewing group. As shown in Table 3-1, the non-reviewing group consisted of four males and seventeen females, whose average age was 20.43. Their average score on the Intermediate Aural-Oral Training course in the first semester was 85.95. The reviewing group consisted of twenty females whose average age was 20.69. Their average score on the Intermediate Aural-Oral Training course in the first semester was 86.70. These two groups did not differ greatly in their age and grade they got in the first semester.
Table 3-1: Information about the Subjects
Group Non-reviewing (NRV) Reviewing (RV)
Number of Students 21 20
Age 20.43 20.69
Male Female Male Female
Gender
4 17 0 20
Grade 85.95 86.70
3.2 Materials and Methods
In this section, the materials and methods in the present study are introduced with details. As to the materials, the lecture source, comprehension tests, and all forms of sheets are explained.
3.2.1 Lecture Source
During the experiment, the major listening source for lectures was extracted from the book Learn to Listen, Listen to Learn, which was designed to prepare advanced English learners for the demand of academic lectures and note-taking. Learn to Listen, Listen to Learn was published by Longman in 1991 and it provided the teachers with
its tapes and the teacher’s manual which contained the complete scripts and outlines of each lecture.
Among all the lectures in the book, the lecture, Drink Your Green Tea, was selected to be the material to test the subjects’ comprehension. The rationale for such a choice was that this topic mainly dealt with one of the Chinese customs – tea, with which the subjects were familiar. The whole lecture lasted 14 minutes and 40 seconds at the speed of 128 words per minute and contained 1,884 words with the three required components in a lecture – introduction, body, and conclusion (see Table 3-2).
In order to make the lecture sound natural, the speaker delivered the speech with false starts, repetition, paraphrasing, and pauses.
Table 3-2 Information on the Lecture Lecture Title Drink Your Green Tea
Category Medical
Length 14 minutes 40 seconds Delivery Speed 128 words per minute Total Words 1,884 words
Idea Units 255 information units
Summary In different parts of the world, people have a different preference for drinks. In Asia, however, green tea is a common and highly esteemed drink. Green tea, which is the least fermented, differs from black tea and oolong tea mainly because of its processing. In contrast, black tea requires more elaborate processing such as fermentation and oxidation. As to world tea production, black tea outnumbered green tea.
A lot of studies have been conducted to examine the effects of green tea. It has been found that green tea has positive effects upon preventing or lowering the risk for diseases such as stomach cancer, lung cancer, and esophageal cancer. With view to its beneficial effects, more research on green tea is conducted.
The content of the chosen lecture mainly addressed information about green tea.
In the introduction, green tea was introduced as a preference in Asian countries along with other preferred beverages in other parts of the world. In the body, three issues about green tea were further discussed. First of all, the speaker explained the differences between green tea and other kinds of tea. Then, the research results about the benefits of drinking green tea were reported. The last point the speaker presented
was the ongoing research on green tea. In the conclusion, he encouraged people to drink green tea rather than coffee or black tea.
3.2.2 Listening Comprehension Test
Fifteen multiple-choice questions of two types – five global questions and ten local questions (see Appendix B) were designed to examine the subjects’ listening comprehension. Global questions were addressed like “What is the main idea of the passage?” This served as an index of testing the subjects’ general comprehension on the whole passage. To answer the global questions correctly, the subjects were required to link each piece of information to work out a general picture. However, local questions were designed to test the subjects’ understanding about detailed information such as how to make green tea. For example, “How long does it take to fully process green tea?” Getting a correct response to the local questions would require the subjects to be able to comprehend merely one piece of information. For each question, four possible options were given but only one of the options was correct. Each correct response was given 1 point with a total of 15.
3.2.3 Note-taking Sheet
During the whole experiment, in addition to the comprehension question sheet mentioned previously, the subjects were given a note-taking sheet (see Appendix C) when they listened to the lecture. Before they listened to the lecture, the instructor distributed the note-taking sheet along with instructions. The instructions mainly informed the subjects of taking as many notes as possible while listening to the lecture and notes in any forms were allowed. The note-taking sheet was not collected until they finished the comprehension test since the sheet was intended to serve as a reference for them to answer the comprehension questions.
3.2.4 Recall Sheet
The recall task was conducted one week after the listening comprehension test without informing the subjects of the task in advance. Similar to the note-taking sheet, a recall sheet (see Appendix D) was designed with no specific guidance but it read,
“Please recall what you heard in the previous week as much as possible.” In order to encourage the subjects to recall what they heard, they were told that there was no limitation on length. This sheet was given to the subjects one week after listening to the lecture. However, the reviewing group and the non-reviewing group did not go over the same procedure. The reviewing group as its name suggested was given the chance to review their notes whereas the non-reviewing group was not. The whole task lasted 10 minutes and the sheets were collected back for data analysis.
3.2.5 Interview
In order to gain an in-depth understanding about the subjects note-taking, a qualitative method (i.e., an interview) was adopted. To decide who the interviewees were, the comprehension question sheets were scored and the information units in the recall task were counted first. After that, four subjects, NRV9, RV3, RV7, and RV11, were chosen for an in-depth interview to gain a more comprehensive picture about their listening performance and note quality. NRV9 was classified as high-score-high-number (a student who scored high on the listening comprehension test and wrote down many information units), RV3 as high-score-low-number (a student who scored high on the listening comprehension test and wrote down a few information units), RV11 as low-score-low-number (a student who scored low on the listening comprehension test and wrote down a few information units), and RV 7 as low-score-high-number (a student who scored low on the listening comprehension test and wrote down many information units). The following interview questions were
asked.
1. When you take notes, which is your primary concern? The more words, the better? The more ideas? The more answerable information? The more efficient?
The more complete? Or the more major information?
2. Which one of the criteria do you think can best predict your performance?
3. When you answer questions about main ideas, is note-taking necessary? If yes, then which criterion is the most important?
4. When you answer questions about details, is note-taking necessary? If yes, then which criterion is the most important?
5. Do you think note-taking affects your recall?
6. Do you think reviewing is beneficial in recall?
3.3 Procedures
This section primarily reports the pilot study, the revision of Dunkel’s model, the formal testing procedures, the scoring system, and the statistic tools used in the present study for data analysis.
3.3.1 Pilot Study
To examine the feasibility of the present study, a pilot study was conducted to test the feasibility of Dunkel’s (1988) model which contains five criteria – (1) total of the words, (2) information units, (3) test-answerability, (4) efficiency, and (5) completeness. It helps pinpoint the deficiency and weaknesses of the draft research design and further helps revise and modify the assessment measure in scoring the note sheet and recall sheet.
The participants were one instructor of the English Department of National Taiwan Normal University and two classes of the instructor’s students. The instructor
monitored and directed all the procedures during the experiment. All of her students were English majors in their second year who enrolled in Intermediate Aural-Oral Training. During their regular two-hour meetings, the data were collected. After the data collection, four students who scored low in a standardized listening comprehension test were chosen and considered as low achievers, and another four students who scored 90 in the test were selected and considered as high achievers.
In the pilot study, the material was the lecture titled “Drink Your Green Tea”
Before listening to the lecture, the instructor distributed the note-taking sheet and asked the subjects to take notes in any forms. After listening, the note-taking sheets were collected and a listening comprehension test was administered. The test was designed as a combination of three global questions and six local questions. One week later, on the condition that the subjects were not informed of the recall task in advance, the instructor asked the subjects to write a summary of 120-150 words on the recall sheet without their notes.
After collecting the data, only the note-taking sheets, comprehension tests, and recall sheets of the eight subjects mentioned previously were analyzed. For scoring these sheets, Dunkel’s (1988) model for deciding on the quality and quantity of notes was adopted. The model listed five criteria – total of the words, information units, efficiency, completeness, and test-answerability.
The results showed some general tendencies in relation with the criteria. It was found that the low achievers tended to jot down fewer words, and fewer symbols and numbers than the high achievers. They all wrote down less than 200 words whereas the high achievers had more than 220 words. The number of words indicated how much the students comprehended a lecture and the degree of comprehension ascended with the increasing number of noted words. On the contrary, the fewer words served as a sign that little comprehension was reached. The relationship between the total of
the words and global questions did not appear to be obvious since global questions were designed to test the general understanding. However, the correct responses to local questions showed a strong relationship with the total of the words on the note-taking sheet. The low achievers tended to write down fewer notes and answer fewer local questions correctly.
The second criterion, the number of information units, appeared to be positively correlated with the subjects’ correct responses. Most high achievers incorporated a greater number of information units in their notes than the low achievers. The number of information units was correlated with subjects’ comprehension.
The third criterion, test-answerability, was highly correlated with the subjects’
correct responses. All the high achievers noted down more than 5 pieces of information that later contributed to correct responses. However, the low achievers contained fewer than 3 pieces of information. This criterion showed a strong relationship with the number of correct responses to local questions. The number of the test-answerability was approximately the same as the number of the correct responses to the local questions. Thus, this criterion was proved to be another indicator of the subjects’ comprehension.
A look at the criterion of completeness showed that the greater completeness of the information units could lead to a better comprehension. Most high achievers covered more than three quarters of all the information units. Low achievers incorporated less than three quarters and some of them even wrote down less than half of the information units. The results showed that the greater completeness in the subjects’ notes yielded a better comprehension indeed.
The last criterion, efficiency failed to account for any relationship with the subjects’ listening comprehension. All the subjects scored 20 points out of 100. Thus, this was the only criterion that could not demonstrate the subjects’ degree of listening
comprehension. In general, all the criteria but efficiency showed a positive relationship with the subjects’ listening comprehension.
3.3.2 Revision of Dunkel’s Model
In the pilot study, all the information listed on the subjects’ note sheet was considered the same level without any hierarchy order. However, every piece of information in fact received different degrees of importance. Therefore, in order to see how different levels of information might influence the subjects’ comprehension, two major categories of information were classified: major and minor. The classification followed the outline of the lecture drawn by the publisher. All the issues covered in the lecture were preceded by general statements which were considered major information. The information that followed the general statements was then considered as minor information. So, the model adopted to evaluate the subjects’ note in the present study was Dunkel’s model with one added criterion – major information, as can be seen in Appendix A.
Table 3-3: A Comparison between of Dunkel’s Model and the Revised Model Dunkel’s Model Revised Model for the Present Study 1. Total-number-of-words Score
2. Information-units Count 3. Test-answerability 4. Completeness Score 5. Efficiency Ratio
1. Total-number-of-words Score 2. Information-units Count 3. Test-answerability 4. Completeness Score 5. Efficiency Ratio
6. Major/Minor Information
3.3.3 Formal Testing
The whole experiment was conducted in fall 2006. The data collection lasted for two weeks with one week for the listening task and the other week for the recall task.
During the regular classes, the lecture was played without informing the subjects in advance. Before the lecture was played, the note-taking sheet was distributed to the subjects for note-taking. After listening to the lectures, the subjects proceeded to take the listening comprehension test with the reference to their notes. Immediately after the test, they exchanged their answer sheets with each other to grade the test. One week later, all the subjects were not informed of the recall task in advance. The reviewing group was given back their note-taking sheets for their reference while completing the recall task but the non-reviewing group carried out the recall task without their notes. After two weeks, all the data were collected and further analyzed.
3.3.4 Scoring & Statistical Analysis
This section describes the methods in scoring the comprehension test and counting the information units in the subjects’ notes and their recall task. The tool for statistical analysis is also introduced.
In scoring the comprehension test, each question accounted for one point with a total of fifteen points. When the subjects gave a correct response, one point was given.
However, when their answers were incorrect, no point was given. The analysis of the subjects’ notes followed the adapted model composed of six criteria: (1) total of the words, (2) information units, (3) test-answerability, (4) efficiency, (5) completeness, and (6) major information. As for the counting of information units, the way to define an information unit was based on Aiken, Thomas, & Shennum’s (1975) definition that information units are regarded as sentences or clauses composed of a single subject-verb or subject-verb-object formation which presents a piece of information
not mentioned previously. The adoption of this definition decided the number of the information units in the recall task. As to the other criteria, Dunkel’s definitions were followed. After collecting all the data, a co-rater was recruited to decode the information units both in the note-taking sheets and recall sheets, answerability, and major information through discussion. Finally the statistical analysis was measured through the SPSS statistics package.
3.4 Summary of Chapter 3
The participants in the study were a professor of the English Department of National Taiwan Normal University and her forty-one English majors. These students were randomly divided into two groups – reviewing and non-reviewing. The reviewing group consisted of 20 subjects and the remaining 21 subjects were labeled as the non-reviewing group. After listening to a lecture about green tea, they completed a listening comprehension test consisting of 5 global questions and 10 local questions. One week later, the reviewing group was given the chance to refer to the notes and wrote down as much as they remembered. The non-reviewing group, deprived of the chance of reviewing notes, completed a recall task. The initial analysis was done through the SPSS package and followed up by an interview with four subjects.
