Chapter Four Results and Discussion
This chapter presents the analysis of data and discusses the results of the analysis.
The results obtained from the analysis of the data include four parts: first, the participants’ performances on the “Discourse Structure” tests before and after the treatment; second, the participants’ metacognitive awareness of the strategies they used in taking the DS test and their awareness of the difficulties they encountered in taking the DS test before and after the treatment; third, the participants’ responses toward the instruction of text structure, toward the think-aloud modeling, as well as toward think-aloud practice; finally, the protocols of the four participants’
think-alouds. All the results are presented and discussed both quantitatively and qualitatively to probe the effects of the three stages of treatment on these senior high school EFL students from the above aspects.
4.1 Results
4.1.1 Comparison of the pre-test and post-test scores
In this section, a comparison of the participants’ pre-test and post-test scores is made in an attempt to present the effects of the integrated treatment, which consists of the instruction of text structure, think-aloud modeling and think-aloud practice.
Specifically, the performances on the DS test for all the sixty-nine participants and for three groups of different English proficiencies, i.e. High, Mid, and Low are analyzed.
As shown in Table 4.1, the means of the pre-test and post-test scores for all participants are 12.80 and 15.52 respectively, which represent a gain of 2.72 on the DS test. And a t-test analysis indicates a significant difference between the pre-test and post-test scores performed by sixty-nine participants in this study (t=-5.618,
Languages and Literatures of Chung Hsing University.
p<.01).
Significant differences are also observed between the pre-test and post-test scores by the three groups, i.e. High, Mid and Low, respectively. As shown in Table 4.1, the means of the pre-test and post-test scores for the High group are 15.68 and 18.55, which indicate a gain of 2.87 on the DS test. Moreover, a t-test analysis
demonstrates a significant difference between the two scores for this group (t=-4.769, p<.01). Similar results can be seen in Table 4.1 for the other two groups. After a closer look at the mean scores, it is found that the greatest gain goes to the Mid group, and the slightest to the Low.
Table 4.1
Comparison of Means of Participants’ Performance on DS Tests Before and After the Integrated treatment
Pre-test Post-test T-value H Mean=15.68 Mean=18.55 -4.769**
N=22 SD=2.77 SD=1.90
M Mean=12.58 Mean=16.04 -4.283**
N=24 SD=3.27 SD=2.60
L Mean=10.26 Mean=12.52 -2.664**
N=23 SD=3.21 SD=3.55
All Mean=12.80 Mean=15.52 -5.618**
N=69 SD=3.77 SD=3.79
**p<.01
4.1.2 Comparison of participants’ metacognitive awareness in taking the DS test before and after the integrated treatment
This section presents the effects of the treatment, including the instruction of text structure, think-aloud modeling and think-aloud practice, on the participants’
awareness of the strategies they used and the difficulties they encountered in taking
the DS test. In this study, the participants’ metacognitive awareness in taking the DS test before and after the treatment has been investigated and compared by using questionnaires with the same statements to see if there is any change caused by the treatment.
The first part of the metacognitive awareness questionnaire (Statements 1 to 20) is to tap the participants’ awareness of the strategies they used in taking the DS test.
Table 4.2 shows the statistical results of the participants’ responses toward the strategies they used in taking the DS test prior to and after the treatment.
By comparing the results of the two questionaires, we clearly see that a change has arisen. First, a significant difference is found for six items between the
pre-treatment and post-treatment questionnaires. These six items are Statement 1 “In taking the DS test, I will try to pronounce every word” (t=-2.522, p<.01), Statement 15 “I will try to pay attention to Unity and Coherence of the text” (t=-3.126, p<.01), Statement 16 “I will try to look for possible clues—pronouns” (t=-5.758, p<.01), Statement 17 “I will try to look for possible clues—demonstratives” (t=-6.171, p<.01), Statement 18 “I will try to look for possible clues—lexical ties” (t=-4.070, p<.01), and Statement 19 “I will try to look for possible clues—conjunctions” (t=-2.935, p<.01).
Second, a decrease in the mean scores is found for two items, including Statement 5
“I will try to get the overall meaning of the text” (-.11) and Statement 20 “I will try to look for possible clues—paragraph themes” (-.03). Third, as shown in Table 4.2, the two strategies most frequently used among the participants in taking the DS test after the treatment are Statement 9 “I will try to make inference from contextual clues”
(M=4.57) and Statement 10 “I will try to ignore what I don’t understand and keep
reading” (M=4.51), both of which happen to be the same two strategies most
frequently used in taking the DS test before the treatment. In addition, the two
strategies least frequently used in taking the DS test after the treatment are Statement
14 “I will try to judge the text type” (M=2.65), and Statement 1 “I will try to pronounce every word” (M=2.87), both of which happen to be the two least used strategies in taking the DS test before the treatment. However, as mentioned above, a significant difference is found for Statement 1 in comparing the results before and after thetreatment.
Table 4.2
Comparison of Means of Participants’ Awareness of the Strategies They Used in Taking the DS Test Before and After the Integrated treatment
Items Pre-treatment Post-treatment T-value In taking the DS test, I will try to:
1. Pronounce every word
Mean: 2.55 2.87 -2.522**
SD: .79 1.07 2. Understand the meaning of
each word
Mean: 2.94 3.14 -1.242 SD: 1.01 .93
3. Look for keywords and judge their meanings
Mean: 4.30 4.33 -.314 SD: .75 .63
4. Analyze grammatical structure
Mean: 3.46 3.62 -1.352 SD: .90 .81
5. Get the overall meaning of the text
Mean: 4.33 4.22 1.210 SD: .66 .80
6. Understand the details of the content
Mean: 3.36 3.41 -.426 SD: .89 .81
7. Distinguish the main idea from the supporting information of the text
Mean: 3.82 3.90 -.779
SD: .82 .77
Table 4.2 (continued) 8. Get the author’s point
Mean: 3.80 3.93 -1.584 SD: .81 .79
9. Make inference from contextual clues
Mean: 4.45 4.57 -1.526 SD: .63 .53
10. Ignore what I don’t understand and keep reading
Mean: 4.43 4.51 -.962 SD: .63 .53
11. Reread what I don’t understand
Mean: 3.57 3.61 -.382 SD: .99 .88
12. Make guesses from context for what I don’t understand
Mean: 4.36 4.45 -1.097 SD: .59 .61
13. Relate the text to what I already know about the topic
Mean: 3.87 4.06 -1.300 SD: .89 .76
14. Judge the text type
Mean: 2.62 2.65 -.293 SD: .81 .82
15. Pay attention to Unity and Coherence of the text
Mean: 3.72 4.06 -3.126**
SD: .84 .66 16. Look for possible clues—pronouns
Mean: 3.49 4.10 -5.758**
SD: .88 .71 17. Look for possible clues—demonstratives
Mean: 3.39 4.13 -6.171**
SD: .93 .70 18. Look for possible clues—lexical ties
Mean: 3.87 4.28 -4.070**
SD: .91 .62
Table 4.2 (continued)
19. Look for possible clues—conjunctions
Mean: 4.23 4.45 -2.935**
SD: .65 .58 20. Look for possible clues—paragraph themes
Mean: 3.55 3.52 .314 SD: .78 .83
N=69 **p<.01
The second part of the metacognitive awareness questionnaire is to tap the participants’ awareness of the difficulties they encountered in taking the DS test before and after the integrated treatment (Statements 21 to 32). Table 4.3 shows the statistical results obtained for this purpose.
As shown in Table 4.3, some changes have been found from the comparison of the results of the pre-treatment and post-treatment questionnaires. First, a t-test shows a significant difference for three items, including Statement 29 “In taking the DS test, the reason I have difficulties is that I am not familiar with text structure” (t=2.532, p<.01), Statement 31 “I am not equipped with effective strategies for taking DS tests”
(t=4.899, p<.01), and Statement 32 “I can’t make coherence out of the text” (t=5.834, p<.01). The positive t-value with losses of mean scores indicates that these difficulties are significantly less difficult after the treatment. Second, a gain is found for the mean scores on three items, which are Statement 22 “I can’t figure out lexical cohesion”
(+.15), Statement 24 “Sentence structures are complicated” (+.09) and Statement 26
“I can’t figure out the main idea of the text” (+.19). Third, the acknowledged two
greatest difficulties encountered in taking the DS test after the treatment are Statement
25 “There are idiomatic expressions I don’t know” (M=3.80) and Statement 21 “There
are too many words I don’t know” (M=3.78), which represent the second and third
greatest difficulties before the treatment. However, the greatest difficulty before the
treatment, Statement 32, has become only the eighth most difficult after the treatment.
In addition, the two items causing least difficulty after the treatment are Statement 31
“I am not equipped with effective strategies for taking DS tests” (M=2.97) and Statement 23 “Sentences are too long” (M=2.99).
Table 4.3
Comparison of Means of Participants’ Awareness of the Difficulties They Encountered in Taking the DS test Before and After the Integrated treatment
Items Pre-treatment Post-treatment T-value In taking the DS test, the reasons I
have difficulties are because:
21. There are too many words I don’t know
Mean: 3.84 3.78 .563 SD: .80 .76
22. I can’t figure out lexical cohesion
Mean: 3.52 3.67 -1.487 SD: .76 .89
23. Sentences are too long
Mean: 3.20 2.99 2.031 SD: .93 .98
24. Sentence structures are complicated
Mean: 3.17 3.26 -.715 SD: .89 .90 25. There are idiomatic expressions I
don’t know
Mean: 3.91 3.80 1.210 SD: .70 .68
26. I can’t figure out the main idea of the text
Mean: 3.06 3.25 -1.978
SD: .95 1.02
Table 4.3 (continued)
27. I can’t see possible clues
Mean: 3.48 3.43 .312 SD: .88 1.01
28. I am not familiar with text structure
Mean: 3.54 3.20 2.532**
SD: .83 .81 29. I am not familiar with the topic
Mean: 3.57 3.43 1.054 SD: .87 .92
30. I am not equipped with effective strategies for taking the DS test
Mean: 3.58 2.97 4.899**
SD: .86 .89 31. I can’t make coherence out of the text
Mean: 3.94 3.20 5.834**
SD: .84 .98 N=69 **p<.01
4.1.3 Analysis of Four participants’ think-aloud Protocols
In this section, the transcribed think-aloud protocols of four participants with different levels of English proficiency are analyzed for discussion in terms of occurrences and percentages of categories of cohesive ties used as clues for
confirmation or exclusion of alternatives in taking the DS test (see Appendix H for transcribed protocols).
The comments made by the four participants in taking the DS test while reading
aloud and thinking aloud were coded according to the categories of cohesive devices
listed in Appendix B. A commentary that contained more than one cohesive tie were
multiply coded. Tables 4.4 and 4.5 show the frequencies and percentages of the
participants’ using cohesive ties correctly as clues for confirmation and exclusion of
alternatives, respectively, in taking the DS test
5. In the protocols, three categories have been identified, including (1) References, (2) Conjunctions, and (3) Lexical Ties.
Table 4.4 shows that among the three categories of cohesive ties, Lexical Ties has the highest percentage for all the four participants, with the percentages of 61.5%, 63%, 70%, 77.8% and 67% for the Highest, High, Mid, Low and Total respectively.
These figures show that all the four participants depend most heavily on Lexical Ties than on the other two categories of cohesive ties identified in the protocols (i.e.
References and Conjunctions) as clues to confirm answers in taking the DS test, and that the dependence on Lexical Ties as clues for confirmation of alternatives in taking the DS test increases as the English proficiency of the participants decreases. On the other hand, among the three types of cohesive ties, References has the lowest
frequency for all the four participants, with the percentages of 15.4%, 7.4%, 5%, 0%
and 7.7% for the Highest, High, Mid, Low and Total respectively. As with Lexical Ties, the percentages for References indicate a pattern as far as the participants’
English proficiency is concerned, though the direction is the other way round: The higher the English proficiency is, the higher the frequency of using References as clues is.
As to the individual differences in terms of the frequencies of using cohesive ties as clues for confirmation, the Highest and High participants has the frequencies of more than 25, while the Mid and Low participants no more than 20.
5
As mentioned above, all the protocols were coded by the researcher and her sister Ms. Li-wen Chen, who is an instructor at the Department of Foreign Languages and Literatures of Chung Hsing
University. The coding consistency was originally 89%, and with further discussion, all the
inconsistently coded items were resolved and agreed upon.
Table 4.4
Frequencies and Percentages of Four Participants’ Use of Cohesive Ties as Clues Correctly for Confirmation of Answers in Taking the DS Test
Categories of cohesive ties Frequencies (Percentages) Highest High Mid Low Total References 4 (15.4%) 2 (7.4%) 1 (5%) 0 (0%) 7 (7.7%) Conjunctions 6 (23.1%) 8 (29.6%) 5 (25%) 4 (22.2%) 23 (25.3%) Lexical Ties 16 (61.5%) 17 (63%) 14 (70%) 14 (77.8%) 61 (67%) Total 26 (100%) 27 (100%) 20 (100%) 18 (100%) 91 (100%)
Table 4.5 shows the frequencies and percentages of the four participants’ using cohesive ties as clues correctly to exclude alternatives in taking the DS test identified in their think-aloud protocols. Besides the three categories of cohesive ties identified in Table 4.4, a category named Unspecified is added to refer to situations in which the participants apparently were aware of some cohesive ties and yet did not specify which ones that were on their mind.
Like what we have observed in Table 4.4, the category of cohesive ties that gets the highest total frequency of application is Lexical Ties, and the one that has the lowest frequency is References. However, while the above pattern applies to all the four participants in Table 4.4, here we observe only for the Mid and Low participants to have Lexical Ties as the highest frequency, and only one occurrence of References is found for all participants, which was used by the Highest participant.
As to the individual differences in terms of the frequencies of using cohesive ties
as clues for exclusion of alternatives, we find quite surprisingly that while in Table 4.4
it is the High participant who gets the highest frequency, here the title falls on the
Low participant (17). In addition, Table 4.5 indicates that for the Mid participant, the
strategy of using cohesive ties as clues to exclude alternatives was not frequently used,
for the total occurrence is only three, which belong to Lexical Ties exclusively.
Furthermore, while the Mid and Low participants depend most greatly on Lexical Ties to exclude unlikely alternatives, a largest percentage is Unspecified for both the Highest and High participants.
Table 4.5
Frequencies and Percentages of Four Participants’ Use of Cohesive Ties as Clues Correctly to Exclude Alternatives in Taking the DS Test
Categories of cohesive ties Frequencies (Percentages) Highest High Mid Low Total References 1 (7.6%) 0 (0%) 0 (0%) 0 (0%) 1 (2.4%) Conjunctions 4 (30.8%) 1 (12.5%) 0 (0%) 3 (17.6%) 8 (19.5%) Lexical Ties 3 (23.1%) 2 (25%) 3 (100%) 12 (70.6%) 20 (48.8%) Unspecified 5 (38.5%) 5 (62.5%) 0 (0%) 2 (11.8%) 12 (29.3%) Total 13 (100%) 8 (100%) 3 (100%) 17 (100%) 41 (100%)
In addition to the situations in which the cohesive ties were used correctly either to confirm answers or to exclude alternatives, there are situations when the
participants misused cohesive ties as clues. For both the Highest and the High
participants, one occurrence of misjudgment is found; however, both participants soon discovered their errors as they read along and encountered the challenge of “It doesn’t make sense this way,” and were able to correct the errors right away. For the Highest participant, the conjunction “but” was misinterpreted, while the High participant mismatched a reiterated lexical phrase. With regard to the Mid and Low participants, more misuse of cohesive ties as clues is found, and there are times when they were unable to correct their errors. The Mid participant misused cohesive ties three times, all of which belong to Lexical Ties, but only one was corrected. The total errors of using cohesive ties as clues for the Low participant are eight, with four of them
corrected. One of these errors is the misuse of a pronoun, a subcategory of References,
and is among those corrected.
4.1.4 Results of participants’ response questionnaire
This section presents the participants’ responses toward the integrated treatment.
First, the participants’ perception as to the effect of the instruction of text structure on their performance of the DS test and on other aspects of learning is investigated.
Second, the value of think-aloud modeling in the participants’ learning how text structure and cohesive devices work in text and in taking the DS test is explored.
Third, the participants’ responses toward think-aloud practice is probed. Finally, some questions concerning follow-up activities are also investigated.
As the response questionnaire includes two types of questions—open-ended questions and five-point scale statements, two approaches for analysis of the data were employed. For the five open-ended questions, the responses were collected and coded for analysis
6. A response that contained more than one proposition was multiply coded
7. As for the five five-point scale statements, the means and the SDs were
computed for discussion.
First of all, in answer to the question of whether the instruction of text structure was helpful in learning the essence of text—unity and coherence, all the participants (100%) responded positively. Besides, sixty-eight participants (98.6%) were positive about the value of the instruction in helping them take the DS test. Only one stated that the she had learned about the application of text structure and cohesive devices before the instruction, and therefore did not feel its effect. As for more specific benefits from the instruction, Table 4.6 presents the frequencies and percentages in terms of six categories: (A) Awareness of cohesive ties (47.9%), (B) A grasp of main
6
As mentioned above, the codings were done by the researcher and her sister Ms. Li-wen Chen, who is an instructor at the Department of Foreign Languages and Literatures of Chung Hsing University.
The coding consistency was originally 92%; for the inconsistent items, a discussion was made between the two coders before the final conclusion was reached.
7
For example, when asked in what aspects of learning the instruction was beneficial, a participant might give a response that contained both “writing” and “reading;” in such cases, both “writing” and
“reading” were counted in the categorization.
ideas of text (15.6%), (C) Confirmation of answers on DS tests (14.6%), (D) An increase in word-guessing ability (5.2%), (E) Awareness of text structure (5.2%), and (F) Others (11.5%). For Category (A), twenty-one of the forty-six responses
mentioned specific cohesive ties that the participants learned most about: ten pronouns, five conjunctions, four synonyms/antonyms, one demonstrative and one tense. Also, Category (F) “Others” includes responses like what follows:
“I can keep calm because I know better what clues to look for.” (M24)
8.
“I have become more analytic.” (H7)
“My speed in taking the DS test has become faster.” (H8; H11; L55)
“It is more likely that I guess the answer right.” (M23)
Table 4.6
Questionnaire Results of Participants’ Opinions on the Instruction of text structure (N=69)
Category Frequencies Percentages
(A) Awareness of cohesive ties 46 47.9%
(B) A grasp of main ideas of text 15 15.6%
(C) Confirmation of answers on DS tests 14 14.6%
(D) An increase in word-guessing ability 5 5.2%
(E) Awareness of text structure 5 5.2%
(F) Others 11 11.5%
Total 96 100%
Table 4.7 shows the participants’ views on other aspects of learning in which the instruction of text structure were helpful. Thirty-four cases (47.2%) indicated that they benefited from the instruction in reading comprehension, among whom fourteen indicated that they were better able to get hold of the main ideas of text; nine
commented that they read much faster; another nine mentioned that they would not be
8