國
立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
CHAPTER FOUR RESULTS AND DISCUSSION
In this chapter, the results are discussed in response to the three research questions of the present study. The first section answers the first research question, presenting the correlation between the scores of the multiple-choice writing test of the English Professional Subject (EPS) and those of the picture writing task. In the second section, correlations between the four item types of the EPS indirect writing test and the picture writing task are illustrated in reply to the second research question. The third section describes the participants’ perceptions of the two writing tasks in answer to the third research question.
Correlation between EPS Multiple-choice Writing Test and Picture Writing Task
The section reports the correlation between the 119 participants’ performances in the EPS multiple-choice writing test and in the picture writing task. The objective writing test, containing 30 items, was directly taken from the 2011 edition of the EPS writing test of the Technological and Vocational Education (TVE) Joint College Entrance Exam. The direct writing task was drawn from the 2006 edition of the picture writing test of Scholastic Aptitude Test (SAT) that required the participants to write a 100-word passage on the basis of the three-frame picture sequence.
The multiple-choice writing test with a total score of 75 was machined-scored, while the picture writing task was scored holistically by the two raters on the 20-point scale. Since the two writing tasks were scored within different frameworks, raw scores were used in the analysis of data.
Table 4.1 demonstrates the descriptive statistics of the raw scores for the EPS indirect writing test and for the picture writing task.
‧
Table 4.1 Descriptive Statistics of EPS Multiple-choice Writing Test and Picture Writing Task
The researcher then used the raw scores (see Appendix J for details of the raw data) to calculate the correlation between the two writing tasks. A correlation of .58 (p < .01) was thus attained. Based on Wu and Tu (2009), the significant correlation of .58 suggests that the two writing tasks exhibit positive correlation at the mid level.
The coefficient of correlation is of the same general magnitude as that reported in previous studies (Breland & Gaynor, 1979; Breland & Jones, 1982; Godshalk et al., 1966; Hogan & Mishler, 1980). The correlational result might be attributed to the fact that the EPS multiple-choice writing test is intended to measure writing ability (P. Lin, 2005; National Teachers’ Association [NTA], 2011; You et al., 2002). As Godshalk, Swineford, and Coffman (1966) state, “When objective questions specifically designed to measure writing skills…, they prove to be highly valid” (p. 40). In particular, the indirect writing test focuses on examining organizational ability of writing. According to Benton and Kiewra (1986), writing measures are more
significantly correlated with performances in direct writing when they aim to assess organizational ability.
The result further indicates that although the writing test takes the form of multiple choices, it is of some value to the writing assessment of the students from Department of Applied Foreign Languages (DAFL). Now that the indirect writing test is moderately correlated with direct writing, it lends support to the assumption that the students’ writing ability could, in some way, be assessed by the objective writing test.
Hence, indirect writing tests might be an alternative way in the large-scale writing test
‧ 國
立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
if direct tests are not feasible due to practical constraints.
Correlations between Four Item Types of EPS Multiple-choice Writing Test and Picture Writing Task
The correlations between the scores of the participants in the four item types of the EPS multiple-choice writing test and in the picture writing task are discussed in this section. The four item types included topic sentence selection (TSS), sentence insertion (SI), sentence deletion (SD), and sentence rearrangement (SR). The number of items for each of the four item types was 10, 7, 7, and 6, respectively. Since not all the total scores of the four item types were the same, raw scores were used in the data analysis.
The descriptive statistics of the raw scores for the four item types are presented in Table 4.2.
Table 4.2 Descriptive Statistics of Four Item Types of EPS Multiple-choice Writing Test
Item Type N Lowest
Score
Highest Score
M SD Total
Score
TSS 119 2.5 25 12.40 4.89 25
SI 119 0 17.5 10.84 4.00 17.5
SD 119 2.5 17.5 10.57 3.83 17.5
SR 119 0 15 7.88 3.23 15
The raw scores (see Appendix J for details of the raw data) were then drawn on to calculate the correlations between the subtests in the EPS multiple-choice writing test and the picture writing task.
Table 4.3 then indicates the correlations between the 119 participants’ raw scores on the four items types and on direct writing.
‧ 國
立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
Table 4.3 Correlations between Subtests of EPS Multiple-choice Writing Test and Picture Writing Task (N = 119)
Subtest r Rank
SI .52** 1
SD .51** 2
SR .38** 3
TSS .27** 4
** p < .01
SI correlated moderately (r = .52) with the picture writing holistic score, as did SD (r
= .51). In contrast, SR did not correlate nearly as well (r = .38), nor did TSS (r = .27).
The difference in the degree of the correlations was significant at the .01 level.
In Wu and Tu’s (2009) opinion, the correlation coefficients derived from scores on TSS and on SR with essay total score belong to correlations at the low level. The result of TSS may be attributed to the fact that TSS is intended more to assess
examinees’ reading ability. The ability to pick the main idea for the passage might not entail the ability to actually produce a topic sentence. As for SR correlation result, it is in accordance with those from previous research (Benton & Kiewra, 1986; Godshalk et al., 1966). Godshalk et al. (1966) proved that SR, intended to assess examinees’
ability of rearranging grammatical patterns, lacked concurrent validity as a measure of composition skills.
The results of SI and SD, mid correlation in Wu and Tu’s (2009) classification, are in agreement with those of Godshalk et al. (1966) and of Breland and Jones (1982).
The substantial correlations would be expected, given that the purpose of the two subtests is to assess the participants’ concepts of text structure and coherence (Breland
& Jones, 1982; Godshalk et al., 1966). As Yu (2007) notes, understanding discourse organization may contribute to the acquisition of writing ability.
Judging from the correlational results, among the four subtests, SI and SD could be relied on as a better indication of the DAFL students’ actual writing proficiency.
‧ 國
立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
Nevertheless, TSS and SR, due to the low correlations, would be less appropriate to evaluate the students’ writing ability.
Perceptions of EPS Multiple-choice Writing Test and Picture Writing Task In this section, the results of the questionnaires about the participants’
perceptions of the indirect and direct writing tasks are classified into four subsections:
(a) abilities the participants applied in the two writing tasks; (b) difficulties they faced in the two writing tasks; (c) perceptions of the two types of writing tasks; (d)
perceptions of the EPS multiple-choice writing test of the TVE Joint College Entrance Exam.
Abilities Participants Applied in Indirect and Direct Writing Tasks
The results of the second part of indirect writing questionnaire (IWQ) (Q1-Q4) and those of direct writing questionnaire (DWQ) (Q1) are reported in order to find out the abilities the participants applied while taking the EPS multiple-choice writing test and the picture writing task, respectively. This subsection first deals with the results of the indirect writing test, then with those of the direct writing task.
In an attempt to elicit the abilities applied by the participants in the EPS indirect writing test, nine ability items, based on Grabe and Kaplan’s (1996) taxonomy of language knowledge of writing, were given in each of the four questions of IWQ. The nine ability items included (1) recognizing punctuation; (2) understanding words; (3) recognizing phrases; (4) applying grammar knowledge; (5) applying reading
comprehension ability; (6) recognizing cohesive devices; (7) finding main ideas; (8) establishing coherence in contexts; (9) applying the concept of discourse structure.
Items 1-4 belonged to knowledge at the word and sentence levels and Items 6-9 to knowledge at the above-sentence level. Item 5 was intended to investigate the role reading comprehension played in the multiple-choice writing test.
Table 4.4 indicates the rank order of the abilities the participants (N = 119) used to complete the EPS objective writing test.
‧
Table 4.4 Rank order of Abilities Participants Used to Take EPS Multiple-choice Writing Test (N = 119)
Ability Frequency Percent Rank
TSS
Establishing coherence in contexts 96 80.7% 1
Applying reading comprehension ability 90 75.6% 2
Understanding words 84 70.6% 3
Finding main ideas 83 69.7% 4
Applying the concept of discourse structure 61 51.3% 5
Recognizing phrases 41 34.5% 6
Recognizing cohesive devices 41 34.5% 6
Applying grammar knowledge 18 15.1% 8
Recognizing punctuation 12 10.1% 9
SI
Establishing coherence in contexts 112 94.1% 1
Applying reading comprehension ability 94 79.0% 2
Understanding words 73 61.3% 3
Applying the concept of discourse structure 64 53.8% 4
Finding main ideas 61 51.3% 5
Recognizing cohesive devices 58 48.7% 6
Recognizing phrases 36 30.3% 7
Applying grammar knowledge 28 23.5% 8
Recognizing punctuation 16 13.4% 9
SD
Establishing coherence in contexts 99 83.2% 1
Applying reading comprehension ability 97 81.5% 2
Understanding words 66 55.5% 3
Finding main ideas 58 48.7% 4
Applying the concept of discourse structure 58 48.7% 4
Recognizing phrases 38 31.9% 6
Recognizing cohesive devices 38 31.9% 6
Applying grammar knowledge 18 15.1% 8
Recognizing punctuation 6 5.0% 9
SR
Establishing coherence in contexts 111 93.3% 1
Applying reading comprehension ability 88 73.9% 2
Applying the concept of discourse structure 73 61.3% 3
Recognizing cohesive devices 64 53.8% 4
Understanding words 57 47.9% 5
Finding main ideas 49 41.2% 6
Recognizing phrases 39 32.8% 7
Applying grammar knowledge 33 27.7% 8
Recognizing punctuation 14 11.8% 9
‧ 國
立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
The most frequently utilized ability, over 80.7% in all of the four item types, was establishing coherence in contexts. Applying reading comprehension ability was the second frequently used ability (over 73.9% in all of the four subtests). Two additional abilities—applying the concept of discourse structure and understanding words (over 48.7% and 47.9% in all of the subtests, respectively)—were perceived as being the next most frequent abilities used by the participants. It seems that the students draw on similar abilities in taking the four subtests and that they make use of more knowledge at the above-sentence level than knowledge at the word and sentence levels in the indirect writing test.
In the previous studies, researchers have noted that more sentence-level knowledge is used by examinees in indirect writing tests (Ackerman & Smith, 1988;
Breland & Jones, 1982; Chang, 2003; Cooper, 1984). Nonetheless, the result of the present study indicates that the participants capitalize more on knowledge at the above-sentence level in the EPS indirect writing test. As P. Lin (2005), NTA (2011), You, Chang, Joe, and Chi (2002) find out, the EPS multiple-choice writing test is designed to assess the students’ concepts of discourse structure. Accordingly, when teachers instruct the students how to cope with the writing entrance exam, they tend to require the students to look at the test items from the angle of discourse structure. The students, in this regard, are trained to consciously apply their knowledge at the
discourse organizational level in taking the objective writing test.
Although the indirect writing tests were intended to assess the participants’
writing knowledge, over 73.9% of the students applied reading comprehension ability for each of the four subtests. The result is consistent with the findings of previous research (Cooper, 1984; P. Lin, 2005; Teng, 2002) that in addition to the targeted writing skills, the format of multiple-choices may call for examinees’ reading skills.
As Hughes (2003) points out, the construct validity of the indirect writing test is thus
‧ 國
立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
under threat because it involves abilities other than what it has aimed to assess.
Then, in the picture writing task, Question 1 of DWQ itemized nine abilities, also based on Grabe and Kaplan’s (1996) taxonomy, to elicit the abilities the participants utilized. The nine abilities consisted of: (1) applying punctuation knowledge; (2) applying previously-learned words; (3) applying previously-learned phrases; (4) applying grammar knowledge; (5) using connectives to connect sentences;
(6) applying the concept of topic sentences; (7) using supporting sentences to support main ideas; (8) applying the concept of coherence in contexts; (9) applying the concept of discourse structure. Items 1-4 were classified into knowledge at the word and sentence levels, while items 5-9 into knowledge at the above-sentence level.
The rank order of abilities the participants applied in taking the picture writing task is displayed in Table 4.5.
Table 4.5 Rank order of Abilities Participants Used to Take Picture Writing Task (N = 119)
Ability Frequency Percent Rank
Applying previously-learned words 98 82.4% 1
Applying grammar knowledge 93 78.2% 2
Using connectives to connect sentences 88 73.9% 3
Applying the concept of coherence in contexts 80 67.2% 4
Applying punctuation knowledge 67 56.3% 5
Applying previously-learned phrases 39 32.8% 6
Applying the concept of discourse structure 24 20.2% 7 Applying the concept of topic sentences 23 19.3% 8 Using supporting sentences to support main ideas 23 19.3% 8 In the picture writing task, applying previously-learned words was referred to most frequently, followed by applying grammar knowledge. Both of the two abilities were over 78.2%. Furthermore, using connectives to connect sentences (73.9%) and applying the concept of coherence in contexts (67.2%) were the next two frequently used abilities in the direct writing task. The results reveal that the students draw more
‧ 國
立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
on knowledge at the word and sentence levels in the picture writing task.
The result is in sharp contrast to the findings of the previous studies (Ackerman
& Smith, 1988; Breland & Jones, 1982; Chang, 2003; Cooper, 1984) that knowledge at the discourse organizational level is applied more in direct writing tasks.
Nevertheless, in the present study, the participants utilized more knowledge at the word and sentence levels in the picture writing task. The results may be explained by the following two possible reasons. First, the students do not know how to apply discourse-level knowledge in doing the direct writing task. Because the EPS writing test takes the form of multiple choices, many writing teachers simply slash time on actual writing. They may even follow the traditional teaching approach to English composition—concentrating primarily on grammar analysis and vocabulary teaching (D. Chen, 1998; H. Chen, 2001; Yu, 2007). The participants are thus deprived of the opportunities to learn how to write a unified and coherent passage. As might be expected, the students, less proficient in the target language and inexperienced in composing direct writing, simply stay at the word and sentence levels while doing the picture writing task. Second, the students do not realize that their knowledge at the discourse level also contributes to their composing the direct writing task. Teachers, due to the present testing format, devote most of their class hours to teaching grammar and word usage. The students may thus get a message that putting all grammatically correct sentences together could entail good writing (H. Chen, 2001).
For that reason, the students perceive that it is their writing knowledge about vocabulary and grammar that helps them in the process of composing the picture writing. But actually they may draw on their discourse-level knowledge to complete the writing task unconsciously.
Comparing the abilities utilized by the participants in the two writing tasks, it is found that the students yield different profiles of writing ability in the two writing
‧ 國
立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
tasks. Both of the writing measures demand the students to demonstrate their knowledge of words and concepts of coherence. Nonetheless, taking a close look at the rank orders of the abilities the participants used in the two writing tasks, the researcher notices that the most frequently used abilities in one writing task are placed last in the other. For instance, the participants capitalize on grammar knowledge most to complete the picture writing task, whereas knowledge of grammar is in almost the last place in the multiple-choice writing test. In addition, the knowledge of main ideas is drawn on more in the indirect writing test, while it is used the least in the direct writing task.
In sum, it is evident that the two approaches render unique information about the participants’ writing ability profiles. Consequently, depending solely on one writing approach to examine the students’ writing competence might not fully assess the abilities required in the assessment of writing.
Difficulties Participants Faced in Indirect and Direct Writing Tasks
Questions 5 and 6 of the second part of IWQ and Question 2 of DWQ are about the difficulties undergone by the participants while taking the EPS indirect writing test and the picture writing task, respectively. Thus, this subsection first outlines the difficulties the participants came across in the EPS multiple-choice writing test, followed by the ones they had in the picture writing task.
Question 5 of IWQ requires the participants to determine the difficulty levels of the four item types on a scale of 1 (the most difficult) to 4 (the easiest). In data
analysis, responses were assigned point values from the most difficult (4 points) to the easiest (1 point). Therefore, the higher means an item type obtained, the more difficult it was. As indicated in Table 4.6, the most difficult item type recognized by the
participants (n = 115) was TSS (M = 3.28), followed by SI (M = 2.88), SD (M = 2.15), and SR (M = 1.70).
‧ 國
立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
Table 4.6 Difficulty Levels for Subtests of EPS Multiple-choice Writing Test (n = 115)
Subtest M Rank
TSS 3.28 1
SI 2.88 2
SD 2.15 3
SR 1.70 4
The participants were then required to confirm the difficulties they encountered while answering the most difficult item type in Question 6 of IWQ. Eight statements of the difficulties they might face, in light of Ackerman and Smith’s (1988) indirect writing model, were listed in the question as follows: (1) do not understand the meaning of the words; (2) do not understand the meaning of the phrases; (3) do not understand the meaning of the sentences; (4) do not know which cohesive devices to use; (5) cannot find out main ideas; (6) cannot find out coherence in contexts; (7) cannot understand the meaning of the passage; (8) cannot finish reading the items within time limit. All of the eight items were categorized into the reviewing process of the writing model. Items 1-3 were intended to examine difficulties at the word and sentence levels and items 4-7 at the discourse level. The last item was used to investigate whether the participants had trouble finishing reading the stems within time limit.
The rank order of the difficulties the participants encountered in taking the EPS multiple-choice writing test is shown in Table 4.7.
‧
Table 4.7 Rank Order of Difficulties Participants Encountered to Take EPS Multiple-choice Writing Test
Difficulty Frequency Percent Rank
TSS (n = 66)
Do not understand the meaning of the words 44 66.7% 1
Cannot find out main ideas 43 65.2% 2
Cannot understand the meaning of the passage 38 57.6% 3 Do not understand the meaning of the sentences 37 56.1% 4
Cannot find out coherence in contexts 34 51.5% 5
Cannot finish reading the items within time limit 32 48.5% 6 Do not understand the meaning of the phrases 23 34.8% 7
Cannot understand the meaning of the passage 13 54.2% 4 Do not understand the meaning of the sentences 11 45.8% 5 Cannot finish reading the items within time limit 11 45.8% 5 Do not know which cohesive devices to use 8 33.3% 7 Cannot finish reading the items within time limit 8 50.0% 3 Cannot understand the meaning of the passage 6 37.5% 5
Cannot find out main ideas 4 25.0% 6
Do not understand the meaning of the phrases 2 12.5% 7 Do not know which cohesive devices to use 2 12.5% 7
SR (n = 9)
Cannot find out coherence in contexts 7 77.8% 1
Cannot understand the meaning of the passage 6 66.7% 2 Do not understand the meaning of the words 5 55.6% 3
Cannot find out main ideas 4 44.4% 4
Do not understand the meaning of the sentences 3 33.3% 5 Do not know which cohesive devices to use 2 22.2% 6 Do not understand the meaning of the phrases 1 11.1% 7 Cannot finish reading the items within time limit 1 11.1% 7
‧ 國
立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
For each of the subtests in the indirect writing test, the participants had more difficulties understanding the meaning of the words and finding out coherence in contexts. The two appear to bother the participants the most for they were with quite high frequency (over 55.6% and 51.5% in all of the four subtests, respectively).
The students’ difficulty in understanding the meaning of the words may be explained by the participants’ insufficient vocabulary size (H. Chen, 2001; P. Lin, 2005). Because they are lacking in vocabulary, they could not use guessing word meaning from contexts to facilitate their understanding of the word because the contexts are also unfamiliar to them. That’s why they feel it difficult to understand word meaning in the indirect writing test. As for the difficulty of figuring out coherence in texts, the reason may be that the students have not received enough
The students’ difficulty in understanding the meaning of the words may be explained by the participants’ insufficient vocabulary size (H. Chen, 2001; P. Lin, 2005). Because they are lacking in vocabulary, they could not use guessing word meaning from contexts to facilitate their understanding of the word because the contexts are also unfamiliar to them. That’s why they feel it difficult to understand word meaning in the indirect writing test. As for the difficulty of figuring out coherence in texts, the reason may be that the students have not received enough