國
立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
CHAPTER FOUR
RESULTS AND DISCUSSION
In this chapter, the results of the analysis are presented and discussed in response to the two research questions. The first section shows the inter-rater reliability of the coding results. The second section discusses the results of the Curriculum Checklist and the correspondence between the Vocational High School Curriculum Guideline and the English tests of the Technological and Vocational Educational Examination in order to answer the first research question. The third section aims to present the result of the Speaking Constructs Checklist. The final section discusses the application of the three principles. The purpose of the last two sections is to answer the second research question on the construct validity of the English tests of the TVEE.
Inter-rater Reliability
According to McHugh (2012), inter-rater reliability indicates the extent to which raters assign similar score to the same variable, and the importance of inter-rater
reliability lies in the fact that it represents the extent to which the variable measured correctly. The present study invited two coders to analyze the targeted test items, and thus Cohen’s Kappa coefficient was employed to measure the inter-rater agreement.
The magnitude of Cohen’s Kappa coefficient of the present study is .68.
According to Viera and Garrett (2005), the Kappa value of .61 to .80 constitutes substantial agreement. This suggests the coding consistency between the two coders of the present study is moderately substantial. However, as explained in data analysis
framework, the two coders discussed discrepant items together after individual analysis in order to reach a concordance on every single item.
45
‧
Results of the Curriculum Checklist
The outcome and the result of the Curriculum Checklist are discussed in this section. The checklist consists of two parts: the first part shows the correspondence between the task-oriented benchmark and the tasks in the dialogue section of the TVEE.
The second part shows the distributions of the topics given in the curriculum guideline and the topics used in the dialogue section of the TVEE.
The First Part of the Curriculum Checklist
The first part of the Curriculum Checklist finds the dialogue section of the TVEE put strong emphasis on daily life conversation rather than the dialogue taking place in the classroom setting. The distribution of the numbers of matched items between the English test of the TVEE and benchmark for speaking ability in the VHSCG is extremely
imbalanced. The result of the first part of the Curriculum Checklist is presented in Table 4.1.
Table 4.1 The Result of the First Part of the Curriculum Checklist Item Numbers Types & Benchmark for Speaking
Numbers
Teacher-guided task description
Being able to use classroom English 4
Being able to ask and reply questions based on the
reading texts 0
Being able to participate in classroom speaking practice 0 Being able to read aloud texts, short essays and stories 0 Daily
life-based task description
Being able to conduct simple daily life conversation (e.g. greetings, small talks, shopping and telephone talk, etc.)
56 Being able to describe things in everyday lives 0
From this table we can see the dialogue section of the English test of the TVEE primarily focused on the language use in the daily life. 56 out of 60 test items aim to examine whether candidates are capable of conducting simple daily life conversation, and only 4 out of 60 was categorized into the teacher-guided task description of “being able to use
46
‧ 國
立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
classroom English.” As benchmark for speaking ability in the curriculum guideline were divided into two sorts of tasks, the previous four are guided tasks, while the latter two target the daily life tasks operated in daily life, including everyday conversational tasks and the ability to describe daily stuffs. In sum, the dialogue section put emphasis on the daily-life conversation rather on the teacher-guided classroom conversation.
Among the four tasks absent from the test, three of them can be measured through the paper-pencil examination, and only the task of reading-aloud cannot be tested without performing the task directly. For instance, the ability to ask and reply the question of reading materials can be tested by asking for time, location or explanation. The
participation in the in-class speaking practice can be tested by constructing a conversation between teacher and student, and see if the test taker is able to select one best answer to the question. Lastly, the ability of describing things in daily life can be examined by designing a conversation at the lost and found counter to explore whether test takers are able to describe a certain object.
The Second Part of the Curriculum Checklist
The purpose of the second part of the Curriculum Checklist is to examine whether the topics of the test items match those listed in the VHSCG. Table 4.2 shows the result of the second part of the checklist.
47
‧ 國
立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
Table 4.2 The Result of the Second Part of the Curriculum Checklist6 Frequencies
Topics Numbers and Percentages
Interpersonal relationships 3 (5%)
Leisure activities and entertainment 12 (20%)
Daily life 30 (50%)
Modern technology 2 (3%)
Culture and customs 0 (0%)
Literatures and arts 0 (0%)
Language 0 (0%)
Industrial, commercial and agricultural knowledge 3 (5%)
Environmental education 3 (5%)
Employment 2 (3%)
Travel, transportation and traffic 5 (8%)
Global view 0 (0%)
Total 60 (100%)
The most frequently-involved topic in the dialogue section of the TVEE is daily life. In total, 30 items (50%) were classified in this category. The second most popular topic is leisure activities and entertainment. This topic took up 20%, that is, 12 items. Then the numbers dropped drastically for the third topic down to less than 10%. It is quite obvious that the TVEE has not adopted topics widely enough for the content.
To investigate more thoroughly in this most frequently-used topic in the dialogue section, the present study applies the content description of the topic of daily life in the VHSCG to discuss the distribution of these test items categorized in the topic of daily life.
There are eight content descriptions demonstrated in the VHSCG in the topic of daily life as indicated in Table 4.3.
6 All the topics were included in the six semesters, except the last two topics: (1) travel, transportation and traffic; and (2) global view which were listed only in the third academic year.
48
‧
Table 4.3 The Numbers of the Test Items in the Content Description of Daily Life The Numbers of
Matched Items
Descriptions of Daily Life Topic
Year
Ten test items were related to the topic of daily contact, which has the highest coverage (33%) among the test items of the daily life topic. The communicative purpose of this kind of topic is to solve certain problems happening in speakers’ daily life. One example of daily contact is presented below:
Mary: Hello. Steve?
David: I’m sorry, but Steve is out right now. _______
Mary: Mary, his friend from work.
David: May I take a message?
(A) Who is calling?
(B) Thanks for calling.
(C) Why don’t you call his office?
(D) He will be back in ten minutes.
(TVEE, 2010) In this telephone conversation, the caller and the receiver have to deal with the common situation in the telephone talk: the person the caller wants to talk to is not available.
Therefore, to give the correct answer, the test takers have to identify the only possible choice to this type of conversation for the receiver is to ask the caller’s identity so that the
49
‧
best response is the caller reporting her own name and her relationship with the particular person as it presented.
Besides the topic of daily contact, eating was ranked the second highest category (30%). Among the 9 test items, the one in 2013 is a typical conversation related to the topic of eating:
Jane: Thank you for stopping by. Can I get you anything?
Lisa: _____
Jane: Coming right up.
(A) You have nothing I want from you.
(B) A cup of coffee would be nice.
(C) No, I don’t drink in the daytime.
(D) I don’t know what I have in mind.
(TVEE, 2013) In the conversation, the first speaker showed appreciation for the visiting and offered something to drink by asking a question. If candidates realize the conventional meaning of this question is an offer, they would choose answer B.
Among the eight sub-topics of the daily life category, three were less found, including clothing (3%), housing (7%) and legal education (7%). Although emergency precaution was listed as a suggested topic in the curriculum guideline, the dialogue section of the TVEE did not incorporate any conversation concerning this concept.
Referring back to Table 4.2 about the overall description of major topic
distribution, the reader can find that the second mostly invested topic is ‘leisure activities and entertainment.’ There were 12 test items (20%) belonging to this topic. The example of this topic is presented below:
Jack: What are you going to do for this weekend?
Tom: I’m going to the park for a party.
Jack: A party in the park? ______
Tom: There will be a musical concert featuring Mozart. I love his music.
(A) Is the park going to be big?
‧ 國
立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
This conversation was about the weekend plan and the explanation about a concert held in a park. The interlocutor asked for further clarification by repeating the noun phrase with a question mark, and the other speaker answered his request for clarification. With these two contextual clues, the test takers were able to choose the best response, which is answer C.
Although the topic about ‘travel, transportation and traffic’ was listed as the topic to be covered in the last two semesters in the VHSCG, five test items (8%) were
categorized under this topic, which out-numbered another topics that are emphasized throughout the three-year curriculum, such as interpersonal relationships (5%); industrial, commercial and agricultural knowledge (5%); environmental education (5%);
employment (3%); and modern technology (3%). And here is the sample question of the topic of travel, transportation and traffic:
Nina: Hi, Kim. How was your trip?
Kim: Great. It was fun to meet different people in different cities.
Nina: ______
Kim: Taipei is my favorite.
(A) Where did you visit?
(B) How many cities have you been to?
(C) Which city do you like most?
(D) How much did you spend?
(TVEE, 2014) With one speaker asking about the other’s feeling about the trip and his/her favorite city during the trip, the test takers have to recognize that answer C is the appropriate
interrogative question based on the response given in the stem.
Due to the fact that the coverage of particular topics is rather high, there are some topics that were completely neglected in the dialogue section. Table 4.2 reveals the four missing topics: (1) culture and customs; (2) literatures and arts; (3) language; (4) global view. These topics were completely not covered throughout the 60 target items since the curriculum guideline has been renewed in 2010. These topics were connected to the
51
‧ 國
立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
learners’ humanity training, and such phenomenon could possibly lead to a negative washback effect in terms of the material selection in vocational high schools.
To sum up, the topics applied in the dialogue section of the TVEE generally corresponded to the topics listed in the VHSCG, and they were mostly close to the life experience of vocational high school students. But with an imbalanced emphasis on the topics of daily life, topics concerning humanity education and global view were missing.
The VHSCG developed a wide spectrum of topics that learners are expected to immerse in, but the dialogue section of the TVEE targeted on only few dimensions. The content comprehensibility of the dialogue section is therefore restricted.
Results of the Speaking Constructs Checklist
This section presents the result of the Speaking Constructs Checklist. Firstly, Table 4.4 shows the distribution and the percentage of the numbers of the matched items within three major skills.
Table 4.4 The Item Distribution of Three Major Speaking Constructs Results
Skills Numbers and Percentages
Informational skills 30 (50%)
Interactional skills 30 (50%)
Skills in managing interactions 0 (0%)
Total 60 (100%)
From the above table, we can see the informational and interactional skills have an equal percentage of 50, while none of the test items was found applying the sub-ability of skills in managing interactions among the target items. As the Speaking Constructs Checklist is divided into three major skills: informational; interactional; and skills in managing interactions, this section discusses the outcome of each major skill respectively.
52
‧ 國
立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
The Result of Informational Skills
Fifty percent items tested informational skills, which express factual meaning and thus play the most crucial role in the test of speaking ability in the TVEE. Thirty test items are distributed in 27 sub-skills listed under this major skill. In terms of the skill complexity, 15 items are categorized in level one, 9 in level two, and 6 in level three. The analysis result is presented in Table 4.5.
53
‧
Table 4.5 The Informational Skills of the Speaking Constructs Checklist Informational Skills
Complexity
Level Language Functions and Behavior Numbers and Percentages
Level 1
Describe sequence of events 0 (0%)
Express need 3 (10%)
Express opinions 2 (6%)
Express preferences 2 (6%)
Express requirements 0 (0%)
Elaborate an idea 0 (0%)
Give instructions 1 (3%)
Present an argument 0 (0%)
Provide personal information 1 (3%)
Provide non-personal information 1 (3%)
Provide required information 5 (16%)
Seek permission 0 (0%)
Total in Level One 15 (50%)
Level 2
Analyze 0 (0%)
Draw conclusions 0 (0%)
Give explanations 5 (16%)
Make comparisons 0 (0%)
Indicate attitude 1 (3%)
Make excuses 1 (3%)
Table 4.5 presents the oral abilities at three levels according to task complexity. In
general, the items were not distributed equally among the three levels. Most items, that is,
54
‧ 國
立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
fifteen items (50%), belong to level one; nine items (30%) to level two; and only six items (20%) to level three. From the above table, it is apparent that the dialogue section of the TVEE only focused on certain language functions and behavior. The result of analysis showed the preference of particular kinds of language skills. For example, in the first level, providing required information was the most frequently-tested construct. The test items categorized in this skill were basic question-and-answer conversations, such as:
Nick: Art Café is really nice.
David: Yes, it is. It’s new.
Nick: When did it open?
David: ______
(A) Two weeks ago.
(B) Three times a week.
(C) Every weekend.
(D) Next week.
(TVEE, 2014) The purpose of this item was to examine whether the candidate is able to provide
suitable answer. Candidates must recognize the meaning of the question is to ask for a specific time in the past (answer A), and then delete the irrelevant responses about the frequency (answers B and C) and time in the future (answer D) to provide the
information the other speaker required.
The equally-emphasized speaking skill lied in the second complexity level, giving explanation. Coded 5 times, it was the most frequently-examined speaking construct (16%) in level two. Similar to the skill of giving required information in the first level, the behavior of giving explanation was usually followed after an interrogative structure.
Thus this kind of language behavior required the test takers to understand the purpose of the interrogative question so as to complete a comprehensive dialogue. For example:
Terry: What kind of a seat do you request when you fly?
Susie: I prefer an aisle seat.
Terry: Why?
Susie: _____
(A) So I can stretch my legs.
(B) Sitting in the back of the plane is exciting.
55
‧ 國
立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
(C) It doesn’t usually take much time.
(D) I like to sleep, and I need something to lean against.
(TVEE, 2011) Readers have to identify the topic is about the seat choice on a flight according to the first and second turns of conversation so that they would understand the question in the third turn is a request for explanation. To give an acceptable explanation, readers have to read all the available answers to cross out the unrelated explanations to the benefit of an aisle seat, which has nothing to do with the location on the plane (answer B), time
consumption (answer C) or the need of something to lean against (answer D). However, it requires sufficient background knowledge about the aisle seat, which allows passengers to have more space than a window seat.
Only six test items (20%) were characterized in the third level of complexity.
Among the tasks at this level, making suggestion was the most-targeted skill. This skill was coded twice, and both test items appeared in the dialogue section in 2010. The example of the test items is presented below:
Jim: Are we going to have a vacation this summer?
Molly: Where do you like to go?
Jim: _____
Molly: Sounds great!
(A) I have no idea. What do you think?
(B) It seems that weather will be good.
(C) Let’s go somewhere near the beach.
(D) It depends on how much money we have.
(TVEE, 2010) By reading the first turn, learners can understand the topic in this dialogue is about the leisure activity. Then, test takers have to recognize the female speaker’s agreeing attitude by the second interrogative statement. All the four answers are logical and coherent to the second turn, but only the suggestive statement in answer C is relevant to the response in the last turn.
56
‧ 國
立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
With a strong preference on three to four language behaviors, the TVEE leaves more than 40% of the informational speaking skills unused in the dialogue section.
Twelve out of 27 language subskills were not examined in the TVEE since 2010. Table 4.6 displays these completely neglected language functions in informational skills.
Table 4.6 The Informational Skills not Tested in the TVEE
Complexity Level Language Functions and Behavior not Tested
Level 1
Describe sequence of events Express requirements Elaborate an idea Present an argument Seek permission
Level 2
Analyze
Draw conclusions Paraphrase Summarize Level 3
Complain Justify opinions Speculate
These unused language functions are actually important elements in daily life
conversation even for vocational high school language learners. Some of the functions were the basic communicative skills of expressing speakers’ reasoning process, such as analyzing and justifying opinion. Some skills were helpful when one has to speak for one’s own rights, e.g. expressing requirements and complaining. And still others could be used to enrich the content of conversation, e.g. elaborating ideas, paraphrasing and summarizing. However, they have not been tested in the TVEE since the renewal of the VHSCG in 2010.
The Result of Interactional Skills
The other thirty test items were categorized in the interactional skills, and the analysis result in this section showed a rather huge gap in the complexity level.
Twenty-57
‧
three test items (76%) were considered to be in the basic level of interactional skill, five items (17%) in the second level, and only two items (7%) were put in level three. Table 4.7 displays the analysis result of the interactional skills.
Table 4.7 The Interactional Skills of Speaking Constructs Checklist Interactional Skills
Complexity
Level Language Functions and Behavior Numbers of Matched Items
Level 1 Elicit information 16 (53%)
Elicit opinions 4 (13%)
Elicit clarification 2 (7%)
Establish common ground 0 (0%)
Express purpose 1 (3%)
Total in Level One 23(76%)
Level 2 Correct themselves or others 0 (0%)
Express agreement 2 (7%)
Express disagreement 1 (3%)
Indicate uncertainty 0 (0%)
Indicate understanding or failure to understand 1 (3%) Justify or support statements or opinions of other
speakers
0 (0%)
Modify statements or comments 1 (3%)
Repair breakdowns in interaction 0 (0%)
Recognize other speakers’ purpose 0 (0%)
Total in Level Two 5 (17%)
Level 3 Attempt to persuade others 1 (3%)
Check that they understand or have been understood correctly
1 (3%) Questions assertions made by other speakers 0 (0%) Respond to requests for clarification 0 (0%)
Total in Level Three 2 (7%)
Total Number 30 (100%)
As shown in Table 4.7, there are total 30 items for interactional skills. Among them, 23 items (76%) are at level one, 5 items (17%) at level two and only 2 items (7%) at level
As shown in Table 4.7, there are total 30 items for interactional skills. Among them, 23 items (76%) are at level one, 5 items (17%) at level two and only 2 items (7%) at level