國
立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
CHAPTER FIVE
CONCLUSION
Based on the results and discussion of the present study, this chapter presents a conclusion to wrap up the entire study. First of all, a summary of the major findings is presented to answer the two research questions. Secondly, implications to language teachers in vocational high schools and suggestions to the testing administration of the TVEE are given. Lastly, the limitations of the present study are discussed.
Summary of the Major Findings
The results of the present study can be summarized in connection with the two research questions proposed in the study. The first research question is about the agreement between the dialogue section of the TVEE and the goals highlighted in the VHSCG. The analysis results show that although not evenly distributed, the test items generally correspond to the goals of speaking ability in the VHSCG. The present study finds that the major focus of the dialogue section is to foster the communicative ability on the conversation in daily life settings, however, the conversations in the classroom setting are rarely found; and the dialogue section lacks of items examining learners’ descriptive speaking ability on sequence of events. When it comes to the correspondence of the topics, all the test items can be categorized within the topics suggested in the VHSCG, however, most of the items target at the topic of daily life, leaving many of the VHSCG topics not incorporated into the test.
The second research question goes further to the construct validity of the dialogue section to investigate whether the indirect speaking test is able to delve into the nature of speaking ability. After analyzing each test item, four findings are obtained from the analysis of Speaking Constructs Checklist. First of all, the dialogue section of the TVEE
67
‧ 國
立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
only focuses on informational and interactional skills and does not test the interaction managing skills. Secondly, among the sixty test items, there are 60% of test items located in the most basic complexity level in informational and interactional skills (25% and 35%
respectively). Third, the most highlighted skill in the dialogue section of the TVEE is the basic interactional skill of eliciting information, which consists of 26% of items. Lastly, within the 61 speaking constructs listed in the informational and interactional skills, twenty language functions (32%) are missing from dialogue section of the TVEE from 2010 to 2015. In terms of the testing principles, the application of interactivity and authenticity were found in the test items of the dialogue section, but not the directness since it is a pen and paper test. This can post a threat to the construct validity of the test items.
Implications
Based on the findings of the present study, the researcher proposed implications for the testing administration in the vocational high school education system in Taiwan.
First of all, various speaking skills should be included. Second, the test should cover a greater variety of topics. Third, it should be a goal for the TVEE to develop direct speaking tests.
Including More Speaking Skills
According to the results presented in the previous chapter, the speaking skills involved in the TVEE seem to be repetitive and are those fundamental ones. From the results, it is obvious that test items of the indirect spoken language tests in the TVEE do not represent the overall speaking ability as well as the requirements listed in the
curriculum guideline. Therefore, the present study proposes that the Testing Center of Technological and Vocational Education should implement various communicative
68
‧ 國
立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
speaking skills in the dialogue section to examine test takers’ ability of using various language functions in different speaking contexts.
To involve speaking skills of higher complexity, one of the possible solutions is to extend current short dialogue and use longer conversational texts. As many current test items only record parts of conversation, mostly the opening section of a conversation, the targeted speaking behavior are thus restricted to the basic level and more complicated conversational abilities are left out. There are benefits to develop longer texts in the dialogue section. First, through the processing of longer conversations, sophisticated language behaviors can be elicited. Secondly, developing test items with more turns allows test administrators to examine multiple speaking skills in a single conversational context. Third, the transition and the progress of an authentic communicative context scaffolds language learners into a real-time context, which pushes learners to produce longer conversation with interlocutors they encounter in real life.
Expanding the Variety of Topics
The present study recommends that the dialogue section involve more topics listed in the curriculum guideline other than the topic of daily life like culture and customs, modern technology and arts. Some may argue various topics have already been included in other item types in the TVEE, therefore it may not be necessary to encompass every topic in the dialogue section. However, to the researcher’s best knowledge, no empirical study has been done to explore the topic distribution on the English tests of the TVEE.
The involvement of diversified topics in the dialogue section may encourage teachers and learners to pay more attention to a wider scope when it comes to the material selection during instructions on oral communication. Also, different topics offer plentiful conversational contexts for learners to refer to when language learners are actually involved in certain communicative tasks.
69
‧ 國
立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
Performing Actual Speaking Skills
Practices on oral communicative ability have long been ignored in language program in vocational high schools as the TVEE does not require candidates to perform actual speaking skills (Liao, 2002; Liu, 2009). According to the previous research discussed in the chapter two, indirect tests are no longer encouraged in oral ability examinations for the following reasons: (1) an indirect speaking test does not involve the fundamental phonological elements of oral communication, and (2) it does not require candidates to perform actual speaking skills, which may pose a threat to the validity of the examination. If the TVEE manage to examine speaking skills with direct tests or semi-direct tests, it would without doubt have a positive washback effect on language education in vocational high schools.
Limitations and Suggestions
Although the present study has come up with findings of the quality of the TVEE, the study has some limitations. The first limitation lies in the lack of the feedbacks from the test takers. The primary purpose of the present research aims at the investigation on the constructs of oral communication, the contents of the curriculum guideline, and the test items of the examination. When designing the study, the researcher did not take test takers’ feedback into consideration. The present study thus is not comprehensive enough in presenting the impact of the TVEE. Future studies may elicit the feedback and
perception from both language teachers and students as test users through questionnaire or interview to provide more information to the Testing Center of Technological and Vocational Education.
The second limitation concerns the interpretation of the correlation between students’ actual speaking performance and their performance on the dialogue section of the TVEE. As the dialogue section of the TVEE is an indirect speaking test, test takers
70
‧ 國
立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
may rely on their linguistic knowledge to choose the correct answer without the need of performing the targeted speaking skills. The present study did not collect test takers actual oral performances and compare them with the scores they gained in the TVEE. Without this comparison, the study cannot find out the correlation between learners’ performances in authentic communicative contexts and those of dialogue sections. Therefore, it is recommended that the future research collect the above two data and make a
comprehensive study on the correlation between the two factors to understand the overall validity of the dialogue section of the TVEE.
71
‧ 國
立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
References
Bachman, L. (1990). Constructing measures and measuring constructs. In B. Harley, J.
Cummins, M. Swain and P. Allen (eds.), The development of second language proficiency. Cambridge: Cambridge University Press.
Bachman, L. & Palmer, A. (1996). Language testing in practice. Oxford: University Press.
Berry, V. (1994).The assessment of spoken language under varying interactional conditions. In. Bird, Norman (ed.), Language and learning. Hong Kong: Hong Kong Education Department Institution of Language in Education.
Brown, H. D., & Abeywickrama, P. (2010). Language assessment: Principles and classroom practices. White Plains, NY: Pearson Education.
Brown, H. D., & Sahni, S. (1994). Vistas: An interactive course in English, student test package. Englewood Cliffs, NJ: Prentice Hall Regents.
Brown, H. D. (2001). Teaching by principles: An interactive approach to language pedagogy. White Plains, NY: Longman.
Brown, H. D. (2004). Language assessment: Principles and classroom practices. White Plains. NY: Pearson Education.
Brown, J. D. (1995). The elements of language curriculum: A systematic approach to program design. Boston: Heinle & Heinle.
Brown, J. D. (2005). Testing in language programs: A comprehensive guide to English language assessment. NY: McGraw-Hill College.
Burns, A., & Joyce, H. (1997). Focus on speaking. Sydney: National Center for English Language Teaching and Research.
Bygate, M. (2009). Teaching and testing speaking. In Long, M. H. & Doughty, C. J.
(Eds.), The handbook of language teaching (pp. 412--440). Chichester, U.K:
Wiley-Blackwell.
Canale, M., & Swain, M. (1980). Theoretical bases of communicative approaches to second language teaching and testing. Applied Linguistics, 1(1), 1-47.
Carter, R.A., & McCarthy, M.J. (1995). Grammar and the spoken language. Applied Linguistics, 16, 141–158.
Chang, V. W. (2006). English language education in Taiwan: A comprehensive survey.
Education Resources and Research. 69, 129-144.
72
‧
Chen, M. L. (2007). Vocational high school students' English learning motivation and their English learning behaviors. Unpublished master’s thesis, Tunghai University, Taichung, Taiwan.
Chou. S. K. (2005). 技職校院英語教學之困境與突破 [A dilemma and breakthrough of English teaching in vocational education]. Journal of Education Research, 138, 86-94.
Clark, J. L. D. (1975). Theoretical and technical considerations in oral proficiency testing.
In Jones, R. L. & Spolsky, B. (Eds.), Testing language proficiency, 10-24.
Arlingtong, VA: Center for Applied Linguistics.
Cohen, A. D. (1996). Developing the ability to perform speech acts. Studies in Second Language Acquisition, 18, 253-267.
Cumming, J., & Maxwell, G. (1999). Contextualising authentic assessment. Assessment in Education, 6, 177–94.
Dick, W., Carey, J.O., & L. Carey. (2000). The systematic design of instruction. Upper Saddle River, NJ: Pearson Educational.
Douglas, D. (1997). Language for specific purposes testing. In Clapham, C. & Carson, D., (Eds.), Encyclopedia of language in education. Volume 7: Language testing and assessment. Dordrecht: Kluwer Academic, 111–20.
ETS. (2007). The official guide to the new TOEFL iBT. NY: McGraw-Hill.
Guilford, J. P. (1954). Psychometric method. NY: McGraw Hill.
Han, W. H. (2009). A study of vocational high school students’ English learning
motivation, learning style, learning strategy and English learning achievement.
Unpublished master’s thesis, National Chung Cheng University, Chiayi, Taiwan.
Heaton, J. B. (1998). Writing English language test. London: Longman.
Hu, H. E. (2011). An investigation of Taiwanese vocational high school students' use of metacognitive reading strategies. Unpublished master’s thesis, National Taiwan Normal University, Taipei, Taiwan.
Huang, T. L. (1990) 高級職業學校英文教學研究 [A study on the English language teaching in vocational high schools]. New orientations in English language teaching. 285-204.
Hughes, A. (2003). Testing for language teachers. Cambridge: Cambridge University Press.
Hughes, R. (2011). Teaching and researching speaking. Harlow: Longman.
73
‧ 國
立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
Hung, C. H. & Lo, P. C. (1995).高職英文教師對英語文教育之觀點研究 [Research on the perspectives of English teachers of TEFL in the technological and vocational education system in Taiwan] Proceedings of conference on English teaching in technological and vocational colleges. 121-145.
Joughin, G. (1998). Dimensions of oral assessment. Assessment & Evaluation in Higher Education, 23, 367-378.
Kane, M. T. (2002). Current concerns in validity theory. Journal of Educational Measurement, 38(4), 319-342.
Leu, M. M. (2004) The Use and Evaluation of Teaching Materials for Vocational High Schools in Taiwan. Unpublished master’s thesis, National Taiwan Normal University, Taipei, Taiwan.
Lewkowicz, J. A. (2000). Authenticity in language testing: some outstanding questions.
Language Testing, 17 (1), 43–64
Liao, M. C. (2002). EEFTC English Tests: Renovations and viewpoints of vocational high school students and English teachers. Unpublished master’s thesis, National Taiwan Normal University, Taipei, Taiwan.
Liao, W. Y. (2009). A study of the motivation, attitudes, self-efficacy, and achievements in English learning of vocational high school students in metropolitan Taipei.
Unpublished master’s thesis, National Dong Hwa University, Hualien, Taiwan.
Lin, M. S. (1996). 中華民國高中高職英文教學現況比較研究 [A comparative study on the current status quo of the English language teaching in high schools and vocational high schools in the Republic of China] Proceedings of the twelfth conference on English teaching and learning in the Republic of China. 59-79.
Taipei: Crane Publishing Co.
Liu, C. N. (2009). Evaluating the reading comprehension questions of the SAET and the DRET. Unpublished master’s thesis, Taipei Municipal University of Education, Taipei, Taiwan.
Liu, F. C. (2009). A study on the development of the English tests of the Technological and Vocational Education Joint College Entrance Examination (TVEE) and its interaction with the Guidelines of the Vocational High School English Program (VHSEP). Unpublished master’s thesis, National Chengchi University, Taipei, Taiwan.
74
‧
use of portfolio assessment in the drama teaching of the vocational high school English teaching]. Journal of Research in Education Science. 47(2), 159-174.Lo, G. Y. H. (2011). ESP versus EGP: A case study of an ESP program for vocational high school students of Tourism. Taiwan International ESP Journal, Vol. 3: 2, 71-100
Luoma, S. (2004). Assessing speaking. Cambridge: Cambridge University Press.
Lynch, A. J. (1982). ‘Authenticity’ in language teaching: some implications for the design of listening materials. British Journal of Language Teaching, 20, 9–16.
McHugh, M. L. (2012). Interrater reliability: the kappa statistic. Biochemia Medica, 22(3), 276-282.
Messick, S. (1989a). Meaning and values in test validation: the science and ethics of assessment. Educational Researcher, 5-11.
Messick, S. (1989b). Validity. In R. L. Linn (Ed.). Educational measurement. New York:
American Council on Education, Macmillan, 13- 103.
Messick, S. (1994). The interplay of evidence and consequences in the validation of performance assessments. Educational Researcher, 23(2), 13-23.
Messick, S. (1996). Validity and washback in language testing. ETS Research Report Series, 1996(1), i-18.
MOE. (2014). 高中職應屆畢業生升學就業概況 [An overview of the career of high school and vocational high school graduates], 教育統計簡訊 [Educational Statistical Digest], 10, 1-2
Nicholls, A & Nicholls, S. (1972). Developing a curriculum: a practical guide. London: Allen and Unwin.
Plough, I. C., & Bogart, P. S. (2008). Perceptions of examiner behavior modulate power relations in oral performance testing. Language Assessment Quarterly, 5(3), 195-217.
Prodromou, L. (1995). The backwash effect: from testing to teaching. ELT Journal, 49(1), 13-25.
Qian, D. D. (2009). Comparing direct and semi-direct modes for speaking assessment:
Affective effects on test takers. Language Assessment Quarterly, 6(2), 113-125.
Resnick, L. B., & Resnick, D. P. (1991). Assessing the thinking curriculum: New tools for educational reform. In B. R. Gifford & M. C. O'Connor (Eds.), Changing
75
‧ 國
立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
assessments: Alternative views of aptitude, achievement and instruction, 37-75. Boston: Kluwer.
Shea, D. P. (1994). Perspective and production: Structuring conversational participation across cultural borders. Pragmatics, 4(3), 357-389.
Shih, Y. H., & Lin, M. S. (1996). 我國高職課程與教材評估 [Assessment of vocational high school English curriculum and material]. Proceedings of the fifth
International Conference on English Language Teaching and Learning, 73-88, Taipei: Crane Publishing Co.
Tarone, E., & Liu, G. (1995). Situational context, variation, and second language
acquisition theory. In G. Cook & B. Seidlhofer (Eds.), Principle and practice in applied linguistics: Studies in honour of H G. Widdowson, 107-124. Oxford:
Oxford University Press.
Thornbury, S. (2005). How to teach speaking. Harlow, England: Longman
Tsai, W. C. (2006). A study of students' motivation and strategies in learning automotive industrial English in the automobile department of vocational high school.
Unpublished master’s thesis, National Changhua University of Education, Changhua, Taiwan.
Tyler, R. W. (2010). Basic principles of curriculum and instruction. Chicago: University of Chicago Press.
Underhill, N. (1990) Testing Spoken Language: A Handbook of Oral Testing Techniques.
Cambridge: Cambridge University Press.
Viera, A. J. & Garrett, J. M. (2005). Understanding interobserver agreement: the kappa statistic. Family Medicine, 37(5), 360-363.
Wiggins, G. (1993). Assessment: Authenticity, context, and validity. Phi Delta Kappan, 75(3), 200-208.
Wood, R. (1993). Assessment and testing. Cambridge: Cambridge University Press.
Wu, C. C. (2003). A study of EFL learning strategies used by vocational high school students in Taiwan. Unpublished master’s thesis, National Yunlin University of Science and Technology, Yunlin, Taiwan.
Wu, H. W. (2003). Reading narrative and expository texts: An investigation into the reading strategies used by vocational high school students in Taiwan.
Unpublished master’s thesis, Providence University, Taichung, Taiwan.
76
‧ 國
立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
Wu, J. Y. (2013). The relationship between English learning motivations and learning anxiety of EFL vocational school students: The case study in southern Taiwan.
Unpublished master’s thesis, National Yunlin University of Science and Technology, Yunlin, Taiwan.
You, Y. L., & Chen, P. (2014). The Vocational high school English education in Taiwan.
Secondary Education. 65, 3, 88-107.
Young, R. F. (2000). Interactional competence: Challenges for validity. Paper presented at a joint symposium ‘Interdisciplinary Interfaces with Language Testing’ held at the annual meeting of the American Association for Applied Linguistics and the Language Testing Research Colloquium, 11 March 2000, Vancouver, British Columbia, Canada.
77
‧
Vocational High School Curriculum Guideline 英文Ⅰ－Ⅵ (EnglishⅠ-Ⅵ)
‧
‧
‧
‧
‧
‧
The Dialogue Section of the TVEE in 2010
對話題：第16至25題，請依對話內容，選出一個最適合的答案，使其成為有意 義的對話。
( C ) 16. Cashier: Your total is NT $ 699.
Customer: _______________
Cashier: Yes, Ma’am. Then that’s NT $ 599.
Customer: Do you take credit card?
Cashier: Sure.
(A) I don’t have cash.
(B) Do you need a bag?
(C) I have these coupons.
(D) May I use the shopping cart?
( A )17. Bill: Jean, I’m sorry. I can’t come over today. I have a sore throat.
Jean: Oh, no! Your voice sounds funny. When did you get sick?
Bill: _______________
Jean: I’m sorry to hear that.
(A) Just this morning.
(B) I guess I have a fever.
(C) I’ll go to the doctor later.
(D) I don’t think it’s funny.
( B )18. Boy: Mom, can we ride on the roller coaster?
Mother: Of course, we can.
Boy: Can we go on the Ferris wheel, too?
Mother: _______________
Boy: What? Mom, that’s for babies!
(A) You can go alone if you really want to.
(B) Sure. But let’s try the merry-go-round first.
(C) I am afraid that we don’t have time for that.
(D) Sorry. It is under construction.
( A ) 19. Manager: _______________
Kelly: Pretty much, sir, but I am wondering if there is a dress code.
Manager: Yes, the company wants all the employees to look their best and represent the company well.
(A) Is everything clear so far?
(B) The dress looks great on you.
(C) Do you have any questions?
(D)Welcome to the Sales Department.
( C ) 20. Roger: I am off to the post office.
Rich: Can you mail this airmail for me?
Roger: Sure. _______________
Rich: Yes, you can simply throw it into the mailbox.
(A) You can fax it to me.
(B) Why don’t you mail it yourself?
(C) Have you stamped it?
84
‧
Jimmy: I can’t. I have an English test tomorrow.
Mother: _______________
Jimmy: I haven’t finished reviewing yet. I don’t want to take any chances.
(A) Don’t worry. You’ll be fine.
(B) You should study earlier.
(C) How about a cup of coffee?
(D) Is that the only test tomorrow?
( D ) 22. Teacher: Peter, I think we need to talk.
Peter: Yes, Ma’am.
Teacher: _______________
Peter: I forgot to bring it to school.
(A) What’s wrong with you?
(B) Are you feeling alright?
(C) You didn’t come to class yesterday.
(D) You didn’t turn in your assignment today.
( C ) 23. Jim: Are we going to have a vacation this summer?
Molly: Where do you like to go?
Jim: _______________
Molly: Sounds great!
(A) I have no idea. What do you think?
(B) It seems that weather will be good.
(C) Let’s go somewhere near the beach.
(D) It depends on how much money we have.
( B ) 24. Helen: Do you want me to bring you something for lunch?
Nancy: _______________
Helen: No problem, whatever you ask for.
(A) No bother. I am not hungry at all.
(B) A hamburger, small coffee, and an apple pie.
(C) Where are you going to eat?
(D) Are you going alone or with friends?
( B ) 25. Milly: I have a reservation for two at 6:30.
Host: _______________
Milly: It’s Milly Chang.
Milly: It’s Milly Chang.