Major Findings
The purpose of the present study was to develop a reliable and valid
vocabulary test based on the new word frequency list (WFL) from COCA in the form of multiple choice questions. All the items were carefully designed to make sure that only the subjects’ ability is to be gauged.
The present test was administrated to 1,198 participants, which satisfied the threshold of minimum sample size of a 3PL model in Latent Trait Theory. For each item, the author can obtain the values of the difficulty, discrimination, and pseudo-guessing parameters, and thus gain insights into the validity and reliability of the test.
The positive result of overall model fit confirms that the multiple choice format is an ideal form for a vocabulary size test. According to Latent Trait Theory, the overall model fit, good-fit items, elaborate statistics of the parameters, and high value of reliability show that the present study is a valid and reliable indicator of the vocabulary size of the examinees.
From the study, it may be observed that the lower the word frequency, the higher the word difficulty. In addition, the slope of item difficulty from Level One to Three is steeper than that from Level Four to Six, which means that the item difficulty increases more slowly once the examinees have passed the threshold of the first three
54
levels of word frequency bands. This phenomenon can best be summarized as showing that so long as the learners possess the words from Levels One to Three, they are able to transfer the mental lexicon and word schemata that they already have to learn new words. In addition to the relationship between frequency bands and item difficulty, examinees who are low achievers tend to choose the options in the middle, which can be inferred from the pseudo-guessing parameter of the 3PL LTT model.
Examination of the good-fit and misfit items bear out that when designing test items, the influence of telltale morphemes should be taken under consideration. If stems are embedded with hints of corresponding word parts, the distractors have to be
cautiously designed. Take the stem “to describe something,” for example. The key answer “descriptive” is relatively appealing to the examinees because it is a derived
form of describe, and it is hard to design other distractors among the options. Finally, it may be observed that Form B, which has the higher range of margin words as target words, is more difficult than Form A, and that both forms present good fit items based on the 3PL LTT model. Therefore, it may be reasonably assumed that examinees who take the test can learn whether they have passed certain frequency bands by taking this vocabulary test.
In conclusion, the present study gives strong evidence that learners are equipped with the ability to use their word schemata to distinguish possible distractors. In
55
consequence, it is important for test designers to know how the test-takers use their metalinguistic and morphological knowledge as test-taking strategies.
Pedagogical Implications
The information obtained from this study has important implications for vocabulary teaching and the development of test items. First of all, the decisive threshold of word frequency for learners who want to learn new words above Level Four is learners’ mastery of the first three thousand words. As long as learners pass the first three levels, their progress in learning new words from Levels Four to Six will be much simpler, because learners’ pre-existing vocabulary knowledge of first three levels will be of great help to learners when enlarging their vocabularies. This information holds two implications for pedagogy. On the one hand, the instructors should emphasize high-frequency words, for mastering them is of basic importance in enlarging learners’ vocabulary size and language development. On the other
hand, by using collocation, morphology, and word association, instructors can call up learners’ pre-existing word knowledge, and thus assist them in learning by
incorporating new words in their mental lexicon. Baumann et al. (2003) pointed out that learners should have the ability to acquire new words by themselves with the help of effective vocabulary instruction and teaching activities. That is, instructors
56
should teach learners how to utilize their word schemata and learn new words effectively by extensive reading. With the upcoming curriculum change in Taiwan, teachers can design or choose their own teaching materials to open a wide variety of English elective courses, which means that the teachers can use the present study as the tool to gauge students word band and give them timely help.
In test development, the present study provides a useful tool for instructors to measure students’ vocabulary size in a new way by gauging whether the examinees
have passed a certain frequency level. Moreover, it compensates for the flaw in traditional ways of measuring learner’s vocabulary size, which have a certain degree of fuzziness at the frequency band boundaries in determining learners’ vocabulary size. In addition, the current study indicates that students predictably guess the middle
two options. Test designers should be aware of this when designing test items. In addition to learners’ guessing behavior, because of the important role that word
schemata play in test-taking strategy, test developers should know the impact of telltale word parts and utilize this knowledge to develop good test items.
Limitations of the Research
The findings of this research have several limitations that should be addressed in future research. For one thing, the present study focused on the marginal words in
57
each frequency band; that is, the one hundred words close to the cutoffs for each frequency band. However, there are still large gaps between levels, and the question thus arises of the validity of claiming the subjects have passed a certain level if there is a possibility that they might not know words in the gap. For another, although only one misfit item was identified, it was not modified and retested to gain information for test development.
Directions for Future Research
In order to overcome the limitations above, we would like to make the following suggestions for future research. First, if future research also includes target words randomly chosen from the remaining 900 words in each frequency band, this can better support the assumption that subjects have passed certain thresholds of vocabulary size, and thus bridge the gaps between levels. In addition, it would be useful to carry out qualitative research, such as interviews of examinees after the test to help researchers determine how they applied test-taking strategies during the test. In this way, test developers may learn how learners utilize their pre-existing word knowledge when taking the test, and thereby develop good distractors and stems of good quality. Finally, the problem of positional response bias can be reduced by using a format with five-choice options to ease the guessing effect. It would be interesting to know which format is best for an effective vocabulary size test.
58
REFERENCES
Aizawa, K. (2006). Rethinking frequency markers for English-Japanese dictionaries.
In M. Murata, K. Minamide, Y. Tono and S. Ishikawa (eds) English lexicography in Japan (pp. 108-119). Tokyo:
Taishukan-shoten.
Alderson, J. C., Clapham, C., & Wall, D. (1995). Language test construction and evaluation. Cambridge, England: Cambridge University Press.
Anastasi, A., &Urbina, S. (1997). Psychological testing. New York: Macmillan.
Anderson, R. C., &Freebody, P. (1979). Vocabulary Knowledge (Technical Report No.136). Urbana, IL: Center for the Study of Reading. (ERIC ED177480) Anderson, R. C., &Freebody, P. (1981). Vocabulary knowledge. In J. T. Guthrie (ed.),
Comprehension and teaching: Research Reviews (pp. 77-117). Newark. DE:
International Reading Association.
Attali, Y., & Bar‐ Hillel, M. (2003). Guess where: The position of correct answers in
multiple‐ choice test items as a psychometric variable. Journal of
Educational Measurement, 40(2), 109-128.
Bachman, L. F. (1990). Fundamental considerations in language testing. Oxford, England: Oxford University Press.
59
Bachman, L. F. (2004). Statistical analyses for language assessment: Cambridge, England: Cambridge University Press.
Bachman, L. F., & Palmer, A. S. (1996). Language testing in practice: Designing and developing useful language tests. Oxford, England: Oxford University Press.
Baylor, C., Hula, W., Donovan, N. J., Doyle, P. J., Kendall, D., &Yorkston, K. (2011).
An introduction to item response theory and Rasch models for speech-language pathologists. American Journal of Speech-Language Pathology, 20(3), 243-259.
Bauer, L., & Nation, P. (1993). Word families. International Journal of Lexicography, 6(4), 253-279.
Baumann, J. F., Kame’enui, E. J., & Ash, G. E. (2003). Research on vocabulary
instruction: Voltaire redux. In Lapp, D. (Eds.), Handbook of research on teaching the English language arts (pp752-785). Hoboken, NJ : Taylor &
Francis.
Beglar, D. (2010). A Rasch-based validation of the Vocabulary Size Test. Language Testing, 27(1), 101-118.
Beglar, D., & Hunt, A. (1999). Revising and validating the 2000 word level and university word level vocabulary tests. Language Testing, 16(2), 131-162.
60
Bernhardt, E. B., &Kamil, M. L. (1995). Interpreting relationships between L1 and L2 reading: Consolidating the linguistic threshold and the linguistic
interdependence hypotheses. Applied Linguistics, 16(1), 15-34.
Bertram, R., Baayen, R. H., &Schreuder, R. (2000). Effects of family size for complex words. Journal of Memory and Language, 42(3), 390-405.
Blachowicz, C. L., & Fisher, P. (2004). Vocabulary lessons. Educational Leadership, 61(6), 66-69.
Brown, J. D., & Hudson, T. (2002). Criterion-referenced language testing. Cambridge, England: Cambridge University Press.
Carrell, P. L., &Eisterhold, J. C. (1983). Schema theory and ESL reading pedagogy.
TESOL Quarterly, 17(4), 553-573.
Carver, R. P. (1994). Percentage of unknown vocabulary words in text as a function of the relative difficulty of the text: Implications for instruction. Journal of Literacy Research, 26(4), 413-437.
Campion, M. E., & Elley, W. B. (1971). An academic vocabulary list. New Zealand Council for Educational Research.
Chen, L.-J. (2011). An application of Item Response Theory to Developing and Validating a Vocabulary Levels Test. Unpublished master’s thesis,
National Taiwan Normal University, Taipei, R.O.C.
61
Cheng,M.H.(2014).The Application of Latent Trait Theory to Developing and Validating a Vocabulary Size Test. Retrieved October 18, 2017, from http://etds.lib.ntnu.edu.tw/cgibin/gs32/gsweb.cgi?o=dstdcdr&s=id=%22G N0599211103%22.&searchmode=basic
Christenfeld, N. (1995). Choices from identical options. Psychological Science, 6(1), 50-55.
Cizek, G. J. (1994). The effect of altering the position of options in a multiple-choice examination. Educational and psychological measurement, 54(1), 8-20.
Coady, J., &Huckin, T. (1997). Second language vocabulary acquisition: A rationale for pedagogy. New York: Cambridge University Press.
College Entrance Examination Center. (2000). 大考中心高中英文參考詞彙表.
Retrieved from http://www.ceec.edu.tw/Research/paper_doc/ce37/ce37.htm Corson, D. (1995). Using English words. Dordrecht, The Netherlands: Kluwe
Academic Publishers.
Coxhead, A. (2000). A new academic word list. TESOL quarterly, 34(2), 213-238.
Cronbach, L. J., & Meehl, P. E. (1955). Construct validity in psychological tests. Psychological bulletin, 52(4), 281.
Cronbach, L. J. (1946). Response sets and test validity. Educational and Psychological Measurement, 6(4), 475-494.
Cronbach, L. J. (1950). Further evidence on response sets and test design.
62
Educational and Psychological Measurement, 10, 3-31.
Cronbach, L. J. (1971). Test validation. Educational Measurement, 2, 443-507.
Daller, H., Milton, J., &Treffers-Daller, J. (2007). Modelling and assessing vocabulary knowledge. Cambridge, England: Cambridge University
Press.
Davies, A., Brown, A., Elder, C., Hill, K., Lumley, T., & McNamara, T. (1999).
Dictionary of language testing. Cambridge, England: Cambridge
University Press.
Fagley, N. (1987). Positional response bias in multiple-choice tests of learning: Its relation to testwiseness and guessing strategy. Journal of Educational Psychology, 79(1), 95.
Fan, M. (2000). How big is the gap and how to narrow it? An investigation into the active and passive vocabulary knowledge of L2 learners. RELC Journal, 31(2), 105-119.
Gardner, D., & Davies, M. (2013). A new academic vocabulary list. Applied Linguistics, 35(3), 305-327.
Goulden, R., Nation, P., & Read, J. (1990). How large can a receptive vocabulary be?
Applied Linguistics, 11(4), 341-363
63
Graves, M. F., & Watts-Taffe, S. M. (2002). The place of word consciousness in a research-based vocabulary program. What research has to say about reading instruction, 3, 140-165.
Jeng, H.S., Chang,H.S., Cheng, Y.S. &Gu, Y.S.(2002). 6,480-word list for English teaching and textbook writing at senior high school level in Taiwan. College Entrance Examination Center. Retrieved May. 29, 2013, from
http://www.ceec.edu.tw/research/paper_doc/ce37/2.pdf
Kean, J., & Reilly, J. (2014). Item response theory. Handbook for Clinical Research:
Design, Statistics and Implementation.(pp195-198). New York, NY: Demos
Medical Publishing.
Kučera, H., & Francis, W. N. (1967). Computational analysis of present-day
American English. Providence, RI: Brown University Press.
Haladyna, T. M., & Rodriguez, M. C. (2004). Developing and validating multiple-choice test items. New York: Routledge.
Hambleton, R. K., &Swaminathan, H. (1984). Item response theory: Principles and applications. Boston: Kluwer-Nijhoff Publishing.
Hambleton, R. K., Swaminathan, H., & Rogers, H. J. (1991). Fundamentals of item response theory. Newbury Park, California: Sage Publications.
64
Hanna, G. S., &Dettmer, P. A. (2004). Assessment for effective teaching: Using context-adaptive planning. Boston: Pearson A and B.
Heaton, J. B. (1975). Writing English language tests. New York: Longman.
Heaton, J. B. (1990). Classroom testing. New York: Longman.
Hilton, H. (2008). The link between vocabulary knowledge and spoken L2 fluency.
Language Learning Journal, 36(2), 153.
Hirsh, D., & Nation, P. (1992). What vocabulary size is needed to read unsimplified texts for pleasure?. Reading in a Foreign Language, 8, 689-689.
Hsueh-Chao, M. H., & Nation, P. (2000). Unknown Vocabulary Density and Reading Comprehension. Reading in a Foreign Language, 13(1), 403-430.
Hughes, A. (2003). Testing for language teachers. New York: Cambridge University Press.
Hula, W. D., Fergadiotis, G., & Martin, N. (2012). Model choice and sample size in item response theory analysis of aphasia tests. American Journal of Speech-Language Pathology, 21(2), S38-S50.
Laufer, B. (1982). Does the EFL reader need reading strategies more than language Some experimental evidence. . Paper presented at the Annual Meeting of the
American Council on the Teaching of Foreign Languages, New York.
Laufer, B. (1997). The lexical plight in second language reading: Words you don’t
know, words you think you know, and words you can’t guess. In Coady, J.,
65
&Huckin, T. (Eds.), Second language vocabulary acquisition: A rationale for pedagogy (pp.20-34). Cambridge, England: Cambridge University Press.
Laufer, B. (1998). The development of passive and active vocabulary in a second language: Same or different? Applied Linguistics, 19(2), 255-271.
Laufer, B. (2010). Lexical threshold revisited: Lexical text coverage, learners’
vocabulary size and reading comprehension. Reading in a Foreign Language, 22(1), 15.
Laufer, B., Elder, C., Hill, K., &Congdon, P. (2004). Size and strength: do we need both to measure vocabulary knowledge? Language Testing, 21(2), 202-226.
Laufer, B., &Paribakht, T. S. (1998). The relationship between passive and active vocabularies: Effects of languagelearning context. Language Learning, 48(3), 365-391.
Lee, J. W., &Schallert, D. L. (1997). The relative contribution of L2 language proficiency and L1 reading ability to L2 reading performance: A test of the threshold hypothesis in an EFL context. Tesol Quarterly, 31(4), 713-7 Lin, C.H.(2006). A Quantitative Analysis of the Vocabulary in the First Volume of
Taiwanese Senior High School English Textbooks. Unpublished MA thesis, University.
66
Mackey, W. F. (1967). Language teaching analysis. Bloomington, IN: Indiana University Press.
McCarthy, M. (1990). Vocabulary. Oxford, England: Oxford University Press.
McNamara, T. F., &Candlin, C. N. (1996). Measuring second language performance.
London, England: Longman.
McNamara, W. J., & Weitzman, E. (1945). The effect of choice placement on the difficulty of multiple-choice questions. Journal of Educational
Psychology, 36(2), 103.
Meara, P. M. (1992). EFL vocabulary tests: Swansea: Centre for Applied Language Studies, University of Wales.
Meara, P. M. (1996). The dimensions of lexical competence. In Brown, G., Malmkjaer, K., & Williams, J. (Eds.), Performance and competence in second language acquisition (pp. 35-53). Cambridge, England: Cambridge University Press.
Meara, P. M., & Milton, J. L. (2003). X_Lex: The Swansea vocabulary levels test.
Newbury, England: Express.
Melka, F. (1997). Receptive vs. productive aspects of vocabulary. In Schmitt, N., &
McCarthy, M. (Eds.), Vocabulary: Description, acquisition and pedagogy (pp. 84-102). Cambridge, England: Cambridge University Press.
MelkaTeichroew, F. J. (1982). Receptive versus productive vocabulary: a survey.
67 Interlanguage Studies Bulletin, 6(2), 5-33.
Messick, S. (1980). Test validity and the ethics of assessment. American Psychologist, 35(11), 1012.
Messick, S. (1990). Validity of test interpretation and use. Princeton, NJ: Educational Testing Service. (ERIC ED 395031)
Messick, S. (1996). Validity and washback in language testing. Language Testing, 13(3), 241-256.
Milton, J. (2006). X-Lex: The Swansea vocabulary levels test. In Coombe, C.,
Davidson, P., & Lloyd, D. (Eds.), Proceedings of the 7thand 8thCurrent Trends in English Language Testing (CTELT) Conference: Vol. 4. (pp. 29-39). UAE:
TESOL Arabia.
Milton, J. (2009). Measuring second language vocabulary acquisition. Bristol, England: Multilingual Matters.
Milton, J., &Daller, H. (2007). The interface between theory and learning in
vocabulary acquisition. Paper presented at EUROSLA 2008, Newcastle, UK.
Morgan, B., &Oberdeck, L. M. (1930). Active and passive vocabulary. Studies in Modern Language Teaching, 16, 213-221.
Morton, J. (1979). Word recognition. Ithaca, NY: Cornell University Press.
68
Nagy, W., Anderson, R. C., Schommer, M., Scott, J. A., & Stallman, A. C.
(1989).Morphological families in the internal lexicon. Reading Research Quarterly, 262-282.
Nagy, W. E., Herman, P. A., & Anderson, R. C. (1985). Learning words from context.
Reading Research Quarterly, 233-253.
Nagy, W. E., & Scott, J. A. (1990). Word schemas: Expectations about the form and meaning of new words. Cognition and Instruction, 7(2), 105-127.
Nation, P. (2001). Learning vocabulary in another language. Cambridge, England:
Cambridge University Press.
Nation, P. (2006). How Large a Vocabulary is Needed For Reading and Listening?
Canadian Modern Language Review, 63(1), 59-82.
Nation, P. (1990). Teaching and learning vocabulary. New York: Heinle&Heinle Boston.
Nation, P., &Waring, R. (1997). Vocabulary size, text coverage and word lists. In Schmitt, N., & McCarthy, M. (Eds.), Vocabulary: Description, acquisition and pedagogy (pp. 6-19). Cambridge, England: Cambridge University Press.
Nevo, N. (1989). Test-taking strategies on a multiple-choice test of reading comprehension. Language Testing, 6(2), 199-215.
Palmer, H. E. (1968). The scientific study and teaching of languages. London: Oxford U.P..
69
Paribakht, T. S., &Wesche, M. (1997). Vocabulary enhancement activities and reading for meaning in second language vocabulary acquisition. In Coady, J.,
&Huckin, T. (Eds.), Second language vocabulary acquisition: A rationale for pedagogy (pp. 174-200). Cambridge, England: Cambridge University Press.
Qian, D. D. (2002). Investigating the relationship between vocabulary knowledge and academic reading performance: An assessment perspective. Language
Learning, 52(3), 513-536.
Read, J. (1988). Measuring the Vocabulary Knowledge of Second Langauge Learners.
RELC Journal, 19(2), 12-25.
Read, J. (2000). Assessing vocabulary. Cambridge, England: Cambridge University Press.
Riazi, A. M., &Mosalanejad, N. (2010). Evaluation of Learning Objectives in Iranian High-School and Pre-University English Textbooks Using Bloom'sTaxonomy.
TESL-EJ,13(4).
Schmitt, N. (2010). Researching vocabulary: A vocabulary research manual.
Basingstoke, U.K.: Palgrave Macmillan.
Schmitt, N., & McCarthy, M. (1997). Vocabulary: Description, acquisition and pedagogy. Cambridge, England: Cambridge University Press.
Schmitt, N., Schmitt, D., &Clapham, C. (2001). Developing and exploring the
70
behaviour of two new versions of the Vocabulary Levels Test.
Language Testing, 18(1), 55-88.
Schonell, F., Meddleton, I., Shaw, B., Routh, M., Popham, D., Gill, G., et al. (1956).
A Study of the Oral Vocabulary of Adults. Brisbane and London: University of Queensland Press/ University of London Press.
Schumacker, R. E. (2005). Classical test analysis. Unpublished doctoral thesis, Applied Measurement Associates.
Scott, J. A., & Nagy, W. E. (2009). Developing word consciousness. In Graves, M.
F. (Eds.), Essential readings on vocabulary instruction (pp. 102-113).
Newark, DE: International Reading Association
Slakter, M. J. (1968). The effect of guessing strategy on objective test scores. Journal of Educational Measurement, 5(3), 217-222.
Stæhr, L. S. (2008). Vocabulary size and the skills of listening, reading and writing.
Language learning journal, 36(2), 139-152.
Stæhr, L. S. (2009). Vocabulary knowledge and advanced listening comprehension in English as a foreign language. Studies in Second Language Acquisition, 31(04), 577.
Taylor, P. H. (1966). A study of the effects of instructions in a multiple‐ choice
71
mathematics test. British Journal of Educational Psychology, 36(1), 1-6.
Thorndike, R. L. (1949). Personnel selection; test and measurement techniques. Oxford, England: Wiley.
Thorndike, E.L., &Lorge, I. (1944). The teacher’s word book of 30,000 words.
New York: Teachers College, Columbia University.
Ulijn, J. M., &Strother, J. B. (1990). The effect of syntactic simplification on reading EST texts as L1 and L2. Journal of Research in Reading, 13(1), 38-54.
Waring, R. (1997). A comparison of the receptive and productive vocabulary sizes of some second language learners. Immaculata(1), 53-68.
Webb, S. (2005). Receptive and productive vocabulary learning. Studies in Second Language Acquisition, 27, 33-52.
Webb, S. (2007). The effects of repetition on vocabulary knowledge. Applied Linguistics, 28(1), 46-65.
Webb, S. (2008). Receptive and productive vocabulary sizes of L2 learners. Studies in Second Language Acquisition, 30(1), 79.
West, M. (1953). A general service list of English words. London: Longman, Green and Co.
Nagy, W.E., Herman, P.A. & Anderson R.C. (1985). Learning words from context.
Reading Research Quarterly, 20(2), 20.
72
Xue, G., & Nation, I. S. P. (1984). A university word list. Language learning and communication, 3(2), 215-229.
Zareva, A., Schwanenflugel, P., &Nikolova, Y. (2005). Relationship between lexical competence and language proficiency: Variable sensitivity. Studies in Second Language Acquisition, 27(04), 567-595
73