How to Access Writing
5 Is the task feasible?
The writing task must be workable for both the students who will write and the teacher who will read. Is the topic writable—can a piece of connected discourse actually be composed in response to the topic? Or, is the topic so complex or intricate that composing a coherent essay is practically impossible for either first-language or second-language writers? The same consideration applies from the teacher’s point of view—is the writing task so complex that the teacher cannot determine what constitutes a good or poor composition? The feasibility criterion is also related to the level of difficulty and amount of time for the writing task—is the task such that it is not too difficult to compose within the time limits set for the task? All of these considerations will obviously affect both the validity and reliability of the assessment.
II
In composition evaluation two basic scoring scales are adopted: holistic scales and analytic scales.
Holistic Scoring
Many assessment programs rely on holistic scoring, or the assigning of a single score to a script based on the overall impression of the script. In a typical holistic scoring session, each script is read quickly and then judged against a rating scale, or scoring rubric, that outlines the scoring criteria.
A well-known example of a holistic scoring rubric in ESL is the scale used for the TOEFL Writing Test, formerly known as the Test of Written English (TWE).
TOEFL Writing Scoring Guide 6 points
An essay at this level:
• effectively addresses the writing task;
• is well-organized and well-developed;
• uses clearly appropriate details to support a thesis or illustrate ideas;
• displays consistent facility in use of language;
• demonstrates syntactic variety and appropriate word choice though it may have occasional errors.
5 points
An essay at this level:
• may address some parts of the task more effectively than others;
• is generally well-organized and developed;
• uses details to support a thesis or illustrate an idea;
• displays facility in the use of language;
173 Appendix
• demonstrates some syntactic variety and range of vocabulary, though it will probably have occasional errors.
4 points
An essay at this level:
• addresses the writing topic adequately but may slight parts of the task;
• is adequately organized and developed;
• uses some details to support a thesis or illustrate an idea;
• demonstrates adequate but possibly inconsistent facility with syntax and usage;
• may contain some errors that occasionally obscure meaning.
3 points
An essay at this level may reveal one or more of the following weaknesses:
• inadequate organization or development;
• inappropriate or insufficient details to support or illustrate generalizations;
• a noticeably inappropriate choice of words or word forms;
• an accumulation of errors in sentence structure and / or usage.
2 points
An essay at this level is seriously flawed by one or more of the following weaknesses:
• serious disorganization or underdevelopment;
• little or no detail, or irrelevant specifics;
• serious and frequent errors in sentence structure or usage;
• serious problems with focus.
1 point
An essay at this level:
• may be incoherent;
• may be undeveloped;
• may contain severe and persistent writing errors.
0 point
A paper is rated 0 if it contains no response, merely copies the topic, is off-topic, is written in a foreign language, or consists of only keystroke characters.
Another example of a holistic scoring rubric in ESL is the WRITING ASSESSMENT CRITERIA used for the CAMBRIDGE CERTIFICATE IN ADVANCED ENGLISH (CCIAE).
Writing Assessment Criteria for CCIAE 5 points
Totally positive effect on target reader. Minimal errors. Resourceful, controlled and natural use of language showing a good range of vocabulary and structure. Completion of task: well-organized, good use of cohesive devices, appropriate register, no relevant omissions.
4 points
Sufficiently natural, errors only when more complex language attempted. Some evidence of range
174 Appendix
of vocabulary and structure. A good attempt at achieving the task. Any omissions are only minor.
Attention paid to organization and cohesion; register not always natural but positive effect on target reader achieved.
3 points
Use of English satisfactory, though lacking range and variety. Occasional serious errors should not impede communication although patience is required of readers. Task reasonably attempted with some organization and cohesion. No significant irrelevancies.
2 points
Errors sometimes obscure communication and / or language too elementary. Some attempt at task but notable omissions and / or lack of organization and cohesion would have a negative effect on reader.
1 point
Serious lack of control and / or frequent basic errors. Narrow range of language. Totally inadequate attempt at task.
0 point
Not sufficiently comprehensible language for assessment.
As can be seen from these two examples, the scales contain descriptors of the syntactic and rhetorical qualities of six levels of writing proficiency. Holistic scoring has become widely used in writing assessment over the past 25 years and has a number of positive features. From a practical standpoint, it is faster (and therefore less expensive) to read a script once and assign a single score than to read it several times, each time focusing on a different aspect of the writing. Holistic scoring is intended to focus readers’ attention on the strengths of the writing, not on its deficiencies, so that writers are rewarded for what they do well. Holistic scoring rubrics can be designed to focus readers’ attention on certain aspects of writing, depending on what is deemed most essential in the context, and thus can provide important information about those aspects in an efficient manner.
On the other hand, holistic scoring has several disadvantages, particularly in second-language contexts. One drawback is that a single score does not provide useful diagnostic information about a person’s writing ability, as a single score does not allow raters to distinguish between various aspects of writing such as control of syntax, depth of vocabulary, organization, and so on. This is especially problematic for second-language writers, since different aspects of writing ability develop at different rates for different writers: Some writers have excellent writing skills in terms of content and organization but may have much lower grammatical control, while others may have an excellent grasp of sentence structure but may not know how to organize their writing in a logical way.
Another disadvantage of holistic scoring is that holistic scores are not always easy to interpret, as raters do not necessarily use the same criteria to arrive at the same scores: For example, a certain script might be given a 4 on a holistic scale by one rater because of its rhetorical features (content, organization, development), while another rater might give the same script a 4 because of its linguistic features (control of grammar and vocabulary). Holistic scores have also been shown to correlate with relatively superficial characteristics such as length and handwriting.
175 Appendix
Analytic Scoring
In analytic scoring, scripts are rated on several aspects of writing or criteria rather than given a single score. Depending on the purpose of the assessment, scripts might be rated on such features as content, organization, cohesion, register, vocabulary, grammar, or mechanics. Analytic scoring schemes thus provide more detailed information about a test taker’s performance in different aspects of writing and are for this reason preferred over holistic schemes by many writing specialists.
One of the best-known and most widely-used analytic scales in ESL was created by Jacobs et al. (1981).
In the Jacobs et al. scale, scripts are rated on five aspects of writing: content, organization, vocabulary, language use, and mechanics. The five aspects are differentially weighted to emphasize first content (30 points) and next language use (25 points), with organization and vocabulary weighted equally (20 points) and mechanics receiving very little emphasis (5 points). This scale has been adopted by numerous college level writing programs, and is accompanied by training materials and sample compositions so that users can fairly quickly learn to apply the scale.
Jacobs et al.’s Scoring Profile
Content
30-27 26-22 21-17 16-13
EXCELLENT TO VERY GOOD: knowledgeable·substantive·thorough development of thesis·
relevant to assigned topic
GOOD TO AVERAGE: some knowledge of subject·adequate range·limited development of topic FAIR TO POOR: limited knowledge of subject·little substance·inadequate development of topic VERY POOR: does not show knowledge of subject·non-substantive·not pertinent·OR not enough to evaluate
EXCELLENT TO VERY GOOD: fluent expression·ideas clearly stated / supported·succinct·well-organized·logical sequencing cohesive
GOOD TO AVERAGE: somewhat choppy·loosely organized but main ideas stand out·limited support·logical but incomplete sequencing
FAIR TO POOR: non-fluent·ideas confused or disconnected·lacks logical sequencing and development VERY POOR: does not communicate·no organization·OR not enough to evaluate
Vocabulary
20-18 17-14 13-10 9-7
EXCELLENT TO VERY GOOD: sophisticated range·effective word / idiom choice and usage·
word form mastery·appropriate register
GOOD TO AVERAGE: adequate range·occasional errors of word / idiom form, choice, usage but meaning not obscured
FAIR TO POOR: limited range·frequent errors of word / idiom form, choice, usage·meaning confused or obscured
VERY POOR: essentially translation·little knowledge of English vocabulary, idioms, word form·
OR not enough to evaluate
Language Use
25-22 21-18
17-11
10-5
EXCELLENT TO VERY GOOD: effective complex constructions·few errors of agreement, tense, number, word order / function, articles, pronouns, prepositions
GOOD TO AVERAGE: effective but simple constructions·minor problems in complex constructions·
several errors of agreement, tense, number, word order / function, articles, pronouns, prepositions but meaning seldom obscured
FAIR TO POOR: major problems in simple / complex constructions·frequent errors of negation, agreement, tense, number, word order / function, articles, pronouns, prepositions and / or fragments, run-ons, deletions·meaning confused or obscured
VERY POOR: virtually no mastery of sentence construction rules·dominated by errors does not communicate·OR not enough to evaluate
Mechanics
5 4 3 2
EXCELLENT TO VERY GOOD: demonstrating mastery of conventions·few errors of spelling, punctuation, capitalization, paragraphing
GOOD TO AVERAGE: occasional errors of spelling, punctuation, capitalization, paragraphing FAIR TO POOR: frequent errors of spelling, punctuation, capitalization, paragraphing·poor handwriting·meaning confused or obscured
VERY POOR: no mastery of conventions·dominated by errors of spelling, punctuation, capitalization, paragraphing·handwriting illegible·OR not enough to evaluate
176 Appendix
A slightly different approach to analytic scoring for second-language writing assessment is a set of scales developed for the Test in English for Educational Purposes (TEEP). Instead of a single scale composed of a number of subscales, Weir’s scheme consists of seven scales, each divided into four levels with score points ranging from 0 to 3. The first four scales are related to communicative effectiveness, while the others related to accuracy. Like the Jacobs et al.’s scale, the TEEP scale was extensively piloted and revised to make sure that it could be applied reliably by trained raters.
TEEP Attribute Writing Scales
A Relevance and adequacy of content
0 The answer bears almost no relation to the task set. Totally inadequate answer.
1 Answer of limited relevance to the task set. Possibly major gaps in treatment of topic and / or pointless repetition.
2 For the most part answers the tasks set, though there may be some gaps or redundant information.
3 Relevant and adequate answer to the task set.
B Compositional organization 0 No apparent organization of content.
1 Very little organization of content. Underlying structure not sufficiently controlled.
2 Some organizational skills in evidence, but not adequately controlled.
3 Overall shape and internal pattern clear. Organizational skills adequately controlled.
C Cohesion
0 Cohesion almost totally absent. Writing so fragmentary that comprehension of the intended communication is virtually impossible.
1 Unsatisfactory cohesion may cause difficulty in comprehension of most of the intended communication.
2 For the most part satisfactory cohesion although occasional deficiencies may mean that certain parts of the communication are not always effective.
3 Satisfactory use of cohesion resulting in effective communication.
D Adequacy of vocabulary for purpose
0 Vocabulary inadequate even for the most basic parts of the intended communication.
1 Frequent inadequacies in vocabulary for the task. Perhaps frequent lexical inappropriateness and / or repetition.
2 Some inadequacies in vocabulary for the task. Perhaps some lexical inappropriateness and / or circumlocution.
3 Almost no inadequacies in vocabulary for the task. Only rare inappropriateness and / or circumlocution.
177 Appendix
E Grammar
0 Almost all grammatical patterns inaccurate.
1 Frequent grammatical inaccuracies.
2 Some grammatical inaccuracies.
3 Almost no grammatical inaccuracies.
F Mechanical accuracy I (punctuation) 0 Ignorance of conventions of punctuation.
1 Low standard of accuracy in punctuation.
2 Some inaccuracies in punctuation.
3 Almost no inaccuracies in punctuation.
G Mechanical accuracy II (spelling) 0 Almost all spelling inaccurate.
1 Low standard of accuracy in spelling.
2 Some inaccuracies in spelling.
3 Almost no inaccuracies in spelling.
The primary advantage of an analytic scoring scheme over a holistic scheme is that it provides more useful diagnostic information about students’ writing abilities. However, analytic scoring has a number of other advantages over holistic scoring as well. First, analytic scoring is more useful in rater training, as inexperienced raters can more easily understand and apply the criteria in separate scales than in holistic scales. Analytic scoring is particularly useful for second-language learners, who are more likely to show a marked or uneven profile across different aspects of writing: For example, a script may be quite well-developed but have numerous grammatical errors, or a script may demonstrate an admirable control of syntax but have little or no content. Finally, analytic scoring can be more reliable than holistic scoring: Just as reliability tends to increase when additional items are added to a discrete-point test, so a scoring scheme in which multiple scores are given to each script tends to improve reliability.
The major disadvantage of analytic scoring is that it takes longer than holistic scoring, since readers are required to make more than one decision for every script. An additional problem with some analytic scoring schemes is that, if scores on the different scales are combined to make a composite score, a good deal of the information provided by the analytic scale is lost. It may also be the case that raters who are experienced at using a particular analytic scoring system may actually rate more holistically than analytically if scores are combined into a single score: Experienced raters may target their ratings toward what they expect the total score to come out to be, and revise their analytic scores accordingly.
Reference
Jacobs, H. L., Zinkgraf, S. A., Wormuth, D. R., Hartfiel, V. F., & Hughey, J. B. (1981). Testing ESL Composition. Rowley, MA: Newbury House Publishers.