The present study investigated whether the new AWE system can generate comprehensive, usable feedback for its users so as to help teachers in the aspect of providing writing feedback. However, there are still some limitations and relevant issues worth to be explored in the future.
First, there were only 146 compositions collected and analyzed in the study, and there were only two composition topics. In addition, all the compositions were written by EFL high school students in Taiwan. In future research, the number of student subjects should be enlarged, such as L1 learners or EFL learners from other countries.
There should also be more essays on various topics so as to have a larger number of samples.
Second, the two human raters were both non-native high school teachers and the researcher served as the rater who checked the false alarm messages provided by
Correct English. It is likely that they would misjudge the errors or put the errors in the
wrong categories. If possible, future studies should try to include native teachers as raters in the process of data analysis so that the data collected will be more reliable.Third, to see how students repsond to the feedback provided by AWE systems, future researchers may use the AWE systems in real writing class to see whether or how students use the computerized feedback in their revisions.
Fourth, the developers of AWE systems should try to include the function for
69
users to change the settings. For examples, in the feedback provided by Correct English, contraction errors were constantly flagged, which were not necessarily errors.
If users are allowed to change the settings, they can just “ignore” the same kind of errors and thus reduce the frequency of false alarms.
Finally, there is a need for longitudinal research to investigate whether learners benefit from the writing feedback provided by AWE systems and thus reduce the frequency of the targeted errors in their future writing. By doing so, it will provide guidelines of the use of AWE systems for writing teachers, language learners and developers of the AWE systems.
70
REFERENCES
Attali, Y. (2004) Exploring the feedback and revision features in Criterion. Paper presented at the National Council on Measurement in Education (NCME) held between April12-16, 2004, in San Diego, CA
Attali, Y. & Burstein, J. (2006). Automated essay scoring with e-rater V.2.0. Journal
of Technology, Learning and Assessment, 4(3). Retrieved September 29, 2009
from http://www.jtla.org.Attali, Y., Bridgeman, B., & Trapani, C. (2010). Performance of a Generic Approach in Automated Essay Scoring. Journal of Technology, Learning and Assessment,
10(3). Retrieved July 1, 2011 from http://www.jtla.org
Ben-Simon, A. & Bennett, R.E. (2007). Toward more substantively meaningful automated essay scoring. Journal of Technology, Learning and Assessment, 6(1).
Retrieved September 29, 2009 from http://www.jtla.org.
Burstein, J. & Chodorow, M.(1999). Automated essay scoring for nonnative English speakers. In M. B. Olson (Ed.), Proceedings of the ACL99 workshop on computer
mediated language assessment and evaluation of natural language processing.
College Park, MD: Association of Computational Linguistics.
Burstein, J. & Chodorow, M.(2002). Directions in automated essay analysis. In R.
Kaplan (Ed.), The Oxford handbook of applied linguistics (p.p. 487-497). New York: Oxford University Press.
Burstein, J., Chodorow, M., & Leacock, C. (2003). CriterionSM: Online essay evaluation: An application for automated evaluation of student essays. Proceeding
of the Fifteenth Annual conference on Innovative Application of Artificially Intelligence, Acapulco , Mexico.
Burstein, J. & Chodorow, M. (2003) The e-rater scoring engine: Automated Essay Scoring with natural language processing. In M. D. Shermis and J. C. Burstein (Eds.) Automated Essay Scoring: A cross disciplinary approach (pp. 113-121).
Mahwah, NJ: Lawrence Erlbaum Associates.
Burstein, J., Chodorow, M., & Leacock, C. (2004). Automated essay evaluation: The Criterion online writing service. AI Magazine, 25(3), 27-36
71
Burston, J. (2001) Computer-mediated feedback in composition correction. CALICO
Journal, 19 (1) 37-50
Calfee, R. (2000) To grade or not to grade. IEEE Intelligent Systems, 15 (5), 35-37.
Chen, C-F. & Cheng, W-Y. (2006). The Use of a Computer-based Writing Program:
Facilitation or Frustration? Paper presented at the 23rd International Conference on English Teaching and Learning in the R.O.C. May 27, 2006. Kaohsiung: Wenzao Ursuline College of Languages.
Chen, C-F. & Cheng, W-Y. (2010) Beyond the Design of Automated Writing Evaluation: Pedagogical Practices and Perceived Learning Effectiveness in EFL Writing Classes. Language Learning & Technology, 12(2), 94-112.
Chen, H. J. (2006). Examining the Scoring Mechanism and Feedback Quality of My Access. Proceedings of Tamkang University Conference on Second Language
Writing.
Cheville, J. (2004) Automated scoring technologies and the rising influence of error.
English Journal, 93 (4), 47-52
Cohen, Y., Ben-Simon, A. & Hovav, M. (2003) The effect of specific language features on the complexity of systems for automated essay scoring. Paper presented at the 29th Annual Conference, Manchester, UK.
Dikli, S. (2006). An Overview of automated essay scoring. Journal of Technology,
Learning and Assessment, 5(1). Retrieved September 29, 2009 from http://www.jtla.org.
Elliot, S. (2003) Intellimetric: from here to validity. In M. D. Shermis and J. C.
Burstein (Eds.) Automated Essay Scoring: A cross disciplinary approach (pp.
71-86). Mahwah, NJ: Lawrence Erlbaum Associates.
Ferris, D. (1999). The Case for Grammar Correction in L2 Writing Classes: A Response to Truscott. Journal of Second Language Writing, 8(1), 1-11.
Ferris, D. (2002). Treatment of Error in Second Language Student Writing. Ann Arbor , MI : University of Michigan Press.
72
Ferris, D. (2004) The “Grammar Correction” Debate in L2 Writing: Where are we, and where do we go from here? (and what do we do in the meantime…?) Journal of
Second Language Writing, 13, 49-62.
Ferris, D. (2006). Does error feedback help student writers? New evidence on the short- and long-term effects of written error correction. In K. Hyland & F. Hyland (Eds.), Feedback in second language writing: Contexts and issues (pp. 81–104).
Cambridge : Cambridge University Press.
Francis, N. (2007) Corrective feedback in L2 assessment: negative evidence and interactive practice. Selected papers from the 16th
International Symposium on English Teaching. Taipei: English Teachers Association, Y. N. Leung (Ed.)
376-386.Goldstein, L. (2006). Feedback and revision in second language writing: Contextual, teacher, and student variables. In K. Hyland & F. Hyland (Eds.), Feedback in second
language writing: Contexts and issues (pp. 185–205). Cambridge : Cambridge
University Press.Guenette, D. (2007). Is feedback pedagogically correct? Research design issues in studies of feedback on writing. Journal of Second Language Writing, 16, 40-53.
Herrington, A. & Moran, C. (2001) What happens when machines read our students’
writing? College English, 63 (4), 480-499
Hutchison, D. (2007) An evaluation of computerized essay marking for national curriculum assessment in the UK for 11-year-olds. British Journal of Educational
Technology, 38 (6), 977-989.
Grimes, D. & Warschauer, M. (2010) Utility in a Fallible Tool: A Multi-Site Case Study of Automated Writing Evaluation. Journal of Technology, Learning and
Assessment, 8(6), Retrieved August 30, 2010 from http://www.jtla.org.
Kellogg, R.T., Whiteford, A.P. & Quinlan, T. (2010) Does automated feedback help students learn to write? Journal of Educational Computing Research, 42 (2),
173-196.
Kukich, K. (2000) Beyond automated essay grading. IEEE Intelligent Systems, 15(5), 22-27.
73
Lai, Y. H. (2009) Which do students prefer to evaluate their essays: peers or computer program. British Journal of Educational Technology, 38 (6), 1-23.
Otoshi, J. (2005). An Analysis of the Use of Criterion in a Writing Classroom in Japan . The JALT CALL Journal, 1 (1), 30-38.
Page, E. B. (2003) Project Essay Grade: PEG. In M. D. Shermis and J. C. Burstein (Eds.) Automated Essay Scoring: A cross disciplinary approach (pp. 43-54).
Mahwah, NJ: Lawrence Erlbaum Associates.
Rudner, L. M, Garcia, V., & Welch, C. (2006) An Evaluation of the Intellimetric Essay Scoring System. Journal of Technology, Learning, and Assessment, 4 (4).
Available from http:// www. Jtla.org
Powers, D. E., Burstein, J. C., Chodorow, M. E., &Kukich, K. (2002) Stumping e-rater: challenging the validity of automated essay scoring. Computers in Human
Behavior, 18, 103-134
Scharber, C., Dexter, S. & Riedel, E. (2008) Students’ experiences with and automated essay scorer. Journal of Technology, Learning and Assessment, 7(1) Retrieved September 29, 2009 from http://www.jtla.org
Shermis, M. D. & Burstein, J. C. (2003) Automated Essay Scoring: A cross
disciplinary approach. Mahwah, NJ: Lawrence Erlbaum Associates.
Shermis, M. D., Rayment, M. V., & Barrera, F. (2003) Assessing writing through the curriculum with automated essay scoring. Paper presented at the Annual Meeting of the American Educational Research Association, Chicago, IL.
Shermis, M.D., Shneyderman, A., & Attali, Y. (2008) How important is content in the ratings of essay assessments? Assessment in Education: Principles, Policy &
Practice, 15(1), 91-105.
Shermis, M.D., Garvan, G. W., & Diao, Y. (2008) The impact of automated essay scoring on writing outcomes. Paper presented at the Annual Meetings of the National Council on Measurement in Education, New York, NY.
Truscott, J. (1996). The case against grammar correction in L2 writing classes.
Language Learning, 46, 327–369.
74
Truscott, J. (1999). The case for “The case against grammar correction in L2 writing classes”: A response to Ferris. Journal of Second Language Writing, 8(2), 111-122.
Truscott, J. (2007). The effect of error correction on learners’ ability to write accurately. Journal of Second Language Writing, 16, 255-272.
Truscott, J. & Hsu, A. Y. (2008). Error correction, revision, and learning. Journal of
Second Language Writing, 17, 292-305.
Wang, J. & Brown, M. (2007). Automated essay scoring versus human scoring: a comparative study. Journal of Technology, Learning and Assessment, 6(2) Retrieved September 29, 2009 from http://www.jtla.org
Ware, P. & Warschauer, M. (2006) Electronic feedback and second language writing.
In K. Hyland & F. Hyland (Eds.), Feedback in second language writing: Contexts
and issues (pp. 105-122). Cambridge : Cambridge University Press.
Ware, P. & Warschauer, M. (2006) Automated writing evaluation: defining the classroom research agenda. Language Teaching Research, 10 (2), 157-180.
Warschauer, M. & Grimes, D. (2008) Automated writing assessment in the classroom.
Pedagogies: An International Journal, 3, 22-36
Yang, N. D. (2004). Using My Access in EFL writing. The proceedings of 2004
International Conference and Workshop on TEFL & Applied Linguistics (pp.
550-564). Taipei , Ming Chuan University .
Yeh, Y., Liou, H. C. & Yu, Y. T. (2007). The influence of automated essay evaluation and bilingual concordancing on EFL students’ writing. English Teaching &
Learning, 31 (1), 117-160.
75
Appendix: Teacher’s Rewrite
The Most Impressive Advertisement/ Commercial I’ve Ever Seen (Class A ) A9 It is really a successful commercial that
make me not resist the magic power
of the snack.…makes me unable to resist/ fall for the magical power of the snack.
It concerns not only about what a good camera can capture but about…
A14 And every year will publish the new commercial,…
Every year the company will release a new commercial,…
A24 I knew the feeling very well that when
pets being bad and you couldn’t
punish them …I know the feeling very well. When pets are acting bad, you couldn’t…
A36 No matter which kind of situation or where we are, we all can use Visa.
No matter which kind of situation we are in or….
A38 So I always turned on the TV to the loudest,…
So I always turn up the volume on the TV to the loudest,…
The Most Impressive Advertisement/ Commercial I’ve Ever Seen (Class B) B3 …, the scene quickly turns to New
Zeeland, where is the origin of milk.
…New Zealand, where milk
we can exist with the beautiful nature until we die.
2. It’s only to click the number and answer the phone, then the pizza would
All I have to do is dial the number and order, and the pizza would
76
emerge in my sight. emerge in my sight.
B13 A true love of mom. It shows true love from a mother.
B15 1. Even the father of her did
everything that she wanted or just will good for her.
Her father fulfilled every need of her.
2. They don’t longer have chances to accompany their sweet hands with hands going to school.
They no longer have chances to join their hands and go to school together.
3. …, its background always play with something seems like sorrow. makes me feel impressed is the idea of family.
2. …, why it needs to advertise? …, why does it still need advertising?
B20 …, another scream from men voice. …,there is screaming from men.
B23 1. …, so that it was not to tell where the line is.
…, so that it was impossible to tell where the line is. impressed me the most was about a special drink.
2. Although we had a terrible thing at the work or studying, we only drank this then we would enjoy a relax trip in our imagination.
Although we had terrible things at work or at study, we only have to drink this and we would enjoy a relaxing trip in our imagination.
B25 Why the ad impressiving because while the world cup was took place in 2006, everytime during the games
played exciting, at the moment,
The ad is impressive because while the world cup took place in 2006, every time the games were
77
3. They didn’t have happy ending that
made me barely to cry.
can make more space and less garbage.The bottle can be flatten to a small piece and make …
B29 1. One man explain how amazing and we may recover from the sickness…
One man explains its magical effects and…
2. …a good ad not only bring lots of purchase at also have impacts on the audience deeply for a long time
a good ad not only brings a lot of profits for the company but it also has impacts on…
B30 I very like this TV commercial because it has not only an innovative idea but also particular pride belong to yong people.
I like this commercial very much because it not only has an innovative idea but it also shows a particular pride belonging to young people.
B37 She still stood on the stage with a violin which was all over tapes.
…which had tapes all over.
B38 The advertisement made me impressed is owing to the funny jingle.
What the advertisement impressed me is the funny jingle.
B40 1. But there is a commercial making
me impressive. (X3)
… that impresses me.
2. I think that is so funny. Even impossible
I think that is so funny, even though it is impossible.
B41 Drinking it is just like becoming more thin
Drinking it makes you become thinner.
B43 I have very impressive in commercial about TOYOTA-YARIS.
I saw an impressive commercial…
B44 “What is a kind of the flower? What kind of flower it is?
B46 I have very impressive in the advertise about the supermarket.
I am very impressed with the advertisement for a supermarket.
If you were to go abroad for a year, what would you bring with you ? (Class A) A1 Were it not for the photos, I may forget
the good times in Disneyland and the brilliant days in New York City.
78
to make some record about it,…
daily happenings.I am scared not knowing what to do. I am too scared to do anything.
A14 And computer or cell-phone also can borrow from others, but camra can not.
I can borrow computer or cellphone from other, but I can’t use other people’s camera.
A16 So I want to take a cell phone which
with ability of camera that I could
take pictures…with the function of camera…
A17 He cameras for me made me feel I was important to him though every
time I went home after school, there
was only an empty house to greet me.He took pictures for me and made me feel I was important to him. Every time I went home after school, …
Though past many years, I have
People may ask what I could do with the newspapers and why I don’t just buy local ones,… can do nothing to contact people that is meaningful to me
Without this electronic device, I cannot contact people that are meaningful to me.
A22 , if I come across a breath-taking scene,
maybe a magnificent historical
building or a café in the street corner with sunlight pouring in gently.
, be it a…
A24 Having so many useful functions in it, taking the cell phone with me in a trip is really helpful
With so many useful functions in it, the cell phone is perfect for a trip.
A28 As far as items are concerned, I most likely to take a radio during this year,
As far as items to take are concerned, I am most likely to …
A35 …, and later I would think those things just happened yesterday
I would think of those things as if they just happened yesterday.
A40 My memories can never be covered
and forget
…lost and forgotten.
A43 …, and I can disposal everything but it, that is computer.
I can dispose of anything but not the computer.
79
If you were to go abroad for a year, what would you bring with you ? (Class B) B5 1. It is my habit that everything can’t
change it.
It is my habit not to change anything.
2. Therefore, I bring my favorite coffee pig doll with me to travel other
national such as homelike.
2. I will bring my camera to capture the views where I had visited.
…capture the views which I see.
B7 You want to overtake him, that feeling
is everybody player can realize,
…, which is what every player can realize,…
B9 Why am I so indispensable with is? Why is it so indispensable?
B12 However, once I tried to get quit of the habit, The fact is I failed, and can’t sleep ing a good situation.
I tried to get rid of the habit, but I failed, because I can’t sleep in a good situation otherwise.
B13 It’s just like a camera for photographer as a notebook for me
A camera is to a photographer is what a notebook is to me.
B14 I will keep the ticket as a collection to prove me have visited the meseum.
….as a souvenir to prove that I have visited the museum.
B15 when I visit someone or something looks isn’t very nice.
…or when something doesn’t look very nice.
B20 I am very like to listen music, no matter what kind of them.
I like to listen to music very much, regardless of its type. pictures and I will recall the past memories.
4. It so magic lets my life fill with splendid chapters.
It is so magical that it fills my life with splendid chapters.
B22 1. …the shock due to two different …the culture shock may bother me a
80
culture must be a lot. lot.
2. I can relief my unease and make a memory in words.
I can relieve my anxiety and turn memories into words.
B23 1. …just as I carry my friend with me go abroad together.
…as if I carried a friend with me along the trip.
2. When I can't bear my life differ from the life I came to view,…
When I can’t bear a life different from the life I have now,…
B24 Painting pictures and writing sentences
have much mention than using
2. Mom can grasp my message so that
she can relieve herself.
…she can feel relieved.
B27 I can use the camera function to catch
it and sent to my facebook.
…to capture the images and upload them on my facebook page.
B31 1. …and their face have various expression.
…there are various kinds of expressions on their faces.
2. …but it fulls lost of memory about us.
…it is full of memories about us.
3. …her also can talk with me by heart and we share anything together
…she can also talk to me and share everything with me.
B34 Basing on these reasons, notebook is the best choice for me when I have to go abroad with bringing something.
Based on these reasons, …when it comes to the most important thing I would bring along.
B36 Besides, the picture can recall me that who am I and and why am I here when I lose myself.
Besides, the picture can make me recall who I am and why am I here when…
B37 Watching the pictures has me recall a lot of support from families.
When I watch the pictures, I’ll recall a lot of support from families.
B38 1. …,which can comfort me missing
home.
…my homesickness.
2. A ipod likes my best friend and can't leave me
I can write many things in the notebook.
81
B43 While I travel in foreign country, I would never forget to take my girlfriend's picture.
If I travel to a foreign country, …
B44 1. Reading these letters would remind me of pleasure along with him just
like he accompanies me.
…the pleasure I had with him as if he accompanied me.
B45 It's a cell phone. It can call to my friend,…
With the cell phone, I can call my friends,…
B46 1. When I take necessaries to the travel, I will take the camera too.
Speaking of the necessities during travel, I will think of the camera.
2. But many of figures and views that I can't remember clearly.
I can’t remember every figure and beautiful sight clearly.