In this work, we present StructFeed, a system that helps ESL writers to improve the qual-ity of writings by receiving structural feedback. In the system, a crowdsourcing workflow was proposed to guide crowd workers to annotate topic sentences and relevant keywords through micro-tasks. By aggregating crowd annotations, the system can generate writing hints for directing people to address the structure issues effectively. In a field deploy-ment study, we showed that our system could help ESL writers improve their writings.
In addition, people who received feedback from StructFeed outperformed people who re-ceived feedback from a expert or a crowd worker. StructFeed enables new kind of writing feedback that cannot obtain from other sources. The work pioneers the design space of generating writing feedback with crowdsourcing mechanisms for ESL writers.
Chapter 4
Feedback Orchestration: Supporting Reflection and Awareness in Revision
4.1 Introduction
Writing is rewriting. Revision has been established as one of the most important and complicated components in the writing process [21, 23]. Effective revisions rely on high-quality feedback, and the writer is expected to develop a revision plan by addressing those issues in some order [23]. Multiple revisions are often necessary to fix writing errors locally and to improve content or structure globally. Triggered by different types of feed-back, each revision consists of a series of micro-tasks, such as adding examples to support a claim or fixing grammatical errors. However, novice writers tend to focus mainly on sur-face revisions because they struggle with developing good strategies to deal with structural problems [66, 18, 23].
This work addresses the challenge of supporting novice writers to take advantage of feedback and solve writing issues in their revisions. Previous research has used Online Feedback Exchange systems [24] to collect effective feedback to help designers revise their creative work. While most research focused on improving the diversity and quality of feedback [72, ?, 46, 27, 75], the difficulty in integrating feedback into revisions effi-ciently and effectively has been largely neglected. Recent studies have noticed the critical
gap between feedback provided and the subsequent performance [?, 25, 74] and start to develop approaches that help novices enhance the understanding of feedback [25, 74]. To fill the gap, our research explore a new way that facilitates novice writers to reflect on their writing with feedback and guides them to select good strategies to integrate feedback in revisions.
A formative study was conducted to understand how novice writers integrate feedback into revisions. The first finding was that inexperienced people usually revised in an un-structured way. For example, they used the most intuitive but inefficient approach to edit from beginning to end. They tended to focus on solving easier or specific local issues such as grammar and mechanics instead of more difficult ones such as argumentation and organization. The second finding was that writers having varying background and expe-rience developed distinct revision strategies. Our study and research both suggest that novices need more supports to enhance their abilities and awareness as a writer to achieve structural revision, which is a pattern that experienced writers usually perform [31].
To support effective revisions, we propose Feedback Orchestration, which uses a rhetorical structure to guide novice writers interpret and implement feedback in a flex-ible revision workflow (see Figure 4.1). It enables writers to think and revise structurally based on structured feedback. Moreover, we present ReviseO, a system that implements the concept of Feedback Orchestration. It provides a standard writing rubric to help writ-ers assess the writing goals with multiple criteria. Each feedback is classified into three categories by an automatic method. The system also enables three types of revision work-flows (high-to-low, low-to-high, and all) for supporting writers to solve writing issues by utilizing categorized feedback.
In this work, we ran a field study that allowed writers to revise self-written articles with our system. We used a within-subjects study design to evaluate the perceived useful-ness of the system, and also explore how three standard feedback presentation strategies (high-to-low, low-to-high, and all) affect revision behaviors and task performance. In the study, twelve self-motivated English as a Second Language (ESL) writers were recruited to revise three independent essays of 300-400 words based on structured expert feedback
Author Figure 4.1: The overview of feedback orchestration. It guides writers to integrate feedback into revisions by a categorized structure.
using three different feedback presentation strategies. We present the findings based on analysis of data with interview transcripts and suggest design implications of online feed-back system for writing supports.
The contributions of this work are:
• The first formative study contributing empirical understandings of writers’ revision behaviors while dealing with unstructured feedback in revisions.
• The concept of Feedback Orchestration that uses a rhetorical structure to guide ef-fective revision process by orchestrating feedback of different type.
• The system ReviseO, which supports three standard revision workflows for helping writers resolve issues by receiving expert feedback structured by a standard rubric.
• Results showed that structured feedback helped writers identify weaknesses and promoted reflection. Furthermore, novice writers who experienced three types of workflow increased their awareness and developed good strategies.
The insights obtained from this work help further researchers of related fields under-stand writers’ revision behaviors while using different feedback strategies and contribute to the design of a flexible feedback framework for writing support.