not feel lost in handling such a complicated task of revising a draft. The provision of choice rather than an arbitrary assignment of goals, according to past study findings, could make learners more committed to their own goals. Right below the area where they circled the goals, a table was provided to let learners write down strategies they intended to use in the coming up revision. Learners were given about fifteen minutes to plan their revision with the help of this worksheet. And later they started revising their drafts for a total of thirty minutes. At the end of the class period, about five minutes were allotted for them to check on the original strategy list for what was and was not done successfully. They then performed the same self-assessment the AfL Only Group did. Finally, they completed a five-item goal commitment questionnaire.
The AfL + Goal + Strategy Group went through the same procedures as the AfL + Goal Group did. The only difference was the provision of a strategy list at the time when they set goals and planned for revision. The list simply provided brief verbal descriptions of revision strategies discussed in the feedback lessons given to all three groups. According to studies comparing learning goals and performance goals, the provision of strategies, in addition to setting goals, is crucial when the tasks are complicated and involve learning.
Data Analysis and Results
Inter-rater reliability was first calculated between the two independent raters. As shown in Table 1, their ratings were all positively correlated at significant levels except for the second revision of the AfL + Goal Group. After discussions with statisticians, it was decided that the figures were generally acceptable and further rating may contaminate the results. However, cautions should be exercised when interpreting results regarding the second revision of the AfL + Goal Group.
Table 1. Inter-rater Reliability
Groups Task Statistics Draft Revision
AfL Only 1st Pearson Correlation Coefficient r 0.640 0.578
Significance (two-tailed) p (0.000) (0.000) 2nd Pearson Correlation Coefficient 0.622 0.506
Significance (two-tailed) (0.000) (0.002) 3rd Pearson Correlation Coefficient 0.860 0.834 Significance (two-tailed) (0.000) (0.000) AfL + Goal 1st Pearson Correlation Coefficient 0.576 0.784
Significance (two-tailed) (0.000) (0.000) 2nd Pearson Correlation Coefficient 0.752 0.343
Significance (two-tailed) (0.000)
(0.093)
3rd Pearson Correlation Coefficient 0.958 0.946 Significance (two-tailed) (0.000) (0.000) AfL + Goal 1st Pearson Correlation Coefficient 0.789 0.882+ Strategy Significance (two-tailed) (0.000) (0.000) 2nd Pearson Correlation Coefficient 0.943 0.963
Significance (two-tailed) (0.000) (0.000) 3rd Pearson Correlation Coefficient 0.563 0.397 Significance (two-tailed) (0.001) (0.044)
The second examination was on the first draft between the three groups. If the three groups differed significantly from each other, their draft scores would be used as covariates in further comparisons. Results of analysis of variance on the first draft were shown in Table 2 and Table 3.
Table 2. Summary of ANOVA on First Draft Among Three Groups
df F p η2 obs. power
Groups 2 9.21*** .000 .150 .974
Errors 104 (3.64)
25
Table 3. Follow-up Multiple Comparisons Using Tukey Tests (I) Group (J) Group (I-J) p 95% CI AfL Only AfL +
Goal -0.51 0.484 [-1.56, 0.54]
AfL + Goal + Strategy
1.43**
0.006
[0.35, 2.50]AfL +
Goal AfL Only 0.51 0.484 [-0.54, 1.56]
AfL + Goal + Strategy
1.94***
0.000
[0.83, 3.05]AfL + Goal + Strategy
AfL Only -1.43**
0.006
[-2.50, -0.35]AfL +
Goal -1.94***
0.000
[-3.05, -0.83]The above information indicated that the three groups, as measured by their first drafts, were not the same. The third group, AfL + Goal + Strategy, scored
significantly lower than the first group, AfL Only, by 1.43, and further lower than the second group, AfL + Goal + Strategy, by 1.94. The differences between the AfL + Goal + Strategy Group and the other two were statistically significant. This difference was taken into consideration in follow-up analyses.
Descriptive statistics, i.e, the average holistic scores between the two raters of each group and each piece of writing, are shown in Table 4. The same information was depicted in Figure 3 as three lines representing the changes of scores over time for each of the three groups. As illustrated in the line graph, all groups performed better in revisions than in the relevant drafts. AfL Only Group improved from the first draft to the first revision, and further from the second draft to the second revision; but
there was a sharp drop in the third draft before the score picked up in the third revision. For the AfL + Goal Group, they performed better than the other groups in the first task. Their second draft and revision were relatively the same as the scores they obtained in the first round. But the drop in the third round was more obvious than the AfL Only Group. The AfL + Goal + Strategy Group, however, started much lower than the other two groups, but their improvement over the three tasks was steady.
Eventually in the third round, they outperformed the other groups and obtained the highest scores among all three.
Table 4. Descriptive Statistics of All Drafts and Revisions by All Three Groups
AfL Only (n=40) AfL + Goal (n=35) AfL + Goal + Strategy (n=33) M SD 95% CI M SD 95% CI M SD 95% CI
1st Draft 8.16 1.74 [7.60, 8.72] 8.67 1.69 [8.09, 9.25] 6.73 2.29 [5.91, 7.56]
Revision 9.36 1.76 [8.78, 9.95] 9.66 1.79 [9.01, 10.32] 8.54 2.09 [7.73, 9.34]
2nd Draft 9.50 1.29 [9.07, 9.93] 8.73 1.44 [8.22, 9.24] 7.61 2.30 [6.78, 8.44]
Revision 10.23 1.30 [9.78, 10.68] 9.60 1.08 [9.15, 10.05] 8.98 2.17 [8.19, 9.78]
3rd Draft 7.81 2.25 [7.03, 8.59] 6.65 2.56 [5.71, 7.58] 8.43 1.43 [7.89, 8.98]
Revision 8.28 2.01 [7.63, 8.93] 7.89 2.43 [6.84, 8.94] 9.56 1.34 [9.02, 10.10]
27
Figure 3. Line Graph of Writing Scores for Three Tasks from Three Groups
6 6.5 7 7.5 8 8.5 9 9.5 10 10.5
初稿1 修改稿1 初稿2 修改稿2 初稿3 修改稿3
班級1 班級2 班級3
For research question one on whether learner revisions improved from drafts, each revision score was subtracted by its associated draft score. Descriptive statistics of the resulted figures are presented in Table 5 with means, standard deviations, and 95% confidence intervals. As shown, all the means are positive and none of the range of confidence intervals include zero, which indicated that the improvement on all writings for all groups was statistically significant.
Table 5. Descriptive Statistics of Draft-to-Revision Improvement of Three Groups on Three Tasks
1st Writing 2nd Writing 3rd Writing
Groups M SD 95% CI M SD 95% CI M SD 95% CI
AfL Only 1.22 0.83 [0.94, 1.49] 0.68 0.7 [0.43, 0.92] 0.62 0.71 [0.37, 0.86]
AfL + Goal 1.18 1.14 [0.76, 1.60] 1 0.82 [0.66, 1.34] 0.8 1.19 [0.29, 1.32]
AfL + Goal
+ Strategy 1.71 1.65 [1.08, 2.35] 1.37 1.83 [0.70, 2.04] 1.04 0.87 [0.69, 1.39]
The next question to ask is whether the improvements among the three groups differ in magnitude. The null hypothesis is that the three groups did not differ in terms of levels of improvement. Before comparing the three groups, test of the homogeneity of regression coefficients was conducted (results shown in Table 6). It was found out that the interactive effect between independent variable (Group) and covariate (scores of the first draft) was not significant (all p values are larger than 0.05), indicating that the regression coefficients were homogeneous, which renders justification for the follow-up ANCOVA test.
Table 6. Summary of the Test of Regression Coefficient Homogeneity
1st Writing 2nd Writing 3rd Writing
Source of Variance
df F p df F p df F p
Scores of 1st Draft 2 2.18 0.119 2 2.03 0.137 2 0.12 0.890
Group 1 10.80** 0.001 1 0.19 0.664 1 3.97* 0.050
Scores of 1st Draft ×
Group 2 1.96
0.147
2 1.360.263
2 0.170.841
Within Group Errors 90 (1.26) 83 (0.93) 77 (0.82)
29
ANCOVA analyses were conducted on the improvement between drafts and revisions for all three groups in all three tasks. The results, as shown in Table 7, indicated that for all three tasks, the p values are above the significance level of 0.05.
That is to say, looking at each revision separately, the three groups did not differ from each other in terms of the levels of improvement.
Table 7. Summary of ANCOVA on Improvement among Three Groups in Three Tasks
ANCOVA Summary on the 1st Writing Source of
Variance
df F p η
2obs. power
Score of First
Draft 1 14.34*** 0.000 0.135 0.963
Group 2 0.22
0.803
0.005 0.084Within Group
Errors 92 (1.29)
ANCOVA Summary on the 2nd Writing Source of
Variance
df F p η
2obs. power
Score of First
Draft 1 0.51 0.479 0.006 0.108
Group 2 1.62
0.203
0.037 0.334Within Group
Errors 85 (0.94)
ANCOVA Summary on the 3rd Writing Source of
Variance
df F p η
2obs. power
Score of First
Draft 1 3.79 0.055 0.046 0.485
Group 2 0.84
0.437
0.021 0.189Within Group
Errors 79 (0.81)
Research question three is concerned about learners’ performance on drafts and revisions over time, that is, comparisons among all three drafts and among all three revisions. First, repeated measure ANOVA was conducted on three drafts. As shown in Table 8, the difference among three groups was not significant. That is to say, none of the three groups improve more than others from draft to draft. Results of follow-up
multiple comparisons using Bonferroni tests were shown in Table 9. For the groups, scores on the second draft differed from the first and the third ones.
Table 8. ANOVA Summary on Three Drafts
Source of Variance df F p η2 obs. power
Group 2 2.64 .077 .057 .512
Within Group Errors 87 (6.90)
Table 9. Multiple Comparison on Three Drafts Using Bonferroni Test (I)Writing (J)Writing Difference of Means
(I‐J) p Difference 95% CI
1 2 ‐0.73** 0.001 [‐1.21, ‐0.24]
3 0.35 0.350 [‐0.19, 0.89]
2 1 0.73** 0.001 [0.24, 1.21]
3 1.08*** 0.000 [0.52, 1.63]
3 1 ‐0.35 0.350 [‐0.89, 0.19]
2 ‐1.08*** 0.000 [‐1.63, ‐0.52]
The second part of research question three is concerned about the difference among three revisions in three groups. Test of homogeneity, as shown on Table 10, indicated that the interactive effect between independent variable (Group) and covariate (score of first draft) was significant (p value smaller than 0.05). Therefore, the estimated regression coefficients were calculated and presented in Table 11 for use in follow-up analyses.
Table 10. Summary of the Test of Regression Coefficient Homogeneity on All Revision Scores
Source of Variance
df F p
Group 2 6.56**
.002
Score of 1st Draft 1 98.39***
.000
Group x Score of 1st Draft 2 4.64*.013
Within Group Errors 69 (2.72)
31
Table 11. Estimated regression coefficients for first draft (X) and revisions (Y)
Writings
Results of repeated measure ANOVA on three revisions among three groups, as shown in Table 12, suggested that the three groups differed in their three revision scores. Multiple comparisons in Table 13 showed that the AfL + Goal + Strategy Group, when all three rounds of writing were taken into consideration, performed better than the AfL Only Group in revisions, and the AfL Only Group in turn performed better than the AfL + Goal Group.
Table 12. ANOVA Summary on Three Revisions
Source of Variance
df F p η
2obs. power
Group 2 155.65***
.000
.812 1.000Within Group
Errors 72 (1.07)
Table 13. Multiple Comparison on Revisions by Groups Using Bonferroni Tests
(I)Group (J)Group Mean Difference
(I-J) p Difference 95% CI
Multiple comparisons were performed among the three rounds of writing, as shown in Table 14. When all three groups were assessed together, they performed best in the second revision, followed by the third and then the first.
Table 14. Multiple Comparison on Revisions by Writings Using Bonferroni Tests
(I)Writing (J)Writing Mean Difference
(I‐J) p 95% CI of the Difference 1st 2nd ‐4.56*** 0.000 ‐4.80 ‐4.32
3rd ‐2.01*** 0.000 ‐2.33 ‐1.69 2nd 1st 4.56*** 0.000 4.32 4.80
3rd 2.55*** 0.000 2.23 2.87 3rd 1st 2.01*** 0.000 1.69 2.33 2nd ‐2.55*** 0.000 ‐2.87 ‐2.23