Analysis of MY Access Students’ Essay Submissions

resulted in a questionnaire return rate of 36 students (62.1%). Eight of those students were interviewed in depth about their experiences using MY Access. Whereas only 12.0% of the students did not complete any essay assignments, 37.9% did not complete questionnaires.

This is a problem which may be encountered in a non-teacher-directed autonomous learning experiment, with no credit being given for improvement in scores beyond what the program offers and which leaves students to explore on their own and proceed as far as they feel like.

In this experiment, the essay data on the MY Access program, and the questionnaire data were derived from different sources. This posed problems for matching essay data with questionnaire responses, since matching pairs of essay and questionnaire data could be found for only 14 students because not all students wrote their name or student number on the questionnaire forms.

4.2 Analysis of MY Access Students’ Essay Submissions

The most important question to be answered in this study is “Did students’ essay scores increase between the first submissions and subsequent submissions?” Obviously, if

improvement was noted for most students, then a case could be made that the program was useful in increasing their writing skills.

To survey general trends in student essay scores, student holistic scores were examined from MY Access data. First, a general summary report on holistic scores with no MY Access essay prompt filters applied was obtained. This is presented in Figure 4.2. This might suggest that students’ scores tended to rise then fall from beginning the program to the end. However, this data includes all five essays. Only two students made multiple revised submissions (one student submitted five revisions and another student made 12; see Table 4.4). Also, the graph does not show how many students were submitting revised essays. Therefore such reports cannot be said to show reliably that students were actually improving on the program.

Figure 4.2

Difference in Average Holistic Scores between Revisions 1 and 13 of all Five Essay Prompts

Source: Graph generated by MY Access

The next step was to apply the MY Access essay prompt filter in order to examine individual essays for signs of student progress. MY Access allows this so the data for the practice essay Fantastic Day were graphed and presented in Figure 4.3.

Figure 4.3

Difference in Average Holistic Scores between Revisions 1 and 2 of Practice Essay Prompt

“Fantastic Day”

Source: Graph generated by MY Access

The initial impression from this graph is that mean scores improved from 2.4 to 2.9.

However, the number of students who submitted the first essay draft was 29 with only three students making a revised submission. Clearly, there are no conclusions to be drawn from the mean scores on the practice essay since the number of students for each mean was a different group.

The next four essays (Facing Your Fears, Feeling Proud, Global Language and School Rules: Eating in the Classroom) were then investigated and similar MY Access graphs produced for student results comparing first and last submissions. Once again, the same problem was encountered. Most students made only one attempt at an essay topic. For most essays, only a small number of students submitted a second or more draft of the same essay (see Table 4.4). Therefore, comparing means from holistic scores was also not meaningful for these essays.

MY Access data also provided times and dates of submission. About half of the revised versions were resubmitted on the same day shortly after submitting the first one. Half of the revised versions of essays were resubmitted days or weeks later. This suggests that the students who resubmitted were evenly divided into (1) those who responded to the program’s immediate feedback and (2) those who perhaps saw value in putting the essay away for a few days and then rewriting, which is something writing teachers often suggest.

Looking through the data for exactly how many students submitted revised essays of the same essay prompt revealed that 18 out of the 51 students (35.2%) did this (Table 4.4).

The number of resubmitted essays totaled 34 (including one student who resubmitted 12 times on a single essay prompt). Most (12 of the 18 students) resubmitted on only one essay topic and did this only once.

Table 4.4

Number of Times Essays were Revised by 18 students on 1, 2 and 3 Essay Prompts.

Single revision 2 times

1 essay revised 11 students 1 student

2 essays revised 4 students 1 student

3 essays revised 1 student >>> Same student

For autonomous learning, the implication here is that if given four essay prompts, a majority of students are more likely to move on to the next essay prompt than to rewrite an essay. Among those who elect to rewrite the same essay, almost all will only do it once.

Did mean scores improve for students who made multiple submissions on the same essay? To answer this question, mean differences in holistic scores were compared between the first submission and the last submission, as shown in Table 4.5.

Table 4.5

Mean Differences in Holistic Scores Compared between the First Submission and the Last Submission with Numbers of Students

First submission Last submission

Practice Essay 2.4 (29) 2.9 (3)

Facing Your Fears 3.4 (29) 4.0 (6)

Feeling Good 3.5 (22) 3.8 (10)

Global Language 3.3 (23) 3.3 (4)

School Rules 3.5 (19) -

Overall 3.15 3.5

Comparison of mean score differences might suggest that there appears to be some gain in mean scores over the four tests (3.15 to 3.5). However, the revisions were submitted by only a minority of 18 students (33.3%).

It was therefore decided to extract only the students who had made multiple attempts at each essay. Table 4.6 lists mean scores for each essay submitted by the 18 students who submitted a first essay and a revision. Investigating those students’ scores manually on MY Access, in all cases of essays for which data is available, students made no gains in holistic scores based on first and last submissions. Data on early submissions appeared to be difficult to obtain in some essays, notably Feeling Good where all 10 of the students’ first attempts were flagged as unscored. The reason for this is unclear.

Table 4.6

Mean Differences in Holistic Scores Compared between the First Submission and the Last Submission with Numbers of Students who Submitted Multiple Essays

First submission Last submission Practice Essay 2.4 (2)* 2.9 (3)**

Facing Your Fears 4.0 (6) 4.0 (6)

Feeling Good NS (10) 3.5 (10)***

Global Language 3.3 (4) 3.3 (4)

* one student’s first submission score not available

**Score from MY Access must be calculated on 3 students’ scores, not 2

***all 10 students’ first submission essays not scoreable (NS)

The conclusion here is that after generating holistic student data from MY Access through tables, bar charts or line graphs, the program administrator, the teacher and the even the student should question how the data has actually been derived. This is especially true for autonomous learning groups. A table might show that student mean scores for a group have risen, but this can be misleading if many of the students have only made one submission. This is especially relevant since the graphs generated by MY Access do not show the numbers of students who wrote each essay; to get this data, the student listings have to be scanned on the computer screen or on printouts.

Such misleading instances due to data simplification can be further compounded if students have made multiple submissions but on different essay prompts or topics. It was therefore found that relying on MY Access for data on overall autonomous student progress could be misleading. The overall test results may be valid if the students all wrote their essays in class under supervision, preferably according to a prescribed time limit. If using MY Access in an autonomous learning context, however, meaningful data needs to be manually extracted from the system with filters applied.