The EVRR test score was used for assessing the learning outcome in our study. Table 2 shows the descriptive statistics results. We conducted the repeated measures analysis of variance for learners with four different STM capacities (Quadrants 1 to 4), with LCR types as independent variables and scores measured from the EVRR test as dependent variables. The Mauchly’s test of sphericity for the homogeneous test was conducted before the repeated measures analysis of variance. An important result in Mauchly’s test of sphericity indicates that
the covariance of the three within-subject variables (recognition score, recall score and average score) are not homogeneous. Thus, an adjusted degree of freedom statistic provided by the Greenhouse-Geisser correctional formula was used to do the repeated measures analysis of variance (Hair, Tatham, Anderson & Black, 1998). Otherwise, if the result is not significant, based on sphericity’s assumption, no adjustment to the degree of freedom is needed (Hair et al, 1998).
A significant result after the repeated measures analysis of variance indicates that the mean scores of the four different types (A, B, C and D) are not equal. In such a case, a post-treatment pair-wise comparison is used to compare the mean scores of the four types. Conversely, a result that is not significant indicates that the mean scores of the four different types are equal.
Based on the result shown in Table 3, the pair-wise comparison was not needed. Table 4 shows the analysis results with respect to the eight hypotheses.
Table 2. Descriptive statistics of the four research hypotheses
* The mean scores are presented as percentages
Table 3. Analysis results of repeated measures analysis of variance
Table 4. Analysis results of the eight research hypotheses
The analysis results for H1 in Table 4 show the following:
(1) that the EVRR scores of the learners who were presented with information as LCR type B (p = 0.000) or type C (p = 0.000) were significantly better than the scores of the learners who were presented with information as LCR type A in the recognition test;
(2) that the EVRR scores of the learners who received information as either LCR type B (p
= 0.006) or type C (p = 0.011) were significantly better than the scores of the learners who received information as LCR type A in the recall test. The same result also appeared in average
scores for type B (p = 0.000) and type C (p = 0.000). Therefore, we can conclude that the H1 proposed in this research is accepted. This implies that learners with higher verbal ability and higher visual ability can benefit from learning content that contains either written annotation or pictorial annotation.
For H2, the EVRR scores of the learners who received information as LCR type C were better than the scores of the learners who received information as LCR type A (p = 0.000) in the recognition test. Moreover, the learners who received information as LCR type C exhibited better EVRR scores than the learners who received information as LCR type A (p = 0.000) in the recall test. Average scores of the learners who received information as LCR type C were also better than those of the learners who received information as LCR type A (p = 0.000). Therefore, we can conclude that the H2 is accepted. This implies that learners with higher visual ability can benefit from learning content that contains pictorial annotation.
For H3, Table 4 shows that there is no significant difference in these three scores among the learners who received information as LCR types A, B and C. Therefore, we can conclude that the H3 is also accepted. This implies that learners with lower verbal ability and lower visual ability do not benefit from learning content containing either written annotation or pictorial annotation.
For H4 shown in Table 4, it is evident that scores of the learners who received information as LCR type B are better than those of learners who received information as LCR types A (p = 0.000) and C (p = 0.005) in the recognition test. However, this is not the case in the recall test. Therefore, we can conclude that the H4 is only partially accepted. This implies that learners with higher verbal ability benefit from learning content containing written annotation with regard to recognition.
For H5, we found that recognition scores of the learners who received information as LCR type D were better than those of learners who received information as LCR type A (p = 0.000) and that recall scores of the learners who received information as LCR type D were also better than those of learners who received information as LCR type A (p = 0.001). Also, average scores of the learners who received information as LCR type D were better than those of learners who received information as LCR type A (p = 0.000). Therefore, we can conclude that the H5 proposed in this research is accepted. This implies that learners with higher verbal ability and
higher visual ability can benefit from learning content containing combined written annotation and pictorial annotation.
For H6, recognition scores of the learners who received information as LCR type D were better than those of learners who received information as LCR type A (p = 0.000). However, in the recall test, scores of the learners who received information as LCR type D were not significantly better than those of learners who received information as LCR type A. Average scores of learners who received information as LCR type D were better than those of learners who received information as LCR type A (p = 0.002). Therefore, we can conclude that the H6 is only partially accepted. This implies that learners with higher visual ability can only somewhat benefit from learning content containing combined written annotation and pictorial annotation with regard to recognition.
For H7, Table 4 shows that there is no significant difference in the scores of learners who received information as LCR types A and D in both the recognition and recall tests. Therefore, we can conclude that H7 is also accepted. This implies that learners with lower verbal ability and lower visual ability cannot benefit from learning content containing both written annotation and pictorial annotation.
Finally, for H8, recognition scores of the learners who received information as LCR type D are better than the scores of learners delivered with LCR type A (p = 0.000). However, in the recall test, scores of the learners who received information as LCR type D were not significantly better than those of learners who received information as LCR type A. Average scores of the learners who received information as LCR type D were also better than the learners who received information as LCR type A (p = 0.000). Therefore, we can conclude that the H8 is only partially accepted. This implies that learners with higher verbal ability can only somewhat benefit from learning content containing both written annotation and pictorial annotation with regard to recognition.
In this research, focus group interviews were also conducted to acquire qualitative evidence to support the results from the quantitative experiment. Therefore, eight participants were selected from 156 college students by using the extreme or deviant case sampling method.
The background information of these participants is shown in Table 5. For the sake of privacy, Table 5 uses a coding scheme (SB-Quadrant-EDCS) to replace the real names of the students. SB
means subject, Quadrant is the number of STM ability Quadrant, and W and B of EDCS are the worst and the best results of the EVRR test.
Table 5. Participants’ background information in focus group interview STM ability Average Score of LCR type Code Age SB refers to the subjects who participated in the mobile learning experiment.
Number refers to the quadrant to which the subject belongs.
W refers to the worst EVRR test score.
B refers to the best EVRR test score.
Focus group interview results:
Transcripts of interviews with the Q1 students (SB-1-B and SB-1-W) revealed that written or pictorial annotation can provide better learning outcomes (SB-1-B-83) than no annotation (SB-1-W-62). Furthermore, written annotation or pictorial annotation can help learners learn and remember more English vocabulary items (SB-1-W-17). These are the qualitative findings to support H1.
Transcripts of interviews with the Q2 students show that pictorial annotation can provide better learning effects (SB-2-B-50), more cues (SB-2-B-52), more attractive content (SB-2-W-85) and better representation than words (SB-2-B-81). Finally, pictorial annotation is helpful in remembering more English vocabulary items (SB-2-B-9). Interview transcripts also revealed that these students disagree that written annotation helps them understand the meaning (SB-2-B-38), even though they agree that written annotation can be somewhat useful. These qualitative findings therefore support H2.
Transcripts of interviews with the Q3 students show that the feedback on the four LCR types are similar (SB-3-B-68). When student SB-3-W saw a lot written annotation or pictorial
annotation, the student felt irritated and unable to concentrate on learning (SB-3-W-78). This may be the result of the learner’s insufficient STM . For learners in Q3 (those with lower verbal ability and lower visual ability), more annotation causes a higher cognitive load in their STM, according to the Cognitive Load Theory, and prevents those learners from learning more. From these two findings, it is clear that more written annotation or pictorial annotation causes frustration and increased negativity to Q3 learners. This phenomenon is supported by the transcripts, which show that even though written annotation can help learners memorize vocabulary in a more organized way and understand how to remember vocabulary (SB-3-B-46), learning outcomes in the four LCR types do not vary on a large scale (SB-3-B-25). Another conflicting finding is that content with no annotation (LCR type A) seems to be the most difficult for memorizing the English vocabularies (SB-3-B-27). So, there are qualitative findings that support H3, but there are also some conflicting findings.
Transcripts of interviews with the Q4 students show that written annotation can provide meaningful information (SB-4-B-66). For example, student SB-4-W did not agree with the benefits of written annotation (SB-4-W-72). So, there are not only some qualitative findings that support the hypothesis that Q4 learners have better learning outcomes in the case of written annotation (LCR type B), but there are also some conflicting findings. However, according to the Levels of Processing Theory, if Q4 learners provide no positive feedback with regard to written annotation (LCR type B), they cannot learn and memorize better than students in the baseline group (LCR type A). However, from the transcripts, it is evident that they all agree that written annotation can help them remember more English vocabulary items (SB-4-B-19). This means that Q4 learners could achieve better learning outcomes in the fit cases with regard to learning and memorizing more English vocabulary items. In the next section, these conflicting findings based on feedback made by students in SB-4-W will be discussed further. In the meantime, due to the feedback of students in SB-4-W, the qualitative findings seem to only partially support H4.
Transcripts of interviews with the Q1 (SB-1-B and SB-1-W) students show that the written annotation plus pictorial annotation can provide better learning outcomes (SB-1-B-56).
Furthermore, written or pictorial annotation can help learners learn and remember more English words (SB-1-B-40). These qualitative findings seem to support H5.
Transcripts of interviews with the Q2 (SB-2-B and SB-2-W) students show that pictorial annotation can provide better learning outcomes and cues, and that they are more appealing.
Besides, pictorial annotations are helpful in remembering more English vocabulary items.
However, there are no qualitative findings to show that combined annotations provide better learning outcomes and cues, nor that they are more appealing. The interviews also do not indicate that combined annotations help learners remember more English words. Therefore, we can conclude that H6 is not supported.
Transcripts of interviews with the Q3 (SB-3-B and SB-3-W) students show similar feedback from the four LCR types (SB-3-B-68). The transcripts also reveal that even though written annotations could help learners memorize vocabulary items in a more organized way and understand how to remember vocabulary items (SB-3-B-46), learning outcomes in the four LCR types do not vary on a large scale (SB-3-B-25), and baseline group content (LCR type A) would be the most difficult to memorize (SB-3-B-27). So, the qualitative findings do not seem to support H7.
Transcripts of interviews with the Q4 (SB-4-B and SB-4-W) students show that there are no qualitative findings to support the fact that combined annotations provide students with better learning outcomes with regard to learning and memorizing compared to the results of students in the baseline group. Therefore, we can conclude that H8 is not supported.