Procedure

The experimental procedure consisted of four different steps, as presented in Figure 4, and the whole process took place at the same computer lab. First, participants met with the researcher in the computer lab, where each participant was asked to fill out a background questionnaire. Then students were told that the objective of this experiment was to learn English vocabulary using a mobile phone and that they would go through a procedure with four steps as shown in Figure 4. This step took about 15 minutes.

Figure 4. Experimental procedures

In the second step, all participants were seated in front of individual computers in the lab for the STM ability test using the STM ability test system (Hsieh, 2006). The system design is based on Wright (1988) and Chen, Lee and Chen (2005), with some modifications to fit into our study. The system architecture of STM ability test and examples of written and pictorial content can be found in Appendix 3. There were 60 questions in this STM ability test, 30 questions for written materials and 30 for pictorial materials. Each question was presented on computer screen for 7 seconds and the participants were given 5 seconds to respond. It took 12 minutes (60 questions at 12 seconds each) in total for this step. Immediately after the 156 participants finished the STM ability test, the system recorded each participant’s STM ability which was a value converted to a standard normal distribution with a mean of 0 and a standard deviation of 1.

Based on these results, we then divided the students into four groups, with 61 in Quadrant 1, 36 in Quadrant 2, 30 in Quadrant 3 and 29 in Quadrant 4. For the validity of this working memory test, interested readers can refer to Chen, Lee and Chen (2005).

In the third step, every participant was then immediately assigned a mobile phone to learn the 24 English words delivered by SMS or MMS. All participants individually read the English words sent out by the researcher to their mobile phones. Every participant received the same 24 English words, with 6 words for every representation type. To avoid the learning effect of representation types presented in a fixed order, we adopted an LS-4 design (Table 1) to deliver these four representation types for participants in every group. For example, participant P1 in group 1 received 6 words randomly selected from 24 words represented in type A format, then 6 words randomly selected from the remaining 18 (represented in type B format), then 6 words randomly selected from the remaining 12 (represented in type C format) and lastly the remaining 6 words (represented in type D format). However, participant P2 in group 1 received 6 words randomly selected from 24 words (represented in type B format), then 6 words randomly selected from the remaining 18 (represented in type D format), and then 6 words randomly selected from the remaining 12 (represented in type A format) and lastly the remaining 6 words (represented in type C format). And so it went for all participants. The same procedure was applied for group 2, group 3 and group 4. The average time set for learning one English word in the majority of previous research of L2 experiments was about 2 minutes (Nikolova, 2002; Jones, 2004). Therefore, we set 50 minutes for the learners to learn the 24 English words in our experiment.

Table 1. The LS-4 design

T1 T2 T3 T4

P1 A B C D

P2 B D A C

P3 C A D B

P₄ D C B A

.. .. .. .. ..

P_n .. .. .. ..

Pn: Number of participants in a group

Tn: Treatment of the English Vocabulary corpuses

A = LCR type A; B = LCR type B; C = LCR type C; D = LCR type D 7.6 Data collection instruments

In the fourth step, after viewing the content to learn 24 English words, all participants were immediately asked to sit for the English Vocabulary Recognition and Recall (EVRR) test to assess their English vocabulary learning performance. These recognition and recall tests are often used to examine learners’ English vocabulary knowledge (Al-Seghayer, 2001). However, test and measurement studies indicate that these two forms of testing are quite different and demand separate processing strategies (Cariana & Lee, 2001; Jonassen & Tessmer, 1996). For example, recognition tests usually involve multiple-choice activities in which learners select or guess the correct response from the given alternatives. Such tests may strengthen any existing memory traces (McDaniel & Mason, 1985). Recall, on the other hand, demands the production of responses from memory. It is more difficult than recognition because learners must search for the correct response within their mental representation of the newly experienced information (Cariana & Lee, 2001; Glover, 1989; McDaniel & Mason, 1985).

Figure 5 and Figure 6 show examples of a recognition test item and a recall test item, respectively, in our study. Participants spent approximately 15 minutes completing the EVRR test.

Figure 5. Example of a recognition test item

Figure 6. Example of a recall test item

The final step in our study was the focus group interview, for which 8 participants were selected. To effectively evaluate our hypotheses, the interview questions focused on our proposed hypotheses. The following is a list of questions used to assess what learners learned from participating in the English vocabulary learning experiment. During the interviews, learners were asked three modified open-ended questions that were originally proposed by Al-Seghayer (2001):

¾ Question 1: Which one of the four LCR types is best for helping you to learn and memorize the English vocabulary in the experiment?

¾ Question 2: Which one of the four LCR types can provide better meaning about English words for you in the experiment?

¾ Question 3: What are the good features in this kind of mobile learning environment that help you, as a language learner, to effectively learn English vocabulary?

The open-ended questions were used to allow more freedom of responses, to elicit more information from the participants and to check the accuracy of the quantitative results in the mobile learning experiment. The focus group transcript records were reviewed by the moderator and teaching assistants immediately after the interview. Appendix 4 is a table of transcript records from the focus group interview, which will be cited in the following analysis.

3. Results

The EVRR test score was used for assessing the learning outcome in our study. Table 2 shows the descriptive statistics results. We conducted the repeated measures analysis of variance for learners with four different STM capacities (Quadrants 1 to 4), with LCR types as independent variables and scores measured from the EVRR test as dependent variables. The Mauchly’s test of sphericity for the homogeneous test was conducted before the repeated measures analysis of variance. An important result in Mauchly’s test of sphericity indicates that

the covariance of the three within-subject variables (recognition score, recall score and average score) are not homogeneous. Thus, an adjusted degree of freedom statistic provided by the Greenhouse-Geisser correctional formula was used to do the repeated measures analysis of variance (Hair, Tatham, Anderson & Black, 1998). Otherwise, if the result is not significant, based on sphericity’s assumption, no adjustment to the degree of freedom is needed (Hair et al, 1998).

A significant result after the repeated measures analysis of variance indicates that the mean scores of the four different types (A, B, C and D) are not equal. In such a case, a post-treatment pair-wise comparison is used to compare the mean scores of the four types. Conversely, a result that is not significant indicates that the mean scores of the four different types are equal.

Based on the result shown in Table 3, the pair-wise comparison was not needed. Table 4 shows the analysis results with respect to the eight hypotheses.

Table 2. Descriptive statistics of the four research hypotheses

* The mean scores are presented as percentages

Table 3. Analysis results of repeated measures analysis of variance

Table 4. Analysis results of the eight research hypotheses

The analysis results for H1 in Table 4 show the following:

(1) that the EVRR scores of the learners who were presented with information as LCR type B (p = 0.000) or type C (p = 0.000) were significantly better than the scores of the learners who were presented with information as LCR type A in the recognition test;

(2) that the EVRR scores of the learners who received information as either LCR type B (p

= 0.006) or type C (p = 0.011) were significantly better than the scores of the learners who received information as LCR type A in the recall test. The same result also appeared in average

scores for type B (p = 0.000) and type C (p = 0.000). Therefore, we can conclude that the H1 proposed in this research is accepted. This implies that learners with higher verbal ability and higher visual ability can benefit from learning content that contains either written annotation or pictorial annotation.

For H2, the EVRR scores of the learners who received information as LCR type C were better than the scores of the learners who received information as LCR type A (p = 0.000) in the recognition test. Moreover, the learners who received information as LCR type C exhibited better EVRR scores than the learners who received information as LCR type A (p = 0.000) in the recall test. Average scores of the learners who received information as LCR type C were also better than those of the learners who received information as LCR type A (p = 0.000). Therefore, we can conclude that the H2 is accepted. This implies that learners with higher visual ability can benefit from learning content that contains pictorial annotation.

For H3, Table 4 shows that there is no significant difference in these three scores among the learners who received information as LCR types A, B and C. Therefore, we can conclude that the H3 is also accepted. This implies that learners with lower verbal ability and lower visual ability do not benefit from learning content containing either written annotation or pictorial annotation.

For H4 shown in Table 4, it is evident that scores of the learners who received information as LCR type B are better than those of learners who received information as LCR types A (p = 0.000) and C (p = 0.005) in the recognition test. However, this is not the case in the recall test. Therefore, we can conclude that the H4 is only partially accepted. This implies that learners with higher verbal ability benefit from learning content containing written annotation with regard to recognition.

For H5, we found that recognition scores of the learners who received information as LCR type D were better than those of learners who received information as LCR type A (p = 0.000) and that recall scores of the learners who received information as LCR type D were also better than those of learners who received information as LCR type A (p = 0.001). Also, average scores of the learners who received information as LCR type D were better than those of learners who received information as LCR type A (p = 0.000). Therefore, we can conclude that the H5 proposed in this research is accepted. This implies that learners with higher verbal ability and

higher visual ability can benefit from learning content containing combined written annotation and pictorial annotation.

For H6, recognition scores of the learners who received information as LCR type D were better than those of learners who received information as LCR type A (p = 0.000). However, in the recall test, scores of the learners who received information as LCR type D were not significantly better than those of learners who received information as LCR type A. Average scores of learners who received information as LCR type D were better than those of learners who received information as LCR type A (p = 0.002). Therefore, we can conclude that the H6 is only partially accepted. This implies that learners with higher visual ability can only somewhat benefit from learning content containing combined written annotation and pictorial annotation with regard to recognition.

For H7, Table 4 shows that there is no significant difference in the scores of learners who received information as LCR types A and D in both the recognition and recall tests. Therefore, we can conclude that H7 is also accepted. This implies that learners with lower verbal ability and lower visual ability cannot benefit from learning content containing both written annotation and pictorial annotation.

Finally, for H8, recognition scores of the learners who received information as LCR type D are better than the scores of learners delivered with LCR type A (p = 0.000). However, in the recall test, scores of the learners who received information as LCR type D were not significantly better than those of learners who received information as LCR type A. Average scores of the learners who received information as LCR type D were also better than the learners who received information as LCR type A (p = 0.000). Therefore, we can conclude that the H8 is only partially accepted. This implies that learners with higher verbal ability can only somewhat benefit from learning content containing both written annotation and pictorial annotation with regard to recognition.

In this research, focus group interviews were also conducted to acquire qualitative evidence to support the results from the quantitative experiment. Therefore, eight participants were selected from 156 college students by using the extreme or deviant case sampling method.

The background information of these participants is shown in Table 5. For the sake of privacy, Table 5 uses a coding scheme (SB-Quadrant-EDCS) to replace the real names of the students. SB

means subject, Quadrant is the number of STM ability Quadrant, and W and B of EDCS are the worst and the best results of the EVRR test.

Table 5. Participants’ background information in focus group interview STM ability Average Score of LCR type Code Age SB refers to the subjects who participated in the mobile learning experiment.

Number refers to the quadrant to which the subject belongs.

W refers to the worst EVRR test score.

B refers to the best EVRR test score.

Focus group interview results:

Transcripts of interviews with the Q1 students (SB-1-B and SB-1-W) revealed that written or pictorial annotation can provide better learning outcomes (SB-1-B-83) than no annotation (SB-1-W-62). Furthermore, written annotation or pictorial annotation can help learners learn and remember more English vocabulary items (SB-1-W-17). These are the qualitative findings to support H1.

Transcripts of interviews with the Q2 students show that pictorial annotation can provide better learning effects (SB-2-B-50), more cues (SB-2-B-52), more attractive content (SB-2-W-85) and better representation than words (SB-2-B-81). Finally, pictorial annotation is helpful in remembering more English vocabulary items (SB-2-B-9). Interview transcripts also revealed that these students disagree that written annotation helps them understand the meaning (SB-2-B-38), even though they agree that written annotation can be somewhat useful. These qualitative findings therefore support H2.

Transcripts of interviews with the Q3 students show that the feedback on the four LCR types are similar (SB-3-B-68). When student SB-3-W saw a lot written annotation or pictorial

annotation, the student felt irritated and unable to concentrate on learning (SB-3-W-78). This may be the result of the learner’s insufficient STM . For learners in Q3 (those with lower verbal ability and lower visual ability), more annotation causes a higher cognitive load in their STM, according to the Cognitive Load Theory, and prevents those learners from learning more. From these two findings, it is clear that more written annotation or pictorial annotation causes frustration and increased negativity to Q3 learners. This phenomenon is supported by the transcripts, which show that even though written annotation can help learners memorize vocabulary in a more organized way and understand how to remember vocabulary (SB-3-B-46), learning outcomes in the four LCR types do not vary on a large scale (SB-3-B-25). Another conflicting finding is that content with no annotation (LCR type A) seems to be the most difficult for memorizing the English vocabularies (SB-3-B-27). So, there are qualitative findings that support H3, but there are also some conflicting findings.

Transcripts of interviews with the Q4 students show that written annotation can provide meaningful information (SB-4-B-66). For example, student SB-4-W did not agree with the benefits of written annotation (SB-4-W-72). So, there are not only some qualitative findings that support the hypothesis that Q4 learners have better learning outcomes in the case of written annotation (LCR type B), but there are also some conflicting findings. However, according to the Levels of Processing Theory, if Q4 learners provide no positive feedback with regard to written annotation (LCR type B), they cannot learn and memorize better than students in the baseline group (LCR type A). However, from the transcripts, it is evident that they all agree that written annotation can help them remember more English vocabulary items (SB-4-B-19). This means that Q4 learners could achieve better learning outcomes in the fit cases with regard to learning and memorizing more English vocabulary items. In the next section, these conflicting findings based on feedback made by students in SB-4-W will be discussed further. In the meantime, due to the feedback of students in SB-4-W, the qualitative findings seem to only partially support H4.

Transcripts of interviews with the Q1 (SB-1-B and SB-1-W) students show that the written annotation plus pictorial annotation can provide better learning outcomes (SB-1-B-56).

Furthermore, written or pictorial annotation can help learners learn and remember more English words (SB-1-B-40). These qualitative findings seem to support H5.

Transcripts of interviews with the Q2 (SB-2-B and SB-2-W) students show that pictorial annotation can provide better learning outcomes and cues, and that they are more appealing.

Besides, pictorial annotations are helpful in remembering more English vocabulary items.

However, there are no qualitative findings to show that combined annotations provide better learning outcomes and cues, nor that they are more appealing. The interviews also do not indicate that combined annotations help learners remember more English words. Therefore, we can conclude that H6 is not supported.

Transcripts of interviews with the Q3 (SB-3-B and SB-3-W) students show similar feedback from the four LCR types (SB-3-B-68). The transcripts also reveal that even though written annotations could help learners memorize vocabulary items in a more organized way and understand how to remember vocabulary items (SB-3-B-46), learning outcomes in the four LCR types do not vary on a large scale (SB-3-B-25), and baseline group content (LCR type A) would be the most difficult to memorize (SB-3-B-27). So, the qualitative findings do not seem to support H7.

Transcripts of interviews with the Q4 (SB-4-B and SB-4-W) students show that there are no qualitative findings to support the fact that combined annotations provide students with better learning outcomes with regard to learning and memorizing compared to the results of students in the baseline group. Therefore, we can conclude that H8 is not supported.

4. Discussion

Quantitative results suggest that learners with lower verbal ability and higher visual ability (Q2) will benefit more from learning content with pictorial annotation than they will from learning content with no annotation. Qualitative findings also support this conclusion, indicating that pictorial annotation can help learning and memorizing more than no annotation. From both the quantitative and qualitative findings, the most suitable method to help these learners study in the mobile language learning environment is to provide them with more pictorial annotation and less written annotation. This result matches the finding that providing additional pictorial annotation in learning content can help learners with lower verbal ability and higher visual ability to learn better, because they have better skills for learning content presented in a visual form than they do for learning content presented in a verbal form (Geva & Ryan, 1993;

Harrington & Sawyer, 1992).

Results also suggest that providing basic learning materials can help learners with lower verbal ability and lower visual ability (Q3) to learn better. A possible reason is that, since these

students do not have higher verbal and visual abilities, providing these learners with too many written or pictorial annotation will cause a higher cognitive load in their STM and thus could make them irritable and unable to concentrate. According to the Cognitive Load Theory (Sweller, 1994), learners in such situations would probably ignore or skip that the information that caused the overload. How much information a learner would consider to be an overload is a matter for further study.

According to the research of Geva and Ryan (1993) and Harrington and Sawyer (1992), learners with higher verbal ability exhibit better skills for learning with verbal material.

Therefore, providing them learning content in verbal forms would achieve better results than would providing the content in nonverbal form. Thus, theoretically speaking, for learners with

Therefore, providing them learning content in verbal forms would achieve better results than would providing the content in nonverbal form. Thus, theoretically speaking, for learners with