The initial sample of the present study consisted of 60 students, but 12 of them did not complete all the tasks or failed to finish the final test. Thus, in the results section, only data from the 48 students who had completed all the listening tasks and tests were collected and presented. In addition, the original design of assigning participants into PC group (the control group) and smartphone group (the experiment group) was changed due to some students’ unexpected resistance to using smartphones as the main learning tool.
While students in the PC group used personal computers to listen to the podcasts all the time, it was found that 14 students in the smartphone group gradually switched to computers and used computers more often than smartphones. For those who used smartphones to complete only 20% or 40% of the tasks, it would be unreasonable to include them in the smartphone group; however, it would be interesting to know why they chose computers over smartphones when strongly encouraged or even required to use smartphones during the project. Hence, instead of excluding these 14 students from the experiment, the researcher decided to add one more group, which is the MIX group. In summary, students were divided into three groups: PC group, composed by 25 students who used computers as the tool all the time, smartphone group (SP group), composed by 9 students who used smartphones to complete over 60% of the tasks, and MIX group, composed by 14 students who switched between computers and smartphones but used smartphones to complete no more than 50% of the tasks.
In this chapter, the mean scores of pretests, posttests, and delayed posttests of all three groups are presented. Paired Samples T-test was applied to compare the differences between pretests and posttests as well as posttests and delayed posttests
within groups. Also, the mean scores of pretests, posttests, and delayed posttests of each group were compared through One Way ANOVA to see if there is any difference in their performances. Furthermore, Paired Samples T-test was used to analyze students’ responses to the questionnaires and check if changes in students’
attitudes toward mobile learning can be found.
The results of each group students’ performances on listening, vocabulary, and grammar were presented first, and then the comparisons across groups are reported, followed by the results of the questionnaires.
The Effects of Podcasts on Language Learning Listening Improvement Within Groups
The mean scores of the pre- and posttests of general listening ability are presented in Table 3. The mean score in the posttest of PC group (M = 34.68) increases by 3.84 when compared with the pretest (M = 30.84). The mean score in the posttest of MIX group (M = 35.93) is higher than the pretest (M = 32.00) by 3.93. The mean score of SP group also improves from 33.67 to 36.67.
Table 3: Mean Scores of the Pre- and Post-tests of Listening Ability Group N
Pretest Posttest
Mean SD Mean SD t
PC 25 30.84 3.325 34.68 3.509 -5.280***
MIX 14 32.00 3.508 35.93 4.066 -3.518*
SP 9 33.67 3.202 36.67 2.958 -1.693
*p< .05 ***p< .001
The results of the comparison between pretest and posttest within groups show that students in the PC group and MIX group have improved their listening ability to a significant level, while the mean scores of the SP group are not significantly different
in their pre- and posttest.
Vocabulary Gain Within Groups
In addition to general listening ability, vocabulary gain is another focus of the present study. During the experiment session, students learned words and phrases from two different podcast series, namely BBC The English We Speak and English Bites. The English We Speak produced by BBC is released in the form of audio files and focuses on one single item in one episode. On the other hand, English Bites, produced in the form of video files and using activities instead of word items as the theme, introduced several words or phrases in each episode. Given the different characteristics of the two podcast series and possible differences in learning effectiveness caused by the forms of the programs, the results of participants’ learning gains from BBC The English We Speak and English Bites are displayed in Tables 4 and 5 respectively.
In Table 4, obvious vocabulary gains can be seen in all three groups. Students in the PC group received 2.68 in the pretest and 18.80 in the posttest with an increase of 16.12. A distance of 17.36 can also be found in MIX group between the pretest (M
= 2.93) and posttest (M = 20.29). In SP group, students’ performance in the posttest (M = 17.44) also exceeds that in the pretest (M = 2.67) by 14.77.
Table 4: Mean Scores of the Pre- and Post-tests of Vocabulary (BBC set) Group N
Pretest Posttest
Mean SD Mean SD t
PC 25 2.68 1.796 18.80 6.048 -12.793***
MIX 14 2.93 1.979 20.29 5.837 -12.025***
SP 9 2.67 1.225 17.44 6.598 -7.081***
***p< .001
Table 5 shows a similar trend with Table 4 in that all of the groups presented great improvement in the posttest. The mean score of the posttest of PC group (M = 15.48) is higher than the pretest (M = 6.12) by 9.36. The mean score of MIX group increases by 9 with 5.86 in the pretest and 14.86 in the posttest. Scoring 4.22 in the pretest and 12.78 in the posttest, the mean score of students in the SP group also improves by 8. 56.
Table 5: Mean Scores of the Pre- and Post-tests of Vocabulary (English Bites set) Group N
Pretest Posttest
Mean SD Mean SD t
PC 25 6.12 3.004 15.48 6.192 -8.781***
MIX 14 5.86 2.476 14.86 6.335 -5.542***
SP 9 4.22 2.333 12.78 2.949 -7.910***
***p< .001
The results in Tables 4 and 5 reveal that all three groups of participants had increased their vocabulary size through both series of podcasts and the gains all reach to a significant level (p < .001). In addition, though the forms of the two podcast series vary, both are found to be beneficial for participants’ vocabulary learning.
Grammar Knowledge Gain Within Groups
Subsequent to the outcomes of the vocabulary tests, participants’ performances in grammar knowledge tests can be seen in Table 6. The mean score of the posttest of PC group (M = 7.80) increases 3.60 when compared with the pretest (M = 4.20). The MIX group shows a progress of 4.86 between pretest (M = 4.21) and posttest (M = 9.07), and the mean score of the posttest of SP group (M = 9.00) also surpasses the pretest (M = 4.89) by 4.11. The results in Table 6 indicate that all three groups had made significant progress on grammar knowledge after the one-month podcast
listening.
Table 6: Mean Scores of the Pre- and Post-tests of Grammar Group N
Pretest Posttest
Mean SD Mean SD T
PC 25 4.20 1.555 7.80 2.449 -6.647***
MIX 14 4.21 1.672 9.07 1.817 -8.083***
SP 9 4.89 1.453 9.00 2.179 -6.490***
***p< .001
In addition to comparing the differences in the pretests and posttests to evaluate the effectiveness of learning, Paired Samples T-test was also used to examine and compare the differences in mean scores of the posttests and delayed posttests. Except for the listening ability test, one more set of delayed posttests was held to see if participants’ gains in vocabulary and grammar knowledge maintained with the time passing.
Vocabulary Gain Maintenance within Groups
Tables 7 and 8 demonstrate the mean scores of vocabulary posttests and delayed posttests, showing the results of the BBC set and English Bites set respectively. From Table 7, it can be found that the mean scores of the three groups all drop with time, from 18.80 (PC group), 20.29 (MIX group), 17.44 (SP group) in the pretests to 17.84, 18.71, and 15.67. For both PC group and MIX group, the loss of gains reach to a significant level.
Table 7: Mean Scores of the Post- and Delayed Post-tests of Vocabulary (BBC set) Group N
Posttest Delayed Posttest
Mean SD Mean SD T
PC 25 18.80 1.210 17.84 1.202 3.294**
MIX 14 20.29 5.837 18.71 5.784 3.465**
SP 9 17.44 6.598 15.67 6.481 2.061
**p< .01
Similar to the results in Table 7, a decreasing trend in mean scores can be seen in Table 8. When comparing the mean scores of the posttests (PC group M = 15.48, MIX group M = 14.86, SP group M = 12.78) and that of the delayed posttests (PC group M = 14.32, MIX group M = 13.93, SP group M = 11.00), the loss of gains can be found in each group and the differences between the two tests are statistically significant for all the three groups.
Table 8: Mean Scores of the Post- and Delayed Post-tests of Vocabulary (English Bites set)
Group N
Posttest Delayed Posttest
Mean SD Mean SD t
PC 25 15.48 6.192 14.32 6.243 3.688**
MIX 14 14.86 6.335 13.93 6.257 3.484**
SP 9 12.78 2.949 11.00 3.808 3.108*
*p< .05 **p<.01
Grammar Knowledge Gain Maintenance within Groups
Following the vocabulary test results are the mean scores of grammar tests presented in Table 9. Similarly, decreases from the posttests (PC group M = 7.80, MIX group M = 9.07, SP group M = 9.00) to delayed posttests (PC group M = 6.88, MIX group M = 8.43, SP group M = 8.44) are showed in each group, but only the differences in PC group and MIX group reach the significant level.
Table 9: Mean Scores of the Post- and Delayed Post-tests of Grammar Group N
Posttest Delayed Posttest
Mean SD Mean SD t
PC 25 7.80 2.449 6.88 2.068 3.570**
MIX 14 9.07 1.817 8.43 1.742 2.223*
SP 9 9.00 2.179 8.44 2.186 2.294
*p< .05 **p<.01
The Influence of Tools on Learning Effects
After measuring the effectiveness of learning with podcasts, it was found that regardless of the learning tool, all three groups of the participants made significant progress on listening ability, vocabulary size, and grammar knowledge, except that the SP group’s listening does not improve to a significant level. However, whether the degree of progress is influenced by the tools or not is still unknown. Thus, to examine the possible effects of learning tools, namely personal computers and smartphones in this case, One-way ANOVA was adapted to assess the three groups’ performances in pretests and their increase or decrease of knowledge between tests to see if any significant differences can be found.
Pretest Performance Comparison Across Groups
The results of One-way ANOVA on pretests are summarized in Table 10. It can be found that for all the four sets of tests, there is no significant difference in the performances between any groups. This result indicates that participants in the three groups showed similar level of listening proficiency and similar amount of knowledge about target words and grammar points prior to the experiment.
Table 10: Summary of One-way ANOVA on Pretests
Mean Differences Comparison Between Pretest and Posttest Across Groups Given that no apparent difference can be noted in the prior knowledge of the three groups, a further comparison on participants’ mean differences between pretests and posttests was made through One-way ANOVA and the results are displayed in Table 11.
The results reveal that for each set of pre- and posttests, no significant differences in the performance can be found among groups, which suggests that the learning tools in use probably have no great effect on the degree of learning progress in this case.
Table 11: Summary of One-way ANOVA on Mean Differences Between Pretest and
Mean Differences Comparison Between Posttest and Delayed Posttest Across Groups
Following the outcomes of progress comparison is the result of decrease in knowledge between posttest and delayed posttest shown in Table 12. Since there are only one listening pretest and one listening posttest during the one-month experiment in the beginning and the end, no information about listening delayed posttest is presented.
Table 12 shows that no significant difference in performances can be found in all sets of tests, In other words, it seems that participants in the three groups maintained similar amount of learning gains and that the learning tools did not seem to influence the long-term effect of the treatment.
Table 12: Summary of One-way ANOVA on Mean Differences Between Posttests and questionnaire containing questions related to (1) smartphone possession, (2) prior experiences in MALL, and (3) perceptions about MALL. After the experiment, all participants had to respond to another questionnaire regarding (1) reflections on their own learning as well as (2) their thoughts about the content of podcasts, and the MIX group and SP group need to further complete the part addressing (3) perceptions about MALL.
This section will first present participants’ ownership of smartphones and their prior experiences in MALL, followed by students’ self-report on their learning process and feedback on podcast content. Finally the results of participants’
perceptions about MALL before and after the experiment are provided.
Smartphone Ownership and Experiences in MALL
Table 13 demonstrates participants’ possession of smartphones and their
experiences in mobile assisted language learning before the experiment started. It can be found that among all the participants, nearly half of them own a smartphone (48%, n = 23). However, less than one-fourth of all the participants (23%, n = 11) ever used smartphones for English learning. When further asked if they were still using smartphones as an English learning tool, only 6 out of the 11 students gave a positive answer.
Table 13: Results of Smartphone Ownership and MALL Experiences
No. Question Yes No
1 The phone I’m using is smartphone. N percent N Percent
23 48% 25 52%
2. Have you ever used smartphones as
the tool for English learning? 11 23% 37 77%
4. Are you still using smartphones for
English learning now? 6 55% 5 45%
Note. (Questions No. 1 & 2) Total number of participants = 48
(Question No. 4) Total number of participants with MALL experiences = 11
Students’ Reflections on Their Learning
In the posttest questionnaire, students were required to report to what extent they understood the content of the podcasts after listening just once. From Table 14, it can be seen that for the BBC podcast The English We Speak, the great majority of the students in each group comprehended over 80% of the content (PC group 82%, MIC group 100%, SP group 89%). As for English Bites, 80% of the PC group students, 79% of the MIX group students, and 55% of the SP group students can understand over 80% of the content. However, for Grammar Girl, only 16% of the PC group, 14% of the MIX group, and 33% of the SP group students conceived over 80% of the content after they listened to the program once. This result indicates that Grammar
Girl is much more difficult for students to handle.
Table 14: Students’ Self-report on Listening Comprehension Group
Podcast
PC (n= 25) MIX (n= 14) SP (n= 9) Frequency Percent Frequency Percent Frequency Percent
BBC 90% 13 52% 12 86% 6 67%
80%-90% 10 40% 2 14% 2 22%
70%-80% 2 8% 0 0% 1 11%
60%-70% 0 0% 0 0% 0 0%
60% 0 0% 0 0% 0 0%
EB 90% 13 52% 7 50% 3 33%
80%-90% 7 28% 4 29% 2 22%
70%-80% 4 16% 3 21% 4 45%
60%-70% 1 4% 0 0% 0 0%
60% 0 0% 0 0% 0 0%
GG 90% 1 4% 0 0% 0 0%
80%-90% 3 12% 2 14% 3 33%
70%-80% 10 40% 8 57% 4 45%
60%-70% 10 40% 3 22% 2 22%
60% 1 4% 1 7% 0 0%
Students’ Feedback on Podcast Content
In addition to reporting on their comprehension, participants also had to provide feedback on podcast content in terms of interest, comprehensibility, usefulness and preference degree, and the results are shown in Table 15.
For BBC The English We Speak and English Bites, participants are very positive in each aspect. Both of the two podcasts received points higher than 4 in interest, comprehensibility, usefulness and preference degree (BBC: 4.73, 4.29, 4.23, 4.63; EB: 4.19, 4.15, 4.15, 4.10). The results indicate that students considered these two sets of programs to be interesting, useful, and highly comprehensible.
Table 15: Mean Scores of Students’ Feedback on Podcast Content
On the other hand, about the podcast series Grammar Girl, participants are not as positive. Though students thought the content is quite useful (3.94), they also perceived it as challenging (2.56) and not so interesting (2.79). When asked about their degree of preference about the program, the point (2.96) suggests they are not so fond of it.
Students’ Attitudes Toward Mobile Assisted Language Learning Other than participants’ perceptions about the learning materials, their attitudes toward the learning tool, smartphones, is another important aspect to investigate. In this part, students from MIX group and SP group responded to questions regarding (1) the physical conditions of smartphones for learning, (2) the usefulness and effectiveness of smartphones as a language learning tool, and (3) their overall perceptions about learning with smartphones.
Table 16 summarizes the mean scores of items related to the physical conditions and the mean differences between the pre- and posttest. As can be seen in Table 16, a similar pattern was found in MIX group and SP group when responding to questions 3 and 4. For the statement that the smartphone screen is big enough for English learning, both groups became more positive but only to a limited degree after the experiment (MIX: from 3.64 to 4.00, SP: form 3.44 to 3.56). As to the statement
that the virtual keyboard is easy to use when typing words, the points in both groups slightly decreased (MIX: from 3.00 to 2.86, SP: from 3.33 to 3.22).
Table 16: Mean Scores of Students’ Attitudes toward MALL in terms of Physical Conditions of Smartphones for Learning