4 Results…
4.1 Descriptive results
The 9 video materials were played in the same sequence to the three classes, each class with a designated caption mode and all completing the same listening comprehension tests. Table 3 below reveals the test results, including the mean scores (M), standard deviation (SD), numbers of participants (N) and the average scores. The figures reveal a significant difference between the performance of the subtitled group and that of the captioned group. The average score of the subtitled group was 59.62, while the score of the captioned group was 72.79, which was significantly higher than that of the subtitled group by 22 percent. The test results of the control group (non-captioned group) were close to those of the subtitled group. The trajectories of the performances of the 3 groups over the 9 interventions are illustrated in Chart 1 below.
The graphics showed the captioned group outperformed the other groups throughout the 9 tests, except for test 2. A detailed explanation is given in the next section.
Table 3
The Performances of the Subtitled, the Captioned, and Non-Captioned Groups on the 9 Comprehension Tests
Video Subtitled (N=26) Captioned (N=28) Non-Captioned (N=30)
M SD M SD M SD
1 56.923 26.7984 70.357 25.6012 65.000 23.0067 2 68.846 30.3746 83.643 19.8069 84.333 16.9550 3 69.231 30.3214 72.857 26.0849 64.667 32.7723 4 51.154 29.5739 72.143 23.4690 59.000 27.7116 5 61.923 31.8772 72.857 27.0606 62.333 22.6949 6 61.538 25.4074 68.929 27.8008 60.333 25.5266 7 50.385 28.3522 70.357 21.6850 64.667 22.2421 8 62.692 32.4417 68.571 19.3820 58.000 22.1904 9 53.846 27.2876 75.357 27.6864 55.333 28.3735
average 59.62 7.01 72.79 4.59 63.74 8.41
Chart 1 The Performance Trajectories of the Subtitled, Captioned, and Non-Captioned Groups on the 9 Comprehension Tests
To answer the first research question, an analysis of repeated measures ANOVA was conducted. The results were presented in Table 4 below. The figure, P = .021
< .05, indicated that caption modes had significant influence on listening
comprehension. Post hoc tests using the Scheffe method indicated that the captioned group performed significantly better than the subtitled group (P = 0.025) (see Table 5). The difference between the performance of the subtitled group and that of the non-captioned group was insignificant (P = 0.678). And the difference between the performance of the captioned group and that of the non-captioned group was marginally insignificant (P = 0.149).
Table 4
Two-way RM-ANOVA on Caption Modes for Listening Comprehension (Test of between variables)
Source
Type III sum of
square df MS F p-value partial η2
Intercept 3220605.568 1 3220605.568 1178.825 .000 .938
Proficiency 93696.871 1 93696.871 34.295 .000 .305
Mode 22314.996 2 11157.498 4.084 .021 .095
Proficiency * Mode 1860.686 2 930.343 .341 .712 .009
Error 213099.672 78 2732.047
0.00 20.00 40.00 60.00 80.00 100.00
1 2 3 4 5 6 7 8 9
Subtitled Captioned Non-Captioned
Table 5 Subtitled Captioned -13.1703* 4.74519 .025 -25.0120 -1.3286
Non-Captioned -4.1254 4.66842 .678 -15.7754 7.5247
Captioned Subtitled 13.1703* 4.74519 .025 1.3286 25.0120
Non-Captioned 9.0450 4.57823 .149 -2.3800 20.4700 Non-Captioned Subtitled 4.1254 4.66842 .678 -7.5247 15.7754 Captioned -9.0450 4.57823 .149 -20.4700 2.3800
To answer the second research question, language proficiency level was added as the covariate variable in the analysis of RM-ANOVA (Table 4). An interaction of language proficiency levels and caption modes was revealed. The results showed that there was no significant relationship between caption modes and listening
comprehension as a function of language proficiency levels (P = 0.712). In other words, caption modes posed more effect on listening comprehension than linguistic proficiency did. An explanatory account is provided in the ensuing section.
In addition, Table 6 below shows the test results of the high proficiency group with three caption modes, including the mean scores, standard deviation (SD), and average scores of the 9 listening comprehension tests. A vivid illustration of the performance of the high proficiency groups is presented in the followed Chart 2.
Table 6
The Listening Comprehension Test Results of the High Proficiency Groups Using 3 Caption Modes Subtitled (n=13) Captioned (n=14) Non-Captioned (n=15)
mean SD mean SD mean SD
Chart 2 The Performance Trajectories of Listening Comprehension of the High Proficiency Groups Using 3 Caption Modes
Likewise, the test results of the low proficiency groups are revealed in Table 7 below, followed by Chart 3 illustrating the individual performances of the groups using the 3 different caption modes. The data revealed that there was no significant difference of the performances on the listening comprehension tests among the three sub-groups of these learners of the low proficiency level. That is to say, the caption modes posed no impact on the listening comprehension of these less proficient learners.
Table 7
The Listening Comprehension Test Results of the Low Proficiency Groups Using 3 Caption Modes Subtitled (n=13) Captioned (n=14) Non-Captioned (n=15)
score SD score SD score SD
Chart 3 The Performance Trajectories of Listening Comprehension of the Low Proficiency Groups Using 3 Caption Modes
To further validate the effectiveness between caption modes of the high and the low proficiency groups, the repeated measures ANOVA (RM-ANOVA) were
administered again (Tables 8 & 9). The results of the high proficiency groups in Table 8 below, showed that P-value was 0.069. The figure indicated that the impact of caption modes on listening comprehension was marginally significant for learners with higher linguistic profiles, However, the statistical insignificance was due to the small sample sizes of the groups (subtitled: 13, captiond:14, non-captioned: 15).
Further explanation is presented in the subsequent section.
Table 8
RM-ANOVA on Listening Comprehension of the High Proficiency Groups Using 3 Caption Modes (Test of Between-Subjects Effect)
Source
Type III sum of
square df MS F p-value partial η2
Intercept 2206478.687 1 2206478.687 703.897 .000 .948
Mode 17916.696 2 8958.348 2.858 .069 .128
Error 122251.780 39 3134.661
0.00 20.00 40.00 60.00 80.00 100.00
1 2 3 4 5 6 7 8 9
Subtitled Captioned Non-Captioned
Table 9
RM-ANOVA on Listening Comprehension of the Low Proficiency Groups Using 3 Caption Modes (Test of Between-Subjects Effect)
Source Type III sum of square df MS F p-value partial η2
Intercept 1107823.753 1 1107823.753 475.577 .000 .924
Mode 6258.987 2 3129.493 1.343 .273 .064
Error 90847.892 39 2329.433
As for the low proficiency groups, no significant difference was observed among caption modes (P = 0.273, Table 9). In other words, the listening comprehension of learners with lower linguistic profiles was not significantly affected by the difference of caption modes. An elaboration on this phenomenon is also presented in the
following section.
To answer the 3rd research question, a 10-item questionnaire was administered to all participants to collect qualitative data regarding the EFL learners’ perceptions about enhancing listening comprehension through watching online video materials.
The feedback was analyzed based on participant grouping and caption modes for the researcher to discern if these learners responded differently to the questionnaire questions. The statistic results are shown in Table 10 below.
Table 10
Statistic Results of the Questionnaire among the High and the Low Proficiency Learners Using 3 Caption Modes
2. Acquired linguistic components (e.g., vocabulary,
phrases, grammar, etc.) 3.9 (1.3) 3.8 (1.4) 4.4 (0.8) 3.8 (1) 4.2 (1.1) 4.4 (0.9) 4.1 (1.1) 9. Found watching online videos an effective tool in
promoting English proficiency. 3.8 (1.2) 3.3 (0.9) 4.4 (0.8) 4.5 (0.8) 4.7 (0.9) 4.7 (0.9) 4.3 (1.1) 10. Will watch English videos from online streaming
websites (e.g. YouTube & TED) more often 3.3 (1.5) 3.5 (1.2) 4 (0.9) 3.8 (1) 4.4 (0.9) 4.8 (1) 4 (1.2) 1: absolutely disagree; 6 absolutely agree
According to the feedback of the respondents, more than half of the participants reported that they understood the main idea of the videos (Q1: mean = 4.3). Most respondents, regardless of caption modes and linguistic proficiency, agreed that they acquired some knowledge on vocabulary, phrases and grammar throughout the interventions (Q2: mean = 4.1). Many participants felt positive about the
improvements on their English listening comprehension skills over the 3 weeks of interventions (Q3: mean = 4). However, that score, which was the lowest out of the 10 questions, indicated that little or no improvement was also reported by many of the respondents (n = 23). When asked if the textual information on the screen, i.e., Chinese subtitles or English captions, were distracting to their listening
comprehension, 45 out of 54 respondents disagreed on the assumption (Q4: mean = 2.2). When asked if the presence of Chinese subtitles / English captions was helpful to their listening comprehension, 41 out of 54 respondents agreed on the assumption (Q5: mean = 4.6). In response to question 6, 17 out of 26 participants in the subtitled group admitted that they constantly read subtitles during the interventions. As for the captioned group, as many as 25 out of 28 participants confessed that they often read captions. Though many participants responded that they enjoyed learning English through watching videos, some learners were unfavorable toward the pedagogy, which resulted in the mean score being comparatively low out of the 10 questions (Q8: mean = 4.1). Yet almost every participant agreed that watching online video materials serves as an effective tool in promoting English proficiency (Q9: mean = 4.3). When asked if they would watch English videos from online websites such as YouTube and TED more frequently after the study, all the participants were positive to the question (Q10: mean = 4).
To further explore the participants’ afterthoughts and feedbacks, a post-study
a video recorder. The feedback from the interviewees showed that learning English by watching online videos was generally viewed as a helpful approach. While most research participants found the approach interesting, some declared the experience to be not interesting (n=17). When asked to elaborate further on the cause of their disinterest, the respondents explained that it was the result of the frustration and anxiety experienced because the materials presented were incomprehensible. Almost all the interviewees favored the subtitled mode over the captioned one when they were given the option at home. Only 2 learners from the high proficiency group selected the captioned mode in order to acquire more linguistic elements. Eight participants, also from the high proficiency group, would choose the simultaneously display of the subtitles and the captions. They believed that by doing this, they could learn English without missing the gist of the videos.