RESULTS
The current study was set out to explore whether different caption modes affect L2 leaners’ listening comprehension and whether L2 learners’ modality preferences influence their comprehension performance under different caption conditions. The ensuing paragraphs will report on the results of the obtained quantitative and qualitative data. Specifically, the report will begin with the descriptive statistics, which aims to disclose the overview of the participants’ caption viewing behaviors, and will then report on the two-way ANOVA results, post-hoc test analysis and the questionnaire data.
It is important to note that a few participants’ data were not included in the ANOVA and post-hoc analysis due to the following reasons: (1) invalid CRT data, (2) insufficient number of balanced learners, and (3) outlier or atypical behaviors based on the interview data. First, several participants scored “two or more points” under the “non-target words” category in the CRT test, which suggests that the CRT items were probably too challenging for these people and thus did not serve as a valid input modality preference determination mechanism for them (see 3.4.1 for more detailed explanation). Second, the CRT test results indicated that while 46% and 43% of the participants were categorized as visual and auditory learners, only 10% of them were classified as balanced learners. Given that the sample size of the balanced learners was too small to be compared with the other modality groups for meaningful statistical inferences (Faber & Fonseca, 2014), the balanced learners’ data was removed from the current study. Lastly, based on the participants’ interview data, the researcher noticed that a few participants were either familiar with the selected video or fully aware of the purpose of this study. Since these participants might have approached the task with strategies that were not characteristic of normal video viewing behaviors, their data were excluded from the analysis regardless of their comprehension performance.
4.1 Descriptive statistics
Table 3 below presents the descriptive statistics of the participants’ listening comprehension test scores under the four caption conditions. Overall, the mean scores under the four caption conditions appear to show little difference
Table 2. Means and standard deviations on listening comprehension
Level of factor N Mean SD
Total 95 11.21 1.76
No captions 33 11.24 1.62
Full captions 32 11.13 1.77
Partial captions 14 11.57 2.10
Real-time captions 16 11 1.83
Figure 4. Listening comprehension test means of visual and auditory learners under four caption conditions
Notwithstanding, subtle differences emerge when individual differences (i.e., preferred modality in L2 input processing) are considered, as gleaned from Figure 4.
Specifically, auditory learners were found to outperform visual learners under every caption condition except full caption condition. Among the four caption conditions,
9 10 11 12 13 14
No captions Full captions Partial captions Real-time captions
Visual Auditory
auditory learners scored the highest under the partial caption condition (M = 13.14) and the lowest under the full caption condition (M = 11.07). On the contrary, visual learners performed the best under the full caption condition (M = 11.18) and the worst under the partial caption condition (M = 10). The above descriptive data is suggestive of the interplay between L2 learners’ modality preferences and caption viewing conditions and its impact on the participants’ comprehension. Accordingly, a more fine-grained analysis of the participants’ data is reported below.
4.2 Two-way ANOVA analysis
The Shapiro-Wilk normality test results showed that the data did not follow a normal distribution (W = .96, p = .009), which violated the prerequisite for parametric tests. The non-normal distribution might be attributed to the proficiency threshold set for participant recruitment. Specifically, the participants’ L2 proficiency was confined to a particular level—the high-intermediate level (see 3.1 for more details), resulting in a slightly left-skewed distribution of the comprehension test scores. In light of the Shapiro-Wilk normality test result, the researcher adopted a non-parametric generalized technique by converting the original comprehension scores into ranked data. The resulting ranked data allowed the researcher to perform t-test, regression and factorial ANOVA on the obtained data (Thomas, Nelson, & Thomas, 1999).
A 2x4 two-way ANOVA test was then performed with two independent variables (i.e., caption mode and modality preference) and one dependent variable (i.e., listening comprehension test score). The results indicated that caption mode and modality preference were able to explain 15.6% of the variation of the listening comprehension performance (R2 = .156). As shown in Table 4, although caption mode did not significantly affect the participants’ listening comprehension scores (F = .065, p = .978), modality preference exerted a significant effect on comprehension performance (F =
7.78, p = .007), with a moderate effect size (Partial Eta2 = 0.08). Additionally, an interaction is also found between caption mode and modality preference on listening comprehension test score (F = 7.78, p = .007), with a moderate-to-large effect size (Partial Eta2 = 0.11). This indicated that the participants’ comprehension performance under different caption conditions were influenced by their modality preferences.
Table 3. Two-way ANOVA analysis results
df F p Partial Eta2
Caption mode 3 0.065 0.978 0.002
Modality preference 1 7.776 *0.007 0.082
Caption mode*Modality preference 3 3.651 *0.016 0.112
4.3 Post hoc test results
To further examine the interaction between caption mode and modality preference, the Fisher’s LSD post hoc test was employed to evaluate differences among groups (see Table 4). Figures 5 and 6 visually schematize the (differential) effects of the participants’
modality preferences on their comprehension under the four captioning conditions.
As shown in Figure 5, among the four caption conditions, significant between-group difference between visual and auditory L2 learners’ comprehension performance was observed only under the partial caption condition, where auditory learners scored significantly higher than visual learners; such a between-group difference did not surface under other caption conditions. In regard to the respective within-group difference under the four caption conditions, auditory learners scored significantly higher under the partial caption condition than under full and real-time caption conditions (see Figure 6). In contrast, visual learners performed significantly better under the full caption condition than under the partial caption condition (see Figure 6).
Table 4. Fisher’s LSD post hoc test results
Probabilities for Post Hoc Tests Error: Between MS = 669.61, df = 87.000 Caption
Note. (1) NC = No captions; FC = Full captions; PC = Partial captions; RC = Real-time captions (2) The icon “＊” indicates there is a significant difference
Note. The icon “＊” indicates there is a significant difference between two bars.
Figure 5. Visual schematization of between-group comparisons between visual and auditory L2 learners
No captions Full captions Partial captions Real-time captions Visual Auditory
＊
Note. The icon “＊” indicates there is a significant difference between two bars.
Figure 6. Visual schematization of respective within-group comparisons under the four caption conditions
4.4 Questionnaire results
Table 5 displays the results of the five-point Likert scale items. The first item intended to establish the level appropriateness of the video content. The remaining items aimed to shed light on the participants’ perception of caption use and their self-reported attention allocation while viewing captioned video. Since the participants assigned to the no caption condition were not provided with caption support, those under full, partial, and real-time caption conditions are the primary focus in this section.
As shown in Table 5, the average ratings of item 1 under four caption conditions are all above 4 (out of the highest possible rating of 5), suggesting that the participants generally had no problem understanding the main idea of the video. This confirms the level appropriateness of the viewing materials used in the study.
The average ratings on items 2 and 3, which aimed to probe participants’
perception of caption use, showed that the participants who were assigned to the full (M2 = 4.41, M3 = 4.03) and real-time (M2 = 4.19, M3 = 4.06) caption conditions generally considered captions helpful for both L2 comprehension and learning.
9
Table 5. Descriptive statistics of the five-point Likert scale items
(1 = Strongly disagree, 2 = Disagree, 3 = Neutral, 4 = Agree, 5 = Strongly agree)
NC FC PC RC PC-V PC-A
2. I think the captions helped me
understand the audio content. 4.41 2.71 4.19 3.29 2.14 3. I find the captions helpful for
learning English. 4.03 2.71 4.06 3.00 2.43
4. I think the captions in the video
were distracting. 2.41 3.57 2.50 3.43 3.71
Captioned video viewing behaviors
5. I always attended to the captions
while watching the video. 3.19 2.29 3.44 3.00 1.57 6. I focused more on the audio than the
captions while watching the video. 2.81 4.00 2.94 3.43 4.57 7. I focused more on the captions than
the audio while watching the video. 3.47 2.00 3.38 2.57 1.43
Note. NC = No captions; FC = Full captions; PC = Partial captions; RC = Real-time captions;
V=Visual; A= Auditory
Conversely, the participants assigned to the partial caption condition tended to disagree with the helpfulness of captions (M2 = 2.71, M3 = 2.71). The results of item 4, which was designed to investigate whether captions caused distractions while viewing, showed that participants assigned to the partial caption condition found captions more distracting (M4 = 3.57) compared to those assigned to the full (M4 = 2.41) and real-time caption conditions (M4 = 2.50). This tendency was more manifest for auditory L2 learners under the partial caption condition.
As for the participants’ self-reported attention allocation while viewing captioned video, the average ratings of item 5 indicated that the participants’ attention to caption
varied under different viewing conditions and that their attentional investment ranking, ranging from the highest to the lowest, was (1) real-time caption (M5 = 3.44); (2) full caption (M5 = 3.19); and (3) partial caption (M5 = 2.29). A larger discrepancy was found in items 6 and 7 when comparing the full and real-time caption with the partial caption condition. These two items inquired whether the participants attended to auditory input or captions more during the experiment. While the participants assigned to the full and real-time caption conditions showed a slightly heavier reliance on captions (FC: M7 = 3.47, RC: M7 = 3.38) than auditory input (FC: M6 = 2.81, RC: M6 =2.94), those under the partial caption condition focused markedly more on audio (M6 = 4) than on captions (M7 = 2). This is especially true for auditory learners and even visual learners relied slightly more on auditory input than captions under the partial caption condition.