CHAPTER 3 METHODOLOGY
3.6 Statistical Analysis
STATISTICA 13.0 for Windows was used to generate descriptive statistics for the data in this study. A three-way ANOVA was used to explore the effects of the independent variable (i.e., caption modes) and the moderating variables (i.e., working memory capacity measure and modality preference) on the dependent variable (i.e., listening comprehension scores). The grouping conditions from this study resulted in
43
sufficient number of participants per-condition to use three-way ANOVA to generate main effects and interactions among variables. Also, ANOVA was able to run data that contained one interval scale variable (i.e., listening comprehension) and three categorical scale variables (i.e., captions modes, working memory capacity, and modality preference). Independent t-tests were used to compare the means of different conditions. The alpha level of all tests was set at p < .05 as the level of significance for this study.
44
CHAPTER 4 RESULTS
This study was set out to investigate whether working memory capacity and online preferred modality affected the effect of captioning on L2 listening comprehension. The following sections will present the quantitative and qualitative data respectively. To begin with, an overview of the participants’ captioning viewing behaviors will be disclosed throughout the descriptive statistics. Then the three-way ANOVA results will be displayed, along with the results from independent t-tests for post-hoc analysis. The questionnaire data supported by the interviews will be presented at last.
In this study, a total of 84 participants were recruited for the experiment. 12 of them were excluded from later analysis because they reported having already watched the selected (captioned) video. Additionally, they felt that content of the viewing materials was “too easy” for them. None of the above were desirable participant behaviors, which resulted in their absence in later data analysis.
4.1 Quantitative
4.1.1 Descriptive Statistics
Table 2 demonstrates an overview of the participants’ listening comprehension scores vis-à-vis the factor/variable of this study. Although the mean difference between
45
control (no caption) and experimental conditions (full caption) is small (M diff. = .005), larger variance is found in every condition with captions than without captions. Notably, such variance in the full caption conditions increases when taking individual differences (i.e., online preferred modality, working memory capacity, or both) into account. As such pattern consistently manifests throughout Table 2, it calls for a more fine-grained investigation on the analysis of the potential effects and interactions among the variables in this study.
Table 2
Descriptive Statistics of the participants’ listening comprehension test scores.
Variables Level of factors N Mean SD
Total 60 11.15 1.62
CM NC 28 11.17 1.46
FC 32 11.12 1.77
PM A 29 11.55 1.37
V 31 10.77 1.76
WM L 24 10.91 1.44
H 36 11.30 1.73
CM*PM NC*A 14 12.07 1.26
NC*V 14 10.28 1.06
FC*A 15 11.06 1.33
FC*V 17 11.17 2.12
46
CM*PM*WM NC*A*L 4 11.5 0.57
NC*V*L 8 10.5 1.30
FC*A*L 5 11.8 0.83
FC*V*L 7 10.42 1.98
NC*A*H 10 12.3 1.41
NC*V*H 6 10 0.63
FC*A*H 10 10.7 1.41
FC*V*H 10 11.7 2.16
Note: Caption Mode (CM), Preferred Modality (PM), Working Memory (WM), No caption (NC), Full Caption (FC), Auditory (A), Visual (V), Low (L), High (H).
4.1.2 A three-way ANOVA
The Shapiro-Wilk test was used to determine the normality of the listening comprehension scores, which yielded a slightly left-skewed distribution (W=.958, p=.039). Since the results did not follow a normal distribution, this study adopted a
non-parametric generalized technique (Thomas, Nelson, & Thomas, 1999) to convert the raw data into ranked data. This procedure allowed the research to perform normal parametric tests (e.g., t-test, ANOVA, and regression) on the converted comprehension scores.
The results from a three-way ANOVA test revealed that the inclusion of caption mode, online preferred modality, and working memory capacity can explain 27%
variance of the listening comprehension performance (R2=.27). Table 3 displays the
47
main effects of the three independent variables on listening comprehension, showing that only the participants’ preferred modality significantly affected their listening comprehension scores (p = .02). This significance was found to have a close-to-large effect size (η2 = .09). Notably, the auditory learners significantly outperformed the visual learners in their listening comprehension outcomes, as Figure 4 visually schematizes such contrast.
Table 3
A three-way ANOVA result for the effects of captions modes, preferred modality, and
working memory and their interactions on L2 listening comprehension
DF F p Partial
eta-squared (η2)
Observed power (alpha=0.05)
CM 1 0.44 0.5 0.008 0.10
PM 1 5.67 0.02* 0.09 0.64
WM 1 0.001 0.96 0.00002 0.05
CM*PM 1 4.94 0.03* 0.08 0.58
CM*WM 1 0 0.99 0 0.05
CM*PM*WM 1 5.797 0.01* 0.10 0.65
48
Figure 4. Mean difference between auditory and visual L2 learners’ listening
comprehension scores.
The listening advantage of the auditory learners, however, did not always hold true when taking caption mode into consideration. As shown in Table 3, there is an interaction (p = .03; η2 = .08) between the participants’ preferred modality and their caption viewing conditions (i.e., full caption or no caption) on their listening performances. Post-hoc analysis from an independent t-test further confirmed that captions had a significantly disruptive effect on the auditory learners (p = .048). The disruptive effect is visually schematized in Figure 5, showing that when exposed to captions, auditory learners performed substantively poorer than visual learners than without captions. However, without captions, the listening advantage for the auditory
9 9.5 10 10.5 11 11.5 12 12.5 13 13.5
Auditory Visual
49
learners resurfaces, as they significantly outperformed their visual counterparts (p = .00) in the no caption condition. Despite the negative effect of caption imposed on the auditory learners, such an effect was not observed in the visual learners’ listening comprehension under full and no caption conditions (p = .144).
Figure 5. The interactions between caption mode and preferred modality.
Although the effect of captioning did not reach a significant level for the visual learners, a robust variance was observed in their listening performances (SD = 2.12).
Such variance implies that preferred modality may not be the only factor that could potentially modulate the effect of captioning. This study found that whether captions were indeed facilitative to L2 listening comprehension was not only depended on preferred modality, but more importantly, working memory capacity.
9
50
As shown in Table 3, there is an interaction among caption mode, preferred modality, and working memory capacity (p = .01), with a close-to-large effect size (η2
= .10). One may recall that the participants were grouped into having high and low working memory capacity based on their reading span test scores. Figure 6 shows that concerning the low working memory condition, neither did captioning exert a facilitative nor disruptive effect on both auditory and visual learners. That is, regardless of preferred modality, the presence of captions did not affect the listening comprehension of those with lower working memory capacity.
In contrast to the minimal effect captioning had on L2 learners with lower working memory capacity, a significant interaction was observed in the high working memory condition. Similar to the effects seen in Figure 5, the auditory learners with higher working memory outperformed the visual learners in the no caption condition (p = .001).
When exposed to captions, however, it diminished the auditory learners’ listening advantage, as they scored significantly lower than those in the no caption condition (p
= .021). This result reinforces the detrimental effect captions had on auditory learners’
understanding of the audiovisual materials, specifically those with higher working memory capacity.
51
Figure 6. The interactions among caption mode, preferred modality, and working
memory capacity.
Although captions failed to merit auditory learners with higher working memory capacity, it did wonder to their visual counterparts. Figure 6 illustrates that for the visual learners with higher working memory capacity, captioning significantly augmented their listening comprehension (p = .04)—a phenomenon not manifest in the visual learners’ performance data when working memory capacity was not considered. When working memory capacity was reckoned, it accentuated the effect of captions. This phenomenon stipulates that the visual learners with higher working memory capacity may better process multimodal information with captions. The inclusion of working memory capacity, therefore, discovered the important role it has in determining the effect of preferred modality in affecting the listening outcome under different viewing condition (i.e., full and no captions).
52
4.2 Qualitative data
This study used both questionnaire and interviews as a secondary data source to support the quantitative results. Since the questionnaire was designed to probe the participants’ perception/experience with captioned video content, the responses reported below will mainly focused on the participants assigned to the full caption (FC) condition. For the participants in the no caption (NC) condition, their perceptions toward captions will be reported by drawing on insights from their qualitative comments.
Table 4
Descriptive statistics of the questionnaire data with five-point Likert scale items (1 =
Strongly disagree, 2 = Disagree, 3 = Neutral, 4 = Agree, 5 = Strongly agree)
Statements FC
I focused more on the audio than the captions while watching the video.
2.41 4.6 2.7 4.93
I focused more on the captions than the audio while watching the video.
4.47 3.57 4.87 3.87
Note. (1) FC = Full captions; NC = No captions; A = Auditory learners; V = Visual
learners; H = High working memory capacity.
53
Auditory learners under the full caption condition
In Table 4, the questionnaire data showed that the auditory learners under the full caption condition reported that they tended to agree that their focal attention was directed to the oral discourse of the video (M = 4.6), in particular for those with higher working memory capacity (M = 4.93). Despite this, their average rating for the second questionnaire item (“I focused more on the captions than the audio”) is 3.57, indicating that they tended to agree that despite their (auditory) modality preference, they were aware of their attention to captions. This is especially the case for those auditory learners with higher working memory capacity (M = 3.87). This finding is further supported by the qualitative comments from the auditory learners: two thirds of the auditory learners (8/15) from the full caption condition expressed that captions engaged a lot of their attention, four of which even showed strong dislikes towards captions.
Additionally, when asked for their online-viewing preference, two thirds of the auditory learners under the full caption condition preferred not having captions on. Specifically, seven of them evinced that captions could direct their attention unwantedly, two of them found captions could be distracting, and 1 stressed that captions could impose more cognitive load to their understanding of the multimodal information. The followings are a few examples from the interview data:
“…Captions can attract a lot of attention from me, and I don't like it. When this
54
happens, I feel like im not polishing my listening skills. I was paying a lot of attention on captions during video watching….”
“…Captions can be distracting…”
“…I fell captions could attract a lot of my attention during video watching…”
“…Captions can attract a lot of attention, and I don't really like my attention being controlled by captions…”
“…Even though I'd normally turn captions on, I still don't like my attention being controlled by captions…”
“…But with captions, it can impose more cognitive load because you have to constantly attend to so many things. It can also be a little distracting…”
Visual learners under the full caption condition
In contrast, the visual learners’ average rating for the first item is 2.41, indicating that they tended to strongly disagree that they attended to the oral discourse while viewing the captioned video; even for those visual learners with high working memory capacity (M = 2.7). When responding to the second item, the visual learners’ average rating is 4.47, suggesting that they tended to strongly agree that they attended to captions; this tendency is even more apparent for the visual learners with high working memory capacity (M = 4.87).
55
Unlike the auditory learners’ negative comments noted above, visual learners exhibited a more positive attitude towards having captions during their online viewing experience. 6 of them explained that captions would not be distracting; 5 of them regarded captions as facilitative to their understanding of the audiovisual cues; 2 of them stressed that captions made them feel secure, allowing them to verify whether the received input is accurate. Specifically, having captions on would make them feel less cognitively demanded, for they intently relied on reading captions. The followings are a few examples from the interview data:
“…I don't find captions distracting, it can help me understand difficult terms…”
“…Watching video with captions is much easier (less cognitive load) for me;
whereas without captions could be more demanding.”
“…Captions allow me to make sure that what I listen is correct. I really enjoy havnig captions because it allows me to verify whether my listening is correct…”
“When I turn captions on, it's less cognitively challenging because I can just read them. Without captions, however, may be more cognitively challenging for me because I have to force myself to concentrate and to not space out.”
Notably, five visual learners seem to treat captions as an optional asset, namely, an multimodal input that they could draw on when needed:
56
“…I feel I pay more attention to listening, and only briefly attend to captions when I need details to facilitate my comprehension…”
“Captions can be helpful for understanding the content. If I don't want to look at it, I just ignore it.”
Auditory learners under the no caption condition
Seven auditory learners felt that captions were attention-demanding during video-viewing. They specifically concerned that captions directed their attention to specifics, rather than the tenor of the content. In that case, all of them preferred not having captions on during online viewing, for they concerned that captions were not conducive to their listening comprehension:
“…Captions may be helpful for understanding technical terms. I wouldn't turn captions on for this video because it's more focusing on understanding "central ideas”…”
“…I think if we were to train our listening comprehension skills, having captions on may be less desirable…”
“…Captions can draw a lot of attention and maybe this is not helpful for training L2 listening...”
57
Visual learners under the no caption condition
Twelve visual learners stated that they found captions attention-demanding, but not distracting. Importantly, five of them explicitly noted “I typically find captions helpful in promoting my comprehension, but viewing non-captioned videos (i.e., the TED talk video) also does not bother me at all because the video content already provides a lot of contextual visual clues.” Notably, these comments all came from the visual learners with high working memory capacity.
58
CHAPTER 5 DISCUSSION
The two research questions addressed in this study intend to unravel whether L2 learners’ preferred modality and working memory capacity modulate the effect of captioning. In particular, the extent to which the investigated variables interacted with each other will be discussed in the ensuing paragraphs, with an aim to further the understanding of captioning in the context of differentiated teaching/learning.
Prior to discussion, an interesting finding stood out. Specifically, this finding reinforced the need of considering L2 learners’ input processing profiles (i.e., preferred modality and working memory capacity), especially when video instructional materials are used. Overall, the quantitative results showed that when L2 learners’ input processing profiles were not taken into consideration, L2 learners—irrespective of their caption viewing conditions—did not differ significantly in their listening comprehension performances. The comparable outcomes from both caption and no-caption conditions indicated that the presence of no-captions did not significantly influence L2 learners’ listening comprehension.
Overall, although the effect of captions was not significantly accentuated in this study, larger variance of listening scores was observed in the full caption condition (SD
= 1.77). Notably, when L2 learners’ input processing profiles were considered, such
59
variance became greater, especially in conditions with full captions. The fact that larger variance was seen in full caption conditions indicated that the effect of captions may still exist, except selectively. Such selectiveness corroborated with a point made in chapter 2, suggesting that not all L2 learners would benefit from captions. Also pointed out in the previous sections, there is still a discrepancy regarding whether captions are truly beneficial to L2 listening comprehension. The ensuing sections, therefore, present the insights obtained from this study with an empirical quest to close the aforementioned discrepancy.
5.1 RQ1: Does preferred modality modulate the effect of captioning?
The first research question concerns whether participants’ preferred modality (i.e., visual and auditory) would modulate the effect of captioning on L2 learners’ listening comprehension. Statistical results revealed that this input processing factor significantly modulated the effect of captioning. It was also evidenced that modality preference alone resulted in statistically significant difference in the L2 learners’ performance data. Both findings lent support to the hypothesis proposed in chapter 2, that modality predilections, as an input processing factor, could potentially modulate the extent to which the effect of captions is determined. Since preferred modality is theoretically and empirically pertinent to processing multimodal input, the following section will discuss
60
the finding vis-à-vis both Mayer’s CTML and existing empirical evidence from L2 caption research.
5.1.1 Effects of caption mode on auditory learners
The quantitative data revealed that L2 learners who were prescribed as auditory learners performed the best under the no caption condition. When compared to their captioned counterparts, the no-caption listening advantage became even more statistically salient. Therefore, it became clear that L2 auditory learners understood the audiovisual information best when immersed in no-captioned multimodal environment.
The prominent no-captioned effect manifested in the auditory learners may be accounted for by Mayer’s active processing assumption in CTML. In theory, multimodal processing requires efficient learner control in selecting the most relevant input to aid comprehension (Hasler, Kersten, & Sweller, 2007; Mayer, 2001). Similarly, L2 learning context also requires learners to the same—be efficient in allocating attention to cues that are directly beneficial to listening comprehension (Taylor, 2005).
In this case, L2 learners are most likely to depend on the input that matches their modality preference. Furthermore, Oxford (2003) hypothesized that in order to generate the most desirable learning outcome, what is preferred has to match with what is presented (Oxford, 2003). As the auditory learners in this study were found to be a case
61
in point, it is reasonable to extrapolate that the no-caption condition may be a more
favorable environment for them to optimize their multimodal listening outcome.
The observation that having no caption appears to be the optimal listening condition for the auditory learners indicates that visual support (in this case, any form of captions) disrupted, rather than facilitated auditory learners’ understanding of multimodal video content. This explains why their comprehension was impaired when captions were provided. This finding contradicts with the previous studies, where captions were found to aid the listening comprehension process (Chai & Erlam, 2008; Danan, 2004;
Markham & Peter, 2003; Sydorenko, 2010; Taylor, 2005; Vanderplank, 2010; Winke et al., 2010). In fact, prior research maintained that captions were widely used as a strategy to facilitate perceptual processing in listening (Goh, 2000). If captions were designed to facilitate the encoding of the acoustic message, why is it detrimental to the auditory learners’ listening comprehension?
One possible explanation may lie in the disturbance of L2 learners’ optimal processing channel. In the case of the auditory learners, their preferred processing channel is through what they hear rather than what they see. This speculation is established by the qualitative data reported earlier; namely when exposed to captioned video viewing environment, the presence of captions may have reoriented their attention to the visual input—the channel that is not of their processing preference.
62
Moreover, according to the questionnaire data, the auditory learners indicated that presence of captions may have divided the auditory learners’ attentional resources. With limited attentional resources at hand (Mayer, 2001), it is possible that such demand from captions imposed difficulty in processing information from their optimal and most preferred channel. Furthermore, two thirds of the auditory learners did not express positive attitude towards receiving captions during their video-watching. Both sets of data sources, quantitative and qualitative, jointly indicated that the presence of captions may not lead to the optimal multimodal processing environment for the auditory learners. This finding, in turn, helped shed light on recalibrating the listening strategy suitable for those who preferred to “listen” in a multimodally enhanced environment.
5.1.2 Effects of caption modes on visual learners
Notwithstanding the notable effect captions had on the auditory learners, it did not
Notwithstanding the notable effect captions had on the auditory learners, it did not