The study involved three independent variables: (1) multimedia supports (the aforementioned six major supports); (2) literacy purpose (acquisition vs. comprehension); and (3) presentation mode (simultaneous vs. sequential). Literacy purposes and presentation modes are the between-subject independent variables, multimedia supports are the within-subject variables, and total fixation duration is the dependent variable. Accordingly, a 6 (multimedia supports) x 2 (literacy purposes) x 2 (presentation modes) mixed ANOVA was applied to the eye-movement data in hopes of determining whether the relative effects of the six multimedia supports varied significantly between the SmV, SmC, Sqv, and SqC groups.
29
CHAPTER FOUR RESULTS
The main effect of the multimedia supports variable was statistically significant [F = 45.11, p < .001, η2 = 0.48]. This means that the total fixation duration (TFD henceforth), in milliseconds (ms), of the multimedia supports significantly differed regardless of presentation mode and literacy purpose.
The main effect of the literacy purpose variable was statistically significant [F = 13.29, p
< .01, η2 = 0.22]. This entails that the TFD significantly differed when the participants read for vocabulary acquisition and for comprehension. The TFD for all multimedia supports was significantly higher when reading for vocabulary acquisition than for comprehension (M = 633882 ms vs. M = 403419 ms, respectively, p < .01). This indicates that more fixation time was spent on (some) multimedia supports when reading for vocabulary acquisition than for comprehension.
The main effect of the presentation mode variable was statistically significant [F = 9.49, p
< .01, η2 = 0.17]. This means that the TFD for all multimedia supports significantly differed between the simultaneous and sequential presentation modes. The TFD was significantly higher when reading under the sequential presentation mode than the simultaneous presentation mode (M
= 610780 ms vs. M = 426521 ms, respectively; p < .01), suggesting that the participants seemed to spend more time processing (some) multimedia supports when reading under the sequential mode than under the simultaneous mode.
There was a significant interaction between the multimedia supports and literacy purpose variables [F = 14.41, p < .001, η2 = 0.23]. The simple main effects analyses indicated that the TFD of the six multimedia supports significantly differed between reading for vocabulary acquisition and for comprehension purposes (M = 633882 ms vs. M = 403419 ms, respectively; p < .001), with
30
TFD being much longer when reading for acquisition. Post-hoc comparisons were conducted to further testthe simple main effects, which showed that the TFD for glosses was significantly higher than other supports when the participants were reading for vocabulary acquisition (M = 9055 ms, SD = 6550 ms) than for comprehension (M = 5057 ms, SD = 3410 ms), with p < .01. Similarly, the TFD for vocabulary focus was also significantly higher than other supports when the participants were reading for vocabulary acquisition (M = 24931 ms, SD = 18297 ms) than for comprehension (M = 8316 ms, SD = 7937 ms), with p < .001. This means glosses were fixated slightly longer than vocabulary focus, but there was no significant difference between the two. This also entails that glosses and vocabulary focus are two major supports that the participants relied on when reading for acquisition. The TFD for illustrations was significantly higher when the participants read for comprehension (M = 3732 ms, SD = 2406 ms) than for vocabulary acquisition (M = 2401 ms, SD
= 2184 ms), with p < .05; this finding suggests that illustrations are the key support that the participants drew on when reading for comprehension. The aforementioned input processing preferences under the two literacy modes are collaborated by the collapsed heat maps of the participants’ fixation durations across all pages when reading for vocabulary acquisition (Figure 5) and comprehension (Figure 6).
31 Figure 5
Collapsed heat map of the participants’ total fixation duration across all pages when reading for vocabulary acquisition
Figure 6
Collapsed heat map of the participants’ total fixation duration across all pages when reading for comprehension
32
The participants’ differential attention to the multimedia supports under investigation was also observed in post-hoc comparisons under different presentation modes. Specifically, the analysis also showed that there was a statistically significant interaction between the multimedia supports and presentation mode variables [F = 6.98, p < .01, η2 = 0.13]. Simple main effects analyses showed that the TFD of the multimedia supports significantly differed between the simultaneous and sequential presentation modes, with the latter receiving higher TFD than the former (M = 426521 ms vs. M = 610780 ms, respectively, p value < .01). Further testing of the simple main effect showed that the TFD for vocabulary focus was significantly higher under the sequential presentation mode (M = 22804 ms, SD = 15630 ms) than the simultaneous presentation mode (M = 10443 ms, SD = 14751 ms), with p < .01. This indicates that reading under the sequential mode invoked the participants to fixate longer on the vocabulary focus – the most prominent multimedia support they relied on when reading for vocabulary acquisition. On the other hand, no significant differences were found between multimedia supports under the simultaneous presentation mode, suggesting the simultaneous mode allows the participants to evenly distribute their attention toward all supports. Figures 7 and 8 are collapsed heat maps of the participants’ fixation durations across all pages when reading under the sequential and simultaneous presentation modes, respectively.
33 Figure 7
Collapsed heat map of the participants’ total fixation duration across all pages when reading under the sequential mode
Figure 8
Collapsed heat map of the participants’ total fixation duration across all pages when reading under the simultaneous mode
34
Finally, post-hoc analysis did not find any interaction between the literacy purpose and presentation mode variables (p value > .70). There was also no interaction between the multimedia support, presentation mode, and literacy purpose variables (p value > .70). Table 1 below presents the interaction results noted above.
Table 1
Summary of interaction results between multimedia supports, literacy purpose, and presentation mode
35
CHAPTER FIVE
DISCUSSION AND CONCLUSION 5.1 Discussion
The goal of this study was to examine whether literacy purpose (1st research question) and presentation mode (2nd research question) influence what and how L2 learners process micro-level and macro-level multimedia supports. The ensuing paragraphs will discuss findings in relation to the two research questions and VanPatten’s (2004) Input Processing Principles in the context of e-book reading.
In response to the first research question, micro-level supports – glosses and vocabulary focus – were consciously heeded to for a significant amount of time when the participants read for vocabulary acquisition, regardless of presentation mode. On the other hand, macro-level supports – illustrations – were heeded to significantly longer when the participants read for comprehension, regardless of presentation mode. Two interesting observations emerge from the above holistic picture and deserve further discussion:
1st observation: Differential attention to the three micro-level supports when reading for acquisition (vocabulary focus = glosses > footnotes). Reading for vocabulary acquisition guided the L2 learners to pay significantly more attention to vocabulary focus and glosses, but not to another micro-level support under investigation (i.e., footnotes). This fixation pattern is clearly shown in the collapsed heat map of fixation durations of the participants reading for vocabulary acquisition (See Figure 5 above). The heat map indicates that the intensive fixations were mainly attributed to vocabulary focus and glosses; but footnotes were not intensively processed to the same extent.
36
In regard to the first observation, researchers have claimed lexical knowledge – such as those provided in vocabulary focus and glosses – is not only beneficial for word-level acquisition, but also for global comprehension, and there may even be a causal connection between the two (Nassaji, 2006; Lervåg & Aukrust, 2010; Qian, 2002). Notably, of the two micro-level supports, vocabulary focus, in particular, was consistently processed for a significant amount of time among all multimedia supports when analyzed under both the literacy purpose and presentation mode variables. This suggests that the participants consistently deemed vocabulary focus as a key contributing factor to their understanding of the text regardless of whether or not this leads to better reading outcomes.
What deserves further explanation is why footnotes did not receive significant amount of attention from the participants compared to vocabulary focus and glosses. In fact, as noted in the result section, footnotes, in general, received the lowest amount of conscious processing time among not only the micro-level supports, but also among the macro-level supports, regardless of literacy purpose and presentation mode. This statistical analysis was confirmed by the heat map of the eye fixation duration data. The heat map in Figure 5 above indicated that the participants rarely processed the footnotes (as gleaned by the lack of fixation duration on footnotes compared to all other supports). The discrepancy between the amount of fixation time on footnotes and on the other two micro-level counterparts (vocabulary focus and glosses) might have to do with what has been included in the footnotes (nature of information) and how the micro-level information in footnotes is positioned on a digital page (layout). Specifically, glosses and vocabulary focus provide direct foreground (lexical) information, whereas footnotes incorporate indirect background information of specific words; therefore, footnotes may not have directly facilitated
37
the participants’ lexical knowledge for acquisition purposes. In terms of layout, AbuSeileek (2011) found micro-level supports located at the bottom of a text (as in the case of footnotes) were significantly outperformed by other multimedia supports located closer to the text. In this vein, the distance between footnotes and the text may impose greater cognitive load due to split attention effect. Split attention effect occurs when the participants need to divide their conscious attention between two physically separated information in order to integrate them into their learning process;
thus increasing their cognitive load and diminishing their learning efficacy (Sweller, 2006).
2nd observation: The participants paid more attention to the three macro-level supports when reading for comprehension, but to different degrees (illustrations > photos >
infographics). Reading for comprehension induced the L2 learners to intensively attend to illustrations, but not the other two macro-level supports (infographics and photos). Figure 6 above – a collapsed heat map of fixation durations of the participants reading for comprehension – shows that illustrations were intensively fixated on, whereas infographics and photos were rarely scanned.
In regard to the second observation noted above, the participants spent significantly more time fixating on illustrations than on photos when reading for comprehension, indicating a higher reliance on illustrations to assist their understanding. The differential effects of illustrations and photos can be attributed to the nature of the pictorial format of the two macro-level supports.
Specifically, although both illustrations and photos convey information in pictorial format, photos include real-life snapshots, and perhaps self-evident visual details, which would therefore be less effective in prompting the learners to process further. Illustrations, on the other hand, include less
38
visual details and may be open to different interpretations than real-life photos. This ‘insufficiency’
or ‘ambiguity’ probably has the potency to induce the participants to ‘fill in the gap’ with their schemata and prior knowledge, and strive to interpret the (macro-level) comprehension supporting information conveyed by illustrations, thereby enhancing the depth of input processing and fostering stronger memory traces for the information read (see Ally, 2012; Carpenter & Olson, 2012). In an informal post-study debriefing session, several participants clearly indicated that they made mental effort to ‘study’ the illustrations and thought about ‘the relationship between the illustration and the corresponding passage’ while reading the e-book. Additionally, the graphic nature of illustrations speaks directly to universal and language independent concepts (Jones, 2004). In this regard, illustrations are more likely to bypass the linguistic representation and processing of information, and are directly constructive toward the graphic representations of mental models which L2 learners use when comprehending the gist of the text (Author, 2015). The aforementioned idiosyncratic features of the pictorial qualities of illustrations partially explain why illustrations are consciously processed more than other macro-level supports (i.e., photos) when it comes to understanding the gist of the text. This is confirmed by studies reviewed previously, who believe information displayed in pictorial format aids L2 learners’ overall comprehension (Al-Seghayer, 2001; Shalmani & Sabet, 2010; Yanguas, 2009).
In stark contrast to the superior status of illustrations, infographics – being not the optimal support for enhancing macro-level understanding – convey information via combination of textual and pictorial formats which require the participants to process and convey both linguistic and graphic representations of information into their mental models. According to Mayer and Anderson (1991) and Schnotz (2008), information presented in two different formats (e.g., textual and graphic) that are both processed through the same channel can compete with each other for
39
cognitive resources; thus, instigating L2 learners to split their attention in order to process and integrate them into their mental models. In the case of infographics, macro-level supporting information is simultaneously conveyed through two formats (e.g., keywords and graphic schematization), both of which are processed through a common channel (i.e., visual channel).
Since infographics are built upon the interaction between textual and pictorial information, L2 learners would need to simultaneously process both formats of information in order to grasp what the infographics are trying to illustrate. In this sense, infographics would require more conscious processing time and effort to process and comprehend, yet outcomes in this study revealed infographics received significantly less fixation time. That is, the participants – who were all advanced L2 readers – chose not to invest too much focal attention on infographics (as reflected in their fixation time). The discrepancy between the finding of this study and Mayer and Anderson’s account can be explained by the findings of Ariew and Ercetin’s (2004) study. As reviewed earlier, Ariew and Ercetin found that spending too much time on all available macro-level supports debilitates, rather than facilitates, L2 learners’ reading experiences. In face of this, the advanced L2 learners in this study might have deliberately and efficiently invested their attention on a macro-level support that requires less cognitive effort (i.e., illustrations) over a support that appears more time-consuming or perhaps require more effort to understand (i.e., infographics). In this regard, infographics turned out to be the less-optimal multimedia support.
Although this study found literacy purpose to be a significant modulating factor in L2 learners’ reading behaviors, Author (2015) and Yanguas (2009) found results indicating otherwise.
The discrepancy may be due to the difference in proficiency level of the participants examined.
Lower-proficiency L2 learners, as examined by the above two studies, may not be as competent in reading as advanced-level learners, and would thus need to read at the word-level before reading
40
for meaning, regardless of literacy purpose (Field, 2004). However, advanced learners are relatively more competent in L2 reading and can therefore efficiently adjust their reading behaviors to take advantage of multimedia supports according to different literacy purposes; as illustrated in the current study.
In response to the second research question, the above results suggest that only vocabulary focus received significantly longer conscious attention when the participants were reading under the sequential presentation mode than the simultaneous presentation mode, regardless of literacy purpose. This suggests that the sequential presentation mode – which invoked the participants to process digital content serially – primes the participants to extract information from the type of multimedia support that is particularly beneficial in promoting vocabulary knowledge (i.e., vocabulary focus). The fact that the participants spent significantly longer time processing the vocabulary focus support – a micro-level support incorporating word definitions, example sentences, pronunciations, and parts of speech – indicates that the sequential presentation mode likely instigates L2 learners to automatically read for vocabulary acquisition purposes. This view is collaborated by the heat map of the participants’ fixation duration. Figure 7 above is a collapsed heat map of fixation durations under the sequential mode, demonstrating the relative density of conscious processing devoted by the participants. The heat map shows that the vocabulary focus, in particular the orthographic and parts of speech information, were intensively fixated on under the sequential mode. As a result of the participants’ heavy reliance on vocabulary focus, the corresponding words in text were not fixated on to the same extent, as shown in the heat map in Figure 7, indicating that vocabulary focus was effective in re-orientating the learners’ attention to the supporting information embedded within it (see the discussion on the simultaneous presentation mode below for an opposite pattern).
41
Despite the observation that sequential input presentation reoriented the participants’
attention to vocabulary focus, it did not seem to direct their attention to the other two micro-level supports (i.e., glosses and footnotes) when the other variable – literacy purpose – is not considered.
Unlike vocabulary focus which encompasses information regarding meaning, form, pronunciation, and sentence usage, glosses and footnotes only offer meanings and background information of keywords, respectively, and thus may require less processing time. However, it is interesting to note that the participants decided to deploy more attentional resources to fixate on vocabulary focus than the other two micro-level supports, which might have not required as much time to process. This intriguing finding has important implications on the underlying processes invoked by the sequential presentation mode. Specifically, vocabulary focus includes formal information about a keyword that is not available in the other two micro-level supports. This indicates that reading under the sequential mode seems to have the potency to call on the participants to attend to formal information of keywords (embedded in vocabulary focus), rather than meaning and background information of keywords – a conscious reading condition conducive to L2 (vocabulary) acquisition (see Han, Park, & Combs, 2008). The participants’ attention to the information in vocabulary focus is important in that VanPatten (2004; 2008) has noted L2 learners tend to process passage for meaning; under the default input processing predilection, the participants could have decided to read the digital text without referring to vocabulary focus. However, that the participants still deployed focal attentional resources to process the information in vocabulary focus under the sequential presentation mode is suggestive of the potency of this mode in promoting form-meaning mapping. So, when form-meaning mapping is the primary goal of e-book reading and/or when the instructor does not want the L2 learners to read digital context primarily for meaning, sequential
42
display of digital content might be the optimal presentation mode to encourage L2 learners to take better advantage of the micro-level support (i.e., vocabulary focus).
Another interesting observation under the sequential reading mode is that unlike vocabulary focus, all macro-level supports were barely processed (see Figure 7 above). This seems to indicate that reading under the sequential mode primes the participants to pay more attention to the micro-level, or word-level, supporting information than macro-level ones – a reading behavior characteristic of bottom-up reading. Koda (2010) claimed that L2 learning processes occur in a hierarchical, or sequential, manner where understanding the text at the word level (word recognitions, word definitions, and sentence structures) need to take place before global understanding or inferences are made about the entire text at the higher level. The sequential input presentation mode seems to call on the linear (bottom-up) reading processes described by Koda.
Simultaneous input presentation mode, on the other hand, seems to trigger a qualitatively different reading process. Specifically, as noted in the results section, we did not observe any significant differences in the participants’ total fixation duration data on all multimedia supports under the simultaneous presentation mode. That is, simultaneous presentation mode might have encouraged the L2 learners to consciously divide their attention among both micro-level and macro-level supports and take equal advantage of both types of supports to facilitate their reading process. As can be seen from Figure 8 above – a collapsed heat map of fixation durations under the simultaneous mode – there was no significant difference among the micro- and macro-level supports in terms of fixation intensity. Accordingly, simultaneous presentation mode appears to invoke an interactive, non-linear reading condition where both micro-level understanding of target words (bottom-up processing) and macro-level understanding of the entire text (top-down
Simultaneous input presentation mode, on the other hand, seems to trigger a qualitatively different reading process. Specifically, as noted in the results section, we did not observe any significant differences in the participants’ total fixation duration data on all multimedia supports under the simultaneous presentation mode. That is, simultaneous presentation mode might have encouraged the L2 learners to consciously divide their attention among both micro-level and macro-level supports and take equal advantage of both types of supports to facilitate their reading process. As can be seen from Figure 8 above – a collapsed heat map of fixation durations under the simultaneous mode – there was no significant difference among the micro- and macro-level supports in terms of fixation intensity. Accordingly, simultaneous presentation mode appears to invoke an interactive, non-linear reading condition where both micro-level understanding of target words (bottom-up processing) and macro-level understanding of the entire text (top-down