Study three: Reading comprehension of statistics texts

This study was the stage two of the whole research, which focused on the relationships between statistics text and readers. As stated in Section 1.4.3, the purpose of this study was to explore how different versions of modified statistics texts written in English about the concept of boxplots relate to students’ reading comprehension. The modified statistics texts were designed by taking the two components under the attribute of integrated visual and verbal information, which were conceptualized in Study Two. The components under this attribute are types of information provided in the boxplot (i.e., scaled-labeled, scaled-not labeled, and not scaled-labeled) and sequences of boxplot and the corresponding verbal information (i.e., verbal first and boxplot first). The discussions on this attribute and its components were elaborated in Section 5.1.2.5. Accordingly, the research questions for Study Three stated in Section 1.4.3 were modified as follows:

1. How the information provided in boxplot (scaled-labeled, scaled-not labeled, not scaled-labeled) and different sequences of verbal and the referred boxplot (verbal first, boxplot first) affect student reading comprehension of statistics reading materials?

2. How the information provided in boxplot (scaled-labeled, scaled-not labeled, not scaled-labeled) and different sequences of verbal and the referred boxplot (verbal

100

first, boxplot first) affect student reading comprehension of statistics reading materials, controlled by prior knowledge in statistics and competency in English?

3.3.1 Participants

173 pre-service EFL teachers from four different universities in Aceh Province, Indonesia, participated in this experimental study. In addition, there were also 117 pre-service EFL teachers involved in the piloting stage of the instruments development. That is, for designing texts, English competency test, and test for reading comprehension of statistics texts. At the time this study was conducted, the students were in their second or third year who have taken or were taking introductory statistics course.

3.3.2 Design

The experiment had two factors. The first factor, information, was a between-subjects variable with three levels, whether the information provided in the boxplot was scaled-labeled, scaled-not scaled-labeled, or not scaled-labeled. The second factor, sequence, was a within-subjects variable with two levels, whether the sequence of verbal information and the referred visual (i.e., boxplot) was verbal first or visual first. This design resulted in six experimental groups with the number of participants in each condition ranged from 26 to 31. Participants were randomly assigned to conditions.

3.3.3 Materials

There were four instruments included as materials in this study: statistics reading materials, test for reading comprehension of statistics texts, test for prior knowledge in statistics, and test for English competency. All the instruments, except the test for reading competency, were developed and piloted at the same time with the same students. Meanwhile, the instrument for prior knowledge in statistics was developed

101

and validated concurrently with the research in study one. The details processes of instrument development would be described in the followings.

3.3.3.1 Statistics reading materials

The topic included in the reading material was related to boxplots as the statistical graph for representing distributions. In designing the reading materials, the suggestions from study of Lem et al. (2015) about how to present boxplots in learning materials to reduce students’ common misinterpretations were followed. The suggestions included presenting multiple representations (MRs) by aligning histogram with boxplot and refutational texts about area misinterpretation in a warning box.

Underlying assumptions. The statistics learning material introducing boxplots within the topic of data distribution was modified into six versions. These versions were differed by taking the two components under the attribute of integrated verbal and visual information from the framework of accessibility of statistics text (see Section 5.1). The two components applied in modifying the statistics text are: (1) the sequences of boxplots and the corresponding verbal information; and (2) the verbal information provided in the boxplots including label and data scale.

There are two assumptions taken as the basis in modifying texts. First, students who read the statistics learning material presenting the boxplots first before the corresponding verbal information would extract the information from the boxplots and then use it as scaffolds in processing the subsequent verbal information (Eitel, Scheiter, Schüler, Nyström, & Holmqvist, 2013; Molitor, Ballstaedt, & Mandl, 1989;

Schnotz et al., 2014), hence have better comprehension. Second, since too many information to be processed in a box plot may reduce students’ working memory capacity, the labeled-scaled boxplots would benefit students’ comprehension.

Moreover, comprehending texts in second language takes more difficult processes

102

and requires more efforts, providing more informative graph might reduce these efforts (Freedman & Shah, 2002). Visual-verbal sequence facilitates effective reading strategy, whereas scaled-labeled boxplots may reduce efforts in comprehending the boxplots. Thus, the combination of visual-verbal sequence and scaled-labeled boxplots could be assumed to facilitate comprehension. Students who receive the learning texts with visual-verbal sequence and scaled-labeled boxplots might have better comprehension than students who would receive learning material with verbal-visual sequence and not labeled or not scaled boxplots.

Categories for modifying texts. A statistics learning material introducing boxplot was designed which contains learning goals and material contents. The material contents include the discussions on introduction to the function and components of boxplot, properties of boxplot, and interpreting distribution using boxplots. These contents can be categorized into the three components of statistical cognitions (i.e., statistical basic knowledge, statistical reasoning, and statistical thinking). Discussions with an expert mathematics education researcher were conducted to confirm the content validity of the designed text. In addition, the face validity of the text was also performed by inviting one statistics lecture and one English lecturer to discuss the compatibility of the texts, regarding its contents and language used, for second year of Indonesian pre-service EFL teachers. Three pre-service EFL teachers were subsequently invited to read the texts and were interviewed regarding the difficulties they encountered when comprehending the text. Several revisions were done including words used and sentence structures. This revised text was also used along with the reading comprehension test in piloting stage for the purpose of item analysis. The text was then modified into six different versions based on the text and graph sequences and information provision in the boxplots (see Table 3.3.1). The six versions of text can be found in Appendices C1 to C6.

103 Table 3.3.1. The six versions of statistics texts

Sequence

Information

Scaled-Labelled

Scaled-Not Labelled

Not

Scaled-Labelled

Text-Graph Version 1 Version 3 Version 5

Graph-Text Version 2 Version 4 Version 6

3.3.3.2 Reading comprehension test

Three types of test items were designed based on the definitions of each component of statistical cognitions (Garfield et al., 2010; Garfield & Franklin, 2011), which include statistical basic knowledge, statistical reasoning and statistical thinking. Statistical basic knowledge items assess knowledge about boxplot and its components.

Students were asked to identify the five elements of a given boxplot (minimum, Q1, median, Q3, maximum); to define the meaning of Q1 for the given problem context;

and to state what percentage of data is presented in a specific part of the given boxplot.

Statistical reasoning items assess the ability to interpret information from a given boxplot using the knowledge about the meaning of components of boxplot. Students were asked to state whether the given statements were either true, false or undetermined, which can be answered by understanding the meaning of minimum value, Q3, and median values. Statistical thinking items assess the ability to understand the limitations of boxplot which can influence result. Students were asked to state whether the given statements were either true, false or undetermined, which can be answered if they understand about the limitations of boxplot and their implications.

On the other hand, Yang and Li (2016) proposed three categories of reading comprehension of mathematical texts which relate to three cluster of task complexity for assessing mathematical literacy in PISA (OECD, 2004). The categories are retrieving or recognizing, interpreting or connecting, and reflecting or reasoning. By

104

considering the design of study, in which participants answered the test after reading text without provided chance to refer back to text, the first two categories were adapted in this study as the second dimension in defining the structure of reading comprehension test items. The content validity of this structure was conducted through discussions with an expert mathematics education researcher. The structure of designed items regarding the statistical cognitions and reading task categories is given in Table 3.3.2. The face validity of the instrument was conducted at the same time and with the same three pre-service EFL teachers who were interviewed in the text design validation process. After some revisions, thirteen revised questions were included in the test (see Appendix C.7). The Cronbach alpha of reliability coefficient for this instrument was 0.605 for retrieving and 0.620 for interpreting.

Table 3.3.2. Structure of test items for measuring reading comprehension of statistics texts

In developing the instrument for measuring students’ prior knowledge in statistics, four steps were involved. In the first step, content validity for instrument items were assured by referring to related literatures in statistics education. Garfield and Ben-Zvi (2008) suggested two basic concepts are required for learning data distributions, i.e., the concepts of data and variable and the concept of data display. Therefore, the two

105

concepts were involved in designing the instrument for measuring students’ prior knowledge related to data distributions. On the other hand, since statistics assessments should include the three components of statistical cognitions (Garfield

& Franklin, 2011), the statistical basic knowledge, statistical reasoning, and statistical thinking are assigned as another dimension for designing the test items. Based on this criteria, 16 main questions with two or three sub-questions for each main question were developed as the initial version of instrument. All the items were in form of open-ended questions, to which students were required to write down their answer. In the second step, this initial instrument was administered to 15 pre-service EFL teachers who were taking introductory statistics course. To investigate the difficulties students might encounter when answering the items, three of these students were invited for semi-structured interviews with the researcher. The third step was conducted to revise the initial instrument. That is, by taking into consideration the various answers obtained in step two, the multiple choice options were designed for each item. Besides, several sentences of the items were revised when necessary and some items were omitted. The discussions with a mathematics educator were conducted during these processes and two experts who were university professors in the statistics were asked for expert validating process. Then, the instrument was revised accordingly which resulted in 20 multiple choice questions. The last step was piloting the revised instrument, in which 62 pre-service teachers majoring in EFL and Islamic education were involved. Exploratory factor analyses and reliability measure were executed for the data obtained in this stage, which resulted in a 15-item instrument.

The revised instrument containing 15 items was subsequently validated with 348 pre-service teachers majoring in EFL, mathematics, and physics. Confirmatory factor analyses and scale reliability were measured, which confirmed the validity of the instrument. Table 3.3.3 presents the structure of test items based on categories

106

of the three components of statistical cognitions. Some items could be assigned into more than one components of statistical cognitions. The final version of the instrument for measuring prior knowledge in statistics is provided in Appendix C.8.

Table 3.3.3 Structure of test items for the instrument of prior knowledge in statistics

Cognition Category Item

Knowledge Reading information in a graph S1 Understanding the idea for constructing graph

S2*, S3, S8, S10*

Describing statistical concept from the type of data

Reasoning Analyzing the given phenomena by selecting appropriate graph

S2*, S10*

Predicting the phenomenon from the given graph

S12*, S13(A), S14(A)

Thinking Thinking to select suitable graph for the given question

S2*, S5, S10*

Thinking what variable needed to answer the given question

S9, S15(A)

Note: (*) the items can be assigned into more than one components of statistical cognitions

3.3.3.4 Competency in English

The test items including in the instrument for measuring students’ competency in English were taken from TOEFL preparation test for pre-service EFL teachers. In the initial process of selecting the items, a graduate student majoring in teaching EFL was asked to answer the items and then interviewed to investigate her difficulties and to discuss about the feasibility of the items for pre-service EFL teachers in the second

107

year. After some revisions, an 18-item instrument was resulted and piloted with 67 pre-service EFL teachers. Based on the analysis results of data obtained from the pilot study, 15 items remained in the instrument. The structure of the 15 items is presented in Table 3.3.4. The items involved in the instrument for measuring competency in English is provided in Appendix C.9.

Table 3.3.4 Structure of test items for English competency instrument

Competency Type of question Items

Grammar

Comparison E1, E3

Adverb E4

Passive voice E8

Adjective E9

Clauses E2, E5, E6, E7

Reading

Detail E10, E14

Reference E12

Inference E11, E13, E15

108

109