Development and validation of conceptions of statistics instrument

4.2.1 Overview

The findings from the phenomenographic study on conceptions of statistics (section 4.1.3) were utilized to develop the Conceptions of Statistics Instrument (CSI). Initially, students’ statements were referred to in constructing CSI items. For instance, the

132

statements “in statistics we learn about numbers, like mean, median, modes...” and

“statistics is to calculate something...” given by S11 and S12 were transformed into a single statement: “Statistics is all about calculating some values such as mean, median, and standard deviation.” The transformed statements were translated from Bahasa Indonesia to English and validated by a non-Indonesian speaker author to ensure proper usage of terms to reflect the identified category. Accordingly, some statements were revised. For instance, the statement “In statistics we learn about how to present information in graphs,” which was initially classified into the graphing category by one author and into the representing category by another, was divided into two different statements: “In statistics we learn about how to present data in graphs.” for the graphing category and “Statistics is about data which provide information” for the representing category. Finally, four to five items remained in each category. Each item requires respondents to indicate their level of agreement on a 5-point scale, ranging from 1 (strongly disagree) to 5 (strongly agree).

Two-stage procedure was applied to validate CSI. The first was the piloting stage in which item analyses were used to improve the validity of the initial draft of CSI for further investigation; the second was the modeling stage in which exploratory and confirmatory analyses were applied.

4.2.2 Piloting stage

The initial draft of CSI was piloted with 63 pre-service teachers majoring in mathematics and Islamic religion education. The two majors were selected not only to provide convenient sampling but also to confirm external validity. That is, despite the item scales being built upon the study of EFL majors, the instrument was expected to be applicable for other majors. Although the sample size used in this stage might be questioned, the Kaiser-Meyer-Olkin (KMO) index of .74 indicates feasibility of running factor analysis (Field 2000). Moreover, the minimum sample size

133

recommended by Mundfrom, Shaw, and Ke (2005) was 60 for a three-factor solution with low communalities when the ratio of variables to factors is eight. Nonetheless, this limitation was considered and Exploratory Factor Analysis (EFA) was run again at the modeling stage. In order to increase the explanatory power of the total variance, item reduction was conducted for those with loading scores below .40 (i.e., C1, C2, C21, C22, C23, C24, C25, and C26). This resulted in three factors containing five, five, and eight items, respectively, which explained 51.9% of the total variance.

4.2.3 Modeling Stage

Data gathered from the three majors of pre-service teachers, EFL, Physics, and Mathematics (n = 116), were used for EFA at this stage by considering that the different backgrounds of pre-service teachers may result in variations of structures of conceptions (Parpala, Lindblom‐Ylänne, & Rytkönen, 2011). Prior to EFA, an ANOVA test was performed to compare mean scores of each factor found at the piloting stage among the three groups of data. The resultant statistic for each factor (i.e., the first factor: F(2, 113) = 1.59; p = .21; the second factor: F(2, 113) = 1.79; p = .17; the third factor: F(2, 113) = 4.78; p = .10) showed no statistically significant difference in the mean factor scores among the three groups of pre-service teachers. Thus, the EFA were then run with all data from the three groups.

Several tests for basic assumptions were executed, including KMO measure of sampling adequacy and Bartlett’s test of sphericity. KMO statistics ranged between 0 and 1, where values greater than .50 are recommended as the least acceptable, while the significant Bartlett’s value indicates appropriate factor analysis (Field, 2000).

KMO value of .72 and significance (p < .00) for Bartlett’s test of sphericity resulting from our analyses indicated that factor analysis was appropriate. Furthermore, the diagonal elements of anti-image correlations that are greater than .5 and the very small off-diagonal elements indicated the adequacy of the sample for each variable

134

and a good model (Field, 2000). EFA yielded six factors with eigenvalues greater than 1.0, while examination on a scree plot suggested two factor solutions. Hence, multiple factor analyses were run by setting the number of factors extracted at two, three, four, five, and six as suggested by Costello and Osborne (2005). Eventually, the one

‘cleanest’ factor structure was obtained, i.e., a 3-factor solution with 18 items.

Nonetheless, there were two items that did not belong to the assumed factor: C3, which loaded on factor 1 when it should have been on factor 3; and C12, which loaded on factor 3 when it should have been on factor 2. A possible reason for the inconsistencies is the multiple information contained in these item statements. The item C3 might have led students to focus on ‘how to analyze data,’ which might have been understood as ‘calculation’ by some students rather than as ‘making decisions in life.’ The item C12 might have led to the focus on context for ‘future profession’

rather than ‘doing research.’ Since three items for each factor are acceptable (Costello

& Osborne, 2005), C3 and C12 were eliminated from further analysis.

This 16-item CSI was subsequently administered to 232 other pre-service teachers majoring in mathematics (n = 112) and EFL (n = 120 EFL). Table 4.2.1 shows the results of confirmatory factor analysis (CFA), confirming the factor structure of CSI found in the exploratory stage. According to Hair, Anderson, Tatham, and Black (1998), the Cronbach’s alpha result of .60–.70 was at the lowest limit of acceptability for the internal consistent reliability coefficient based on correlation between variables.

Because the Chi-square test is sensitive to sample size, the goodness-of-fit index (GFI), the adjusted goodness-of-fit index (AGFI), the comparative fit indexes (CFI), and the root mean square error of approximation (RMSEA) were also conducted to compensate for sensitivity. A relative chi-square of less than 3 can indicate a good fit;

a rule of thumb for values on GFI and AGFI is that values close to 1 (over .90) are considered representative of a well-fitting model; a CFI greater than .80 is sometimes

135

permissible, while an RMSEA of less than .08 is considered acceptable (Hu & Bentler, 1999). The CSI administered to participants in this study was in Indonesian language which can be found in Appendix A.3. The English translation version of CSI is provided in Appendix A.4.

Table 4.2.1 Confirmatory Factor Analysis Loadings Factor

C14 Statistics is about how to make graphs and tables .73 C19 Statistics is all about calculating some values such

as mean, median, standard deviation .60

C20 In statistics we learn about how to present data in

graphs .75

Factor 2

Pure Methodology

(0.69) C4 In statistics we learn about the way to investigate

about some issue .61

C6 Statistics is a tool for analyzing data found from a

research .48

C10 Statistics is about a collection of methods for

solving problems .78

C16 Statistics is about designing a research from

finding problems to interpreting data .53 Factor 3

Methodology in Context

(0.71) C5 Statistics is a knowledge to help explain and solve

matters in the world .62

C8 Statistics is about data that provide information .46 C9 Statistics is about using data to confirm some

issue .45

C11 Statistics is something we can use to understand

some daily life situations .60

C15 Statistics is about a set of data which correspond

to a problem .49

C17 Statistics can be used to evaluate information .67 C18 There are a lot of things in my daily life connected

to statistics .65

136

Figure 4.2.1 Three-factor structure of Conceptions of Statistics of Indonesian Pre-service Teachers (N = 232)

This process identified three factors that were then interpreted and named as technique, pure methodology, and methodology in context. The technique factor consisted of five items in which statistics was conceived as working with data by following particular procedures. The four items in the pure-methodology factor represented statistics as a method for performing investigation and analyzing data to solve problems. The third factor, methodology in context, consisted of seven items related to conceptions of statistics, as understanding and handling daily life problems

137

and evaluating information. Figure 4.2.1 depicts the fitted model for conceptions of statistics with standardized solutions.

4.2.4 Discussions

4.2.4.1 The meaning of three factors of conceptions of statistics

The main objective of this study was to develop and validate an instrument for measuring students’ conceptions of statistics. Factor analyses indicated that the three-factor structure of CSI derived from 16 items was a good fit to the sample data, which supports the contention that students conceive statistics in qualitative different ways. Moreover, it was found that the three factors could be confirmed back to the six categories from the phenomenographic phase (section 4.2).

The first factor, technique, corresponded to the calculating and graphing categories. The second factor, pure methodology, corresponded to the analyzing category and several items representing the investigating category. The third factor, methodology in context, corresponded to some aspects of the investigating category, in addition to the representing and thinking categories. Despite the correspondences, it is acknowledged that great differentiations in conceptions of statistics which have been generally proposed in the previous phenomenographic studies cannot be revealed using CSI.

4.2.4.2 Conceptions of data

In addition, while delving into the item statements of each factor of CSI, different conceptions of data underlying conceptions of statistics were noticed. Further exploration into students’ statements from phenomenographic phase was then undertaken to justify these differences. As the results, three types of conceptions of data were found. First, it was identified from the five items loaded in the technique factor that statistical data was acknowledged simply as a set of numerical numbers.

138

This type of conception of data emerges from the calculating and graphing categories.

Although they might be aware about the data contexts, students in these two categories overlooked the connection of the contexts to the procedures performed.

Such conceptions of data may limit students’ ability to interpret data, which is among the components of statistical reasoning skills (Garfield & Ben-Zvi, 2004).

Second, some items in the pure-methodology factor disclosed the conception of data as a set of numbers in problems, which served as the object to be analyzed using methods for solving problems (see C6, C10, and C16 in Table 4.7). This function of data is somewhat similar to that in the previous type, except that, in this type, students were aware of the connection of problems to be solved with the analysis performed. This type of conception was also found in the analyzing category, in which problems were specified as statistical tasks. Thus, context is still ignored in this type of conception of data, inhibiting the validity of the analysis that needs to be meaningful in the context.

Last, some items in the methodology-in-context factor revealed that data was conceived as information for investigation in real contexts (see C8, C9, and C15 in Table 4.7). This type of conception also emerged in the categories of representing, investigating, and thinking. Students in the three categories perceived data as numbers both in context and in the investigation process, where data can be viewed as the best representative and then transformed as the result. Data in the investigation process can be used not only as a means for supporting hypotheses but also as the object for criticizing and evaluating the processes involved in generating the data.

This conception of data seems to move toward critical thinking on quantitative information, which is one component of statistical literacy (Gal, 2004).

The three types of conceptions of data, emerging from our study on conceptions of statistics, support the important role of data in learning statistics (e.g.,

139

Konold et al. 2015) and developing statistical thinking (Scheaffer, 2006). The types and importance of conceptions of data have not been identified in previous studies related to conceptions of statistics (e.g., Gordon, 2004; Petocz & Reid, 2005; Yang, 2014). Konold et al. (2015) has acknowledged four perspectives on statistical data—

pointers, case values, classifiers, and aggregates. However, the three types of conceptions of data are different from the four perspectives. The former is closer to the construct of beliefs, while the latter is closer to the construct of cognitions.

4.2.4.3 Limitations of study

There are two main limitations of the current study which should be taken into consideration for further study. First, the participants involved in this study were pre-service teachers majoring EFL, Islamic religion, mathematics, and physics, who were from Aceh Province, Indonesia. Due to the limited sample size and geographical site of the participants, the generalization of the results to the entire Indonesian pre-service teachers may be restricted. Further study could be extended to involve more major subjects undertaken by pre-service teachers in different geographical sites of the country so that the results could be discussed in stronger and deeper meanings.

Second, some statistics in this study may not indicate good reliability and validity of CSI. The values of Cronbach’s alpha ranging from .69 to .71 could lead us to question the fitness of the model, although these values are considered acceptable by some scholars (e.g., Hair et al., 1998). Similarly, the CFI of .89, which was sometimes permissible, could be questioned. Further study is required if one aims to establish the predictive validity of CSI and to improve its external validity in different cultural contexts (Chan & Elliott, 2000).

140 4.2.5 Conclusion

The 16-item CSI for assessing Indonesian pre-service teachers’ conceptions of statistics has been developed and validated in this study. The CSI can contribute by filling a gap in statistics education literature, since no reference has been found related to quantitative instruments for measuring conceptions of statistics. The CSI provides a useful tool for statistics teachers and researchers who aim at efficiently obtaining information about students’ conceptions of statistics. Additionally, it can be used to investigate the relationships between conceptions of statistics and other constructs such as attitudes toward statistics, as suggested by Gal, Ginsburg, and Schau (1997). The 16-item CSI for assessing Indonesian pre-service teachers’

conceptions of statistics has been developed and validated in this study.

Furthermore, the qualitative and quantitative studies were integrated to identify three types of conceptions of data that emerged from conceptions of statistics.

This identification may open another research field within this area, such as how students’ conceptions of data can predict their conceptions of statistics and influence their reasoning about data. Revealing the relationship might explain students’ limited perspectives when reasoning about data.