Construct validity of multiple achievement goals: A multitrait-multimethod approach
Abstract
The aim of this study was to examine three different instruments which have been used in research conducted in the physical domain to measure mastery, performance-approach and performance-avoidance goals. The construct validity of the assessment tools, including a determination of their convergent and discriminant validity as well as a method effect, was tested via several confirmatory factor analyses. Four hundred and fifty athletes from different sport clubs were involved in the present study. Participants were asked to fill out three
different multiple achievement goal instruments as well as two external criteria scales. The results based on CFA showed that all of the three instruments measured the same latent construct of multiple achievement goals. The convergent validity as well as the discriminant validity was supported. Evidence for a limited method effect in terms of the different measures was also provided.
Keywords: construct validity, MTMM, multiple achievement goals, convergent validity, discriminant validity, method effect.
Motivation-related issues have been one of the most popular topics in sport and exercise psychology research (Roberts, 2001). Over the past several decades, achievement goal frameworks have provided one of the main models laying the groundwork for the study of motivational processes in the physical domain. Differences in how people interpret and respond to their achievement-related activities have been found to be dependent on the achievement goal(s) emphasized. Nicholls (1984; 1989), in particular, has argued that how individuals define success and construe their level of competence underpins distinctions in achievement goals. When perceptions of high competence and subjective sport success are tied to outperforming others, people’s goals revolve around an ego/performance emphasis. In contrast, a sense of competence and personal success gained by exhibiting high effort, task mastery and learning are assumed to reflect a task/mastery goal focus. In general, research conducted in sport and other achievement contexts has revealed mastery (or task) goals to be associated with adaptive cognitive, affective and behavioral responses. The results regarding the concomitants of performance (or ego) goals have been equivocal (Biddle, Wang,
Kavussanu, & Spray, 2003; Duda, 2001; Hardy, 1997; Midgley, Kaplan, & Middleton, 2001). In order to resolve these ambiguous findings for ego/performance goals, educational researchers (Elliot & Church, 1997; Middleton & Midgley, 1997; Skaalvik, 1997) have argued for a revision of the task-ego dichotomy framework to include an approach-avoidance distinction in performance (ego) goals. When oriented toward a performance-approach goal, the focus is on the attainment of favorable judgements of normatively-defined competence whereas performance-avoidance goals reflect a concern with not demonstrating normatively-referenced incompetence. Elliot and colleagues (Elliot & McGregor, 1999; Elliot & Moller, 2003) propose that performance-approach goals tend to correspond to positive processes and outcomes and/or be unrelated to negative processes and outcomes, whereas
performance-avoidance goals are more likely to be linked to a negative achievement pattern. Elliot and McGregor (2001) have divided achievement goals in terms of their competence definition (mastery and performance) and competence valence (approach and avoidance) to further extend the three goal framework to a 2 X 2 model. This extension resulted in a new achievement goal being proposed, namely a mastery-avoidance goal that centers on not demonstrating absolute and/or self-referenced incompetence.
To date, when compared to the extensive sport literature grounded in the dichotomous model (Duda, 2001; Roberts, 2001), research based on the trichotomous and 2 X 2
achievement goal frameworks in the physical domain is limited. In studies of the concomitants of multiple goals within sport (or physical education), there are several
instruments that have been used to access achievement goals including the Achievement Goal Questionnaire in Sport (AGQ-S; Conroy, Elliot, & Hofer, 2003), the Approach and Avoidance Achievement in Sport Questionnaire (AAASQ; Cury, Laurent, de Tonnac, & Sot, 1999), and
The Multidimensional Hierarchical Model of Goal Orientations Questionnaire (MHMGO; Papaioannou, Dimitros, Kosmidou, & Nikolaos, 2002). All of these assessment tools have proposed to measure achievement goal constructs but with some differences in the
terminology employed. From a measurement-building point of view, all three scales are grounded in similar theoretical bases and have been found to possess acceptable
psychometric properties regarding their reliability and validity.
The Achievement Goal Questionnaire in Sport (Conroy et al., 2003) was adapted from the Achievement Goal Questionnaire (AGQ; Elliot & McGregor, 2001) which comprised four different achievement goal subscales accessing mastery-approach, mastery-avoidance,
performance-approach and performance-avoidance goals. The AGQ-S contains 12 items with three items per subscale. Utilizing the same competence definition (i.e., absolute or
intrapersonal competence), the mastery-approach goal subscale captures a focus on task mastery in sport and the mastery-avoidance goal subscale is assumed to tap the emphasis athletes placed on not doing worse than before. On the other hand, the performance-approach goal subscale assesses the emphasis placed on outperforming others in sport settings while the items comprising the performance-avoidance goal subscale are assumed to measure a concern with avoiding doing worse than others within the athletic milieu. At the present time, the AGQ-S is one of the most popular measures utilised in research centred on multiple achievement goals in sport (Conroy, Kaye, & Coatsworth, 2006; Conroy et al., 2003; Nien & Duda, 2008; Stoeber, Stoll, Pescheck, & Otto, 2008). In the case of a university recreational athlete sample from the US, the AGQ-S was found to have acceptable internal reliability and latent mean stability (Conroy et al., 2003). Further, Conroy and associates (2003)
demonstrated the construct validity and factorial invariance of the AGQ-S over a 3 week period.
Based on Elliot and Church (1997) and Middleton and Midgley (1997), Cury and colleagues developed the Approach and Avoidance Achievement in Sport Questionnaire (AAASQ; Cury et al., 1999). In a series of studies involving French high school students (mean age 14-15 years), the factorial validity, predictive validity and test-retest reliability of the AAASQ were examined. The AAASQ is comprised of three 5-item subscales tapping mastery, performance-approach and performance-avoidance goals. The results stemming from both exploratory and confirmatory factor analyses supported the hypothesized factor structure of the AAASQ. This instrument was also found to possess good test-retest reliability and has been used in a number of studies (Cury, Da Fonseca, Rufo, & Sarrazin, 2002; Guan, McBride, & Xiang, 2007).
Grounded in a multidimensional hierarchical model of achievement goals (MHMGO), Papaioannou and others (Papaioannou et al., 2002) have argued for the consideration of different levels of generality ranging from human action (most global) to the task specific situational level (state level) in the study of achievement goals. At the middle or what is termed the contextual level (i.e., centered on dispositional goals in a particular context) which is oriented toward the sport or physical education achievement setting, a measure (i.e., The Multidimensional Hierarchical Model of Goal Orientations-Sport) was developed based on the work of Elliot and Church (1997). The MHMGO is comprised of four different subscales accessing personal development, ego-strengthening, ego-protection and social acceptance goals. Similar to the concept of task/mastery goals in dichotomous models (Ames, 1992; Dweck & Leggett, 1988; Nicholls, 1984), personal development goals revolve around the experience of learning, task mastery and competence development. Tied to the demonstration of a normative sense of competence, an ego-strengthening goal focuses on obtaining
favourable competence judgements while an ego-protection goal reflects an emphasis on avoiding incompetence judgements (Papaioannou, 1999). The social acceptance goal is similar to the concept of social goals as introduced by Ewing (1981) and recently re-emphasized by Allen (2003; 2005)1.
The psychometric properties of the MHMGO were examined in several studies involving Greek junior and senior high school students. The internal reliability of this instrument was supported (mean α above .80) and results stemming from exploratory and confirmatory factor analyses also supported the presumed underlying four-factor model. Several recent studies have also utilised this questionnaire (Papaioannou, Ampatzoglou,
1 The original terminology used when referring to subscales of the MHMGQ has recently been changed to mastery (personal development), performance-approach (ego-strengthening) and performance-avoidance goal (ego-protection; Papaioannou et al., 2008)
Kalogiannis, & Sagovits, 2008; Papaioannou, 2006; Papaioannou, Milosis, Kosmidou, & Tsigilis, 2007) when examining motivational processes in the physical domain.
Achievement goal research has made a major contribution to our understanding of achievement-related processes and outcomes in the physical domain as well as in the classroom setting (e.g., Elliot & Moller, 2003; Roberts, 2001). Recently, in order to address the current controversy regarding the nature and impact of goals within the educational domain, Grant and Dweck (2003) examined the implications of mastery and performance goals in a series of studies. The results of their research suggested that the impact of
achievement goals is dependent on how they are operationalized. With similar issues in mind, Smith, Duda, Allen, and Hall (2002) tested the factorial validity and correlates of several multiple achievement goal instruments in the classroom context. Smith and associates (2002) found the performance-avoidance goal subscale to be tapping several constructs including impression management (Skaalvik, 1997), fear of failure (Elliot & Church, 1997) and the desire to avoid the demonstration of incompetence (Middleton & Midgeley, 1997). From a theoretical point of view, the issues raised by Grant and Dweck (2003) and the work of Smith et al. (2002) highlight the need to clarify whether the more recently created achievement goal construct (i.e., performance-avoidance goals) is a uni- or multi-dimensional concept in the achievement domain of sport.
The psychometric quality of measurement tools used in sport psychology research directly impact the quality of the results obtained. Studies, marked by a theoretical basis and the employment of established measurement tools whose validity and reliability have been psychometrically tested, are necessary to meaningful contributions to the sport psychology literature (Gill, Dzewaltowski, & Deeter, 1988). With respect to research grounded in dichotomous achievement goal frameworks, different terminology has been advocated by
sport psychology and other researchers when referring to different achievement goals (Ames, 1992; Dweck, 1999; Nicholls, 1989). Sometimes scholars use the same terminology to represent the same construct, or in some cases, different constructs. The potential disadvantage of such a lack of agreement in the labelling of constructs was previously
addressed by Marsh (1994) as reflecting Jingle-Jangle fallacies. Marsh (1994) has argued that this potential problem could be solved by examining the construct validity of different
seemingly related measures. Grounded in a multiple achievement goal framework (Elliot & Church, 1997), the purpose of the present study is to test the construct validity of three prevalent measures of mastery, performance-approach and performance-avoidance goals in the sport domain. More specifically, via a multitrait-multimethod approach, we determined the convergent validity (within factor validity or testing the factor loadings of items which are supposed to load on the underlying factors), discriminant validity (between-network validity or examining the strength of correlations between latent factors/constructs) of the three measures of multiple goals as well as unwanted method effects.
There are at least two ways to conduct multitrait-multimethod (MTMM). One tactic is to employ the traditional and well-known correlation matrix approach created by Campbell and Fiske (1959). However, with the limitations of the statistical assumptions embedded in the traditional approach (e.g., this approach does not control for measurement error), recently the employment of confirmatory factor analysis (CFA) within structural equation modelling has become a popular analytic strategy for MTMM (Cote, 1995; Marsh, 1989). To provide further evidence regarding the construct validity of the targeted questionnaires, we examined the relationship of the multiple goals assessed by each measurement tool to distinguished and well validated external criteria. The targeted external criteria were variation in the emphasis placed on task and ego goals as measured by an established dichotomous goal questionnaire,
namely the Task and Ego Orientation in Sport Questionnaire (Duda, 1989), as well as effort and ability beliefs regarding the causes of success (Duda & White, 1992).
In the present study, we hypothesized that an acceptable level of convergent validity among the targeted measured items to their hypothesized latent factors would emerge. We also expected that the discriminant validity between the different achievement goals would be supported. In addition, we expected limited method effects would emerge within each
questionnaire and across the different instruments. With respect to the expected relationships to external criteria, we predicted a task goal orientation (as assessed by the TEOSQ) to be positively associated with measures of mastery goals embedded in the multiple goal
questionnaires. We expected an ego goal orientation (as assessed by the TEOSQ) to positively relate to performance-approach and negatively relate to performance-avoidance goals. In terms of beliefs about the causes of success in sport, we expected effort beliefs to positively relate to the emphasis placed on both mastery and performance-approach goals. We also hypothesized that the endorsement of ability beliefs would positively link to both performance-approach and performance-avoidance goals.
Methods Participants
Four hundred and fifty athletes (55 % of the sample being male; Mean age = 22.17, SD = 6.59 years) from different universities and sports clubs around the UK voluntarily
participated in this study. The participants in this sample represented a variety of sports including team sports (i.e., American football, soccer, hockey, netball and rugby) and individual sports (i.e., athletics, badminton, golf, judo and swimming). A majority of the participants were European-white (91 %) and, as a group, they tended to be experienced athletes (M = 5.85 years, SD = 2.58). Ethical consent to conduct the study was obtained and
data were collected with the informed consent of the coaches and the athletes. A multi-section questionnaire, also tapping variables that are not reported here, was administered to the athletes before or after training by the principal investigator and took approximately 20 minutes to complete.
Measures
Achievement Goal Questionnaire in Sport (AGQ-S)
The AGQ-S was designed to tap athletes’ concerns with demonstrating high competence or avoiding the demonstration of incompetence in the athletic domain. Study participants responded on a scale ranging from 1 = ”not at all like me” to 7 = ”completely like me”. The AGQ-S is comprised of four subscales tapping the emphasis placed on mastery-approach goals (e.g., “It’s important to me to perform as well as I possibly can”),
approach goals (e.g., “It’s important to me to do well compared to others”), and performance-avoidance goals (e.g., “I just want to avoid performing worse than others”). In the present study, responses to the mastery-avoidance goal subscale were eliminated from further
analyses because there was no corresponding goal dimension on the other targeted measures. Further information regarding the psychometric features of the mastery-avoidance goal subscale in this study is available from the first author. Initial research has found the AGQ-S to exhibit acceptable internal consistency and has provided evidence supporting the factorial and external validity of this measure (Conroy et al., 2003).
Approach and Avoidance Achievement in Sport Questionnaire (AAASQ)
The 15 items AAASQ (Cury et al., 1999) was designed to measure three different goal orientations in the athletic domain, namely a mastery goal, performance-approach and performance-avoidance goal orientation. The stem “when I practice sport….” was used and participants responded to a five-point Likert scale from 1 (strongly disagree) to 5 (strongly
agree). Example items included “I want to learn as much as possible”/mastery goal, “It’s important for me to do better than the others”/performance-approach goal and “I try to not make mistakes because I don’t want to be taken for a weak person”/ performance-avoidance goal. Previous studies mainly involving French high school student samples (Cury et al., 1999; Cury et al., 2002) have shown the AAASQ to possess adequate internal reliability, construct validity as well as predictive validity.
The Multidimensional Hierarchical Model of Goal Orientations (MHMGO)
The 24 item MHMGO-sport (Papaioannou et al., 2002) comprised four different
achievement goal subscales tapping the importance of personal development goals (6 items), ego-strengthening goals (6 items), ego-protection goals (6 items) and social acceptance goals (6 items). Responses to the social acceptance goal subscale were removed from further analyses because there was not any corresponding goal subscale in the other targeted achievement goal measures. Following the stem “in sport”, participants responded to a 7-point type Likert scale (1 = strongly disagree, 7 = strongly agree).
Example items are “It’s my goal to always develop my skills”/ personal development goals, “I feel successful when I am the only one who can do a skill”/ ego-strengthening goals and “It’s my tendency to avoid skills and games in which I may be teased, because of my ability”/ ego-protection goals. Past research has revealed the MHMGO-sport to possess adequate reliability and validity (Papaioannou et al., 2008; Papaioannou, Milosis, Kosmidou, & Tsigilis, 2007).
Task and Ego Orientation in Sport Questionnaire (TEOSQ)
The 13 item TEOSQ (Duda, 1989) assesses individual differences in the proneness for task and ego involvement in sport. When responding to the items, participants provided their responses on a 5-point Likert type scale (1 = strongly disagree, 5 = strongly agree). Following
the stem “I feel most successful in my sport when…” athletes answered to 7 task items (e.g., I do my very best) and 6 ego items (e.g., I play better than my teammates). The psychometric properties of TEOSQ have been extensively tested and have proved to be adequate (e.g., Duda & Whitehead, 1998).
Causes of Success in Sport Questionnaire
The 11 item Causes of Success in Sport Questionnaire (Duda & White, 1992) assesses participants’ differential beliefs regarding the antecedents of sport success. When responding to the items, participants provided their responses on a 5-point Likert type scale (1 = strongly disagree, 5 = strongly agree). Following the stem “What do you think is most likely to help athletes do well or succeed in their sport?” athletes responded to 7 effort beliefs items (e.g., Athletes succeed if they like improving) and 4 ability belief items (e.g., Athletes succeed if they are better athletes than the others). The internal reliability of the ability belief subscale was found not to be acceptable (α = .44) and this subscale was removed from subsequent analyses.
Statistical analyses
Analyses were conducted on the responses of participants who completed all five questionnaires. As a result, approximately less than 3 % of the participants were eliminated due to missing data. This percentage of missing cases did not exceed the cut-off criterion of 5% as suggested by Roth (1994).
A confirmatory factor analysis approach to analysing multitrait-multimethod (MTMM) was conducted by using the AMOS software package (Arbuckle, 2005) and the maximum-likelihood estimation procedure. In order to test convergent, discriminant validity and method effects systematically, the taxonomy which was first created by Widaman (1985) and then expanded by Marsh (1989) was used in the present study. The taxonomy included the latent
trait factors which are accessed by different measures and the latent method factors which are accessed by the same instruments. This taxonomy captures several priori models such as a correlated traits model (CT model), a correlated traits/correlated uniqueness model (CTCU), a correlated traits/uncorrelated methods model (CTUM), and a correlated traits/correlated methods model (CTCM). The main focus of this approach is to test how well the different priori models fit the data. The operationalizaiton of priori trait factors and method factors may better reflect Campbell and Fiske’s (1959) intentions than their original guidelines which did not control for measurement error.
In terms of testing the convergent and discriminant validity of the targeted instruments, relevant evidence was found by comparing several a priori models mentioned above. For example, if the general trait (1T) model resulted in acceptable supplemental model fit indices, the results would be indicative of a uni-dimensional construct rather than a multi-dimensional construct. If the CTUM model was found to be adequate, then an independent method effect as well as a correlated trait effect was witnessed. The degree of convergent validity could be ascertained by examining the magnitude of indicators’ factor loadings with respect to their underlying constructs. Evidence regarding discriminant validity was provided via the examination of the observed correlations between the different trait factors (Marsh, 1989; Widaman, 1985). In addition to these priori models, the CU (correlated uniqueness) model (as introduced by Marsh and Grayson, 1995) was examined and the method effect was represented by measurement error correlations among indicators in the same instrument rather than across separate method factors. The CU model was based on the assumption of possible multi-dimensional method effects and the observation of high residual correlations provided evidence for common method variance between pairs of indicators (for more details, refer to Marsh & Grayson, 1995).
According to the well-known problem of the chi-square test being sensitive to sample size, other fit indices (i.e., CFI, IFI, TLI, RMSEA and SRMR) were also considered when making comparisons to the baseline model. According to Hu and Bentler’s (1995)
recommendations, we considered the Root Mean Square Error of Approximation (RMSEA) as a measure of absolute fit and the Comparative Fit Index (CFI) and Tucker-Lewis Index (TLI) as indices of incremental fit. Hu and Bentler (1995) propose a good fitting model to be indicated by values close to or greater than .95 for the CFI and TLI, and values of (or less) than .06 for RMSEA and SRMR. As the RMSEA is sensitive to the number of parameters considered, a different criterion was used (.08 -.10 was considered reflective of a mediocre fit; Browne & Cudeck, 1993).
Results Descriptive statistics
The relationships among the subscales of the three multiple achievement goal
instruments as well as their means, standard deviations and internal reliabilities are provided in Table 1. For the participants as a whole, the emphasis on mastery-approach/mastery goals was moderate to high compared to the observed mean for the other two goals. Among the targeted achievement goal questionnaires, performance-avoidance goal subscale scores were lower than the score observed for the performance-approach goal subscales. The Cronbach alpha coefficient found for each of the nine different subscales was above the .70 recommend level recommended by Nunnally (1978). The observed reliabilities for the MHMGO
subscales were the highest followed by the AAASQ and the AGQ-S subscales. CFA-instrument results
The results regarding the factorial validity for each of the three assessment tools are reported below. In terms of the observed factor loadings for each measure, please refer to Table 3.
The model fit indices for the AGQ-S (without the mastery-avoidance subscale) were reasonable except for the observed RMSEA (χ2[25, sample size = 430] = 134.57, p < .001, CFI = .94, IFI = .92, TLI = .94, RMSEA = .10 [.09-.12], SRMR = .08). These results suggest a poor model fit but are aligned with previous work (see Conroy et al, 2003, p. 464, Table 2). The correlations between the mastery-approach and performance-approach goals (r = .26, p < .05) as well as the performance-approach and performance-avoidance goals (r = .66, p < . 05) were significant and positive. There was no significant relationship between mastery-approach and performance-avoidance goals (r = .06, p > .05). The observed associations between the different goals were consistent with previous research (Conroy et al., 2003).
The observed Chi-square and supplemental fit indices of the AAASQ were reasonable (χ2 [87, sample size = 426] = 300.48, p < .001, CFI = .92, IFI = .91, TLI = .92, RMSEA = .07 [.06-.08], SRMR = .077). Mastery-approach goals positively related to performance-approach goals (r = .21, p< .05) but not to performance-avoidance goals (r = -.01, p > .05).
Performance-approach goals were positively correlated to performance-avoidance goals (r = . 39, p < .05).
The model fit for the MHMGQ was also reasonable (ϰ2[133, sample size = 427] = 495.66, p < .001, CFI = .92, IFI = .90, TLI = .92, RMSEA = .08 [.07-.09], SRMR = .07). The correlation between mastery-approach goals and performance-avoidance goals was negative and non-significant (r = -.09, p > .05). The relationships between the two approach goals and the two performance goals are similar to what was observed for the two former instruments (r = .22 & .38 respectively, p < .05).
CFA-MTMM results
Several models were tested via structural equation modelling. The results including overall chi-square, degrees of freedom and model fit indices (i.e., CFI, IFI, TLI, RMSEA and SRMR) are provided in Table 2. According to the suggestions of Marsh (Marsh, 1996a; Marsh, 1996b; Marsh, Bar-Eli, Zach, & Richards, 2006), convergent validity is witnessed by a large and statistically significant trait-factor loading. Discriminant validity is evidenced by large factor correlations (i.e., those approaching 1.0) which indicate structural overlap. Method effects are indicated by large and statistically significant method-effect factor loadings in the CTCM and CTUM model as well as large and statistically significant uniquenesses in the CTCU model.
The model fit indices of most of the models were found to be inadequate and did not converge in a proper solution except in the case of the CTCU model (ϰ2[522, sample size = 436] = 982.53, p < .001, CFI = .96, IFI = .96, TLI = .93, RMSEA = .05 [.041-.049], SRMR = .06). This is a quite common solution (Marsh & Grayson, 1995) and, therefore, we focused on the results of the adequate CTCU model. The factor loadings for the items comprising the three achievement goal instruments can be seen in Table 3. The significant correlated
uniquenesses of each measure are reported in Table 4.
All 42 of the factor loadings to each of the latent factors were significant (p < .05) although the magnitude of the observed factor loadings was smaller compared to their original instrument factor loadings. The magnitude of the factor loadings for the three
performance-approach subscales were similar (mean loading = .68), which the factor loadings for the mastery goal subscales ranged from .57 to .60. The performance-avoidance goal subscales were found to be marked by factor loadings which were similar in magnitude to what were revealed for the mastery goal subscales. According to our findings, evidence for
convergent validity was suggested but the results indicated that there is room for improvement in terms of this important psychometric property.
The correlations between the performance-approach and performance-avoidance goals were statistically significant (r = .57), as were the correlations between the mastery goals and performance-approach goals (r = .21). These findings are consistent with theoretical
arguments (Elliot, 1999) as well as the results from previous studies (Conroy et al., 2003; Elliot & Church, 1997; Nien & Duda, 2008) in both the educational and athletic settings. The association between the mastery goal and performance goal was nonsignificant (r = -.02, p > . 05). Although the results did not reveal correlations close to 1, the magnitude of the
correlation between performance-approach and performance-avoidance goals was medium to high. This finding provides partial support for the discriminant validity of the two
performance goal constructs.
Method effects were evidenced via the significant correlated measurement errors evident for each of the instruments. In the AGQ-S, 28 out of 36 correlated errors were significant (mean = .23). In the AAASQ, 45 out of 105 correlated errors were significant (mean = .18) and in the MHMGO, 79 out of 153 correlated errors were significant (mean = .21). These findings provided partial support for method effects for each of the questionnaires but not across the different measures, as only one proper solution was found.
MTMM with external criteria
In the present study, dichotomous achievement goals and effort beliefs about the causes of sport success were added to the MTMM design to further test the predictive validity of the targeted measures of multiple achievement goals. The results of the CFA-MTMM for the external validity criteria model are presented in Table 2. Although the overall chi-square was significant (ϰ2 [1671, sample size = 431] = 3199.92, p < .01), the supplement model fit
indices suggest that improvement was warranted (CFI = .90, IFI = .89, TLI = .90, RMSEA = . 046[.044-.049], SRMR = .08). After examining the modification indices, two problematic items from the effort belief subscale were removed due to their low factor loading to the effort belief latent factor (i.e., the loading was lower than .50; item 9 and 11). Next, those two items were removed and the model was estimated again. The overall chi-square was
significant (ϰ2 [1552, sample size = 431] = 2999.48, p < .01), and the supplemental model fit indices indicated a reasonably good fit (CFI = .91, IFI = .89, TLI = .91, RMSEA = .047[. 044-.049], SRMR = .07). The standardized correlation between the mastery and task goals and performance-approach and ego goals was .83 and .94, respectively. There was a negative correlation between the performance-avoidance and ego goals (r = -.14, p < .01). The
correlation between the mastery goal and effort beliefs was .41 (p < .01). Performance-approach and the performance-avoidance goals were not significantly related to effort beliefs (ps > .05).
Discussion
The aim of this study was to evaluate the construct validity of three different instruments accessing multiple achievement goals in the sport domain as conceptualised by Elliot and Church (1997). All three scales are held to measure three achievement goals including
mastery, performance-approach and performance-avoidance goals in the physical domain and the major thrust of the present work was to evaluate these assumptions. Based on findings from the CFA approach to MTMM, there was support for the convergent validity of the goals as tapped by the three instruments. Mastery goals were distinguished from the performance-avoidance goals and there were moderate correlations between the mastery goals and performance-approach goals. The median to high correlations between the two performance goals represented an overlap in the construct being assessed. Method effects for each of the
different instruments also emerged but no method effect across measures was observed. Finally, the findings regarding the interplay between multiple achievement goals and the external criteria of task, ego and effort beliefs provided partial support for our hypotheses.
In the present study, the internal reliability of each subscale constituting the three targeted achievement goal instruments was considered acceptable. In addition, the factor loadings to each of the latent factors were also significant and acceptable. The results
emanating from the MTMM SEM analyses provided support for the construct validity of the three instruments to access multiple achievement goals in sport. In addition, the findings regarding the convergent validity of the mastery and performance-approach goal subscales were consistent with previous research (Grant & Dweck, 2003; Smith et al., 2002) and also provided evidence for the uni-dimensionality of the performance-avoidance goal subscales in the physical domain. Significant and acceptable factor loadings for each trait factor supported a three factor multi-dimensional model of achievement goals. In sum, the results indicated that the three targeted multiple achievement goal instruments are measuring the same achievement goal constructs.
From a measurement point of view, it is not surprising that the instrument characterised by more items per subscale (i.e., the MHMGO-sport) was also found to have higher
reliability. This is in line with Cortina’s (1993) argument that as long as the number of the items in a scale increases, the observed alpha for that scale will also increase. The
performance-approach goal subscales also revealed slightly higher values on Cronbach alpha than the other trait factors. This might be explained by the emphasis on normative
comparisons in sport competition that predominates in the culture at large. It should be noted that this finding is not at odds with the literature grounded in dichotomous achievement goal
models (Duda & Whitehead, 1998; for details regarding the observed alpha for task and ego orientation subscales across studies).
From a conceptual standpoint, the findings relevant to discriminant validity indicated that a mastery goal emphasis was clearly distinguished from the importance placed on performance-avoidance goals. The performance-approach goals were correlated with the mastery goals but these associations were positive and low to moderate. Assumed to be sharing a normative definition of competence, the approach and performance-avoidance goal subscales emphasis were associated to a median to high degree (r = .57, p < . 05). The observed relationships between the two performance goals suggested a confirmation of the assumed conceptual congruence among normatively-based goals. Although they are not redundant, our results suggested that athletes view both performance goals as similar to some degree. These findings are consistent with past studies in educational and sport settings (Conroy et al., 2003; Elliot & McGregor, 2001; Papaioannou et al., 2002; Wang, Biddle, & Elliot, 2007).
Method variance is an unwanted effect when considering the psychometric properties of measurement tools. Based on the CFA-MTMM results, a limited and instrument-specific method effect existed among the three scales. The AGQ-S revealed a slightly higher method effect compared to the two other scales. This problem could be attributed to the point that the AGQ-S is marked by fewer items in each subscale. Secondly, the observed higher correlated errors suggest wording problems between subscales. Therefore, it seems prudent that the employment of any of the targeted questionnaires as a measure of multiple achievement goals in sport should be done with caution. That is, the present study raises question regarding the understanding or interpretation of the items comprising the targeted instruments and also
points to the potential limitation of the observed lower reliability of the mastery goal subscale in each of the questionnaires.
Dichotomous achievement goals and effort beliefs of sport success were added into the CFA-MTMM solution to test the predictive validity of the targeted assessments of multiple achievement goals. Taken in their totality, our findings supported Elliot’s arguments as well as the position of others regarding the expected concomitants of the achievement goals embedded in the dichotomous and trichotomous models (Dweck & Leggett, 1988; Elliot, 1997). The present results are also in line with the sport literature grounded in the
dichotomous goal model (Duda & White, 1992). Interestingly, the observed correlations between mastery and task goals, as well as performance-approach and ego goals, were high enough to suggest identical if not very similar conclusions regarding the motivational
processes and outcomes which may be tied to the respective pair of goals (task and mastery, r = .83; performance-approach and ego, r = .94, p < .01). The external criteria results
confirmed the argument proffered by Elliot who suggested that the task-ego distinction in the dichotomous model is “an approach form of motivation” (Elliot, 1997; p. 152).
In the present study, there were several models which did not converge due to improper solutions, such as the appearance of negative factor loadings (i.e., CTUM and CTCM). The observation of the non-converged model in this study could be contributed to the study design which was relatively small (i.e., 3 Traits X 3 Methods). The implications of the present
investigation may also be limited due to the emerging of the common non-converged solutions (Marsh & Grayson, 1995).
The present study provided evidence for the construct validity of the three targeted questionnaires by a contemporary and sophisticated approach to the analysis of MTMM. The convergent validity of the measures was supported and partial evidence for discriminant
validity as well as method effects emerged. The observed correlations between the multiple goals and the task and ego goals provided additional information regarding the predictive validity of the multiple achievement goal framework when applied to the sport domain.
Reference
Allen, J. B. (2003). Social motivation in youth sport. Journal of Sport and Exercise Psychology. 25, 551-567.
Allen, J.B. (2005). Measuring social orientations in sport: An examination of the construct validity of the SMOSS. International Journal of Sport and Exercise Psychology. 3, 23-37.
Ames, C. (1992). Classrooms: goals, structures, and student motivation. Journal of Educational Psychology, 84, 261-271.
Arbuckle, J. L. (2005). AMOS (Version 5.0) [Computer software]. Chicago,IL: SPSS. Biddle, S., Wang, C. K. J., Kavussanu, M., & Spray, C. (2003). Correlates of achievement
goal orientations in physical activity: A systematic review of research. European Journal of Sport Science, 3, 1-20.
Browne, W. & Cudeck, R. (1993). Alternative ways of assessing model fit. In K.A.Bollen. & J.S.Long (Eds.), Testing structural equation models (pp. 445-455). Newbury Park, CA: Sage.
Campbell, D. T. & Fiske, D. W. (1959). Convergent and disceiminant validation by the multitrait-multimethod matrix. Psychological Bulletin, 56, 81-105.
Conroy, D. E., Elliot, A. J., & Hofer, S. M. (2003). A 2 X 2 achievement goals questionnaire for sport: Evidence for factorial invariance, temporal stability, and external validity. Journal of Sport and Exercise Psychology, 25, 456-476.
Conroy, D. E., Kaye, M. P., & Coatsworth, J. D. (2006). Coaching climates and the
destructive effects of mastery-avoidance achievement goals on situational motivation. Journal of Sport and Exercise Psychology, 28, 69-92.
Cortina, J. M. (1993). What is coefficient alpha? An examination of theory and applications. Journal of Applied Psychology, 78, 98-104.
Cote, J. A. (1995). What causes estimation problems when analyzing MTMM data? Advances in Consumer Research, 22, 345-353.
Cury, F. D., Da Fonseca, D., Rufo, M., & Sarrazin, P. (2002). Perceptions of competence, implicit theory of ability, perception of motivational climate, and achievement goals: A test of the trichotomous conceptualization of endorsement of achievement
motivation in the physical education setting. Perceptual and Motor Skills, 95, 233-244.
Cury, F. D., Laurent, E., de Tonnac, A., & Sot, V. (1999). An underexplored aspect of achievement goal theory in sport: Development and predictive validity of the
approach and avoidance achievement in sport questionnaire (AAASQ). In V. Hosek, P. Tilinger, & L. Bilek (Eds.), 10th European Congress of Sport Psychology (pp. 153-155). Prague: Charles University Press.
Duda, J. L. (1989). Relationship between task and ego orientation and the perceive purpose of sport among high school athletes. Journal of Sport and Exercise Psychology, 11, 318-335.
Duda, J. L. (2001). Achievement goal research in sport: Pushing the boundaries and
clarifying some misunderstandings. In G.C.Roberts (Ed.), Advances in Motivation in Sport and Exercise. (pp. 129-182). Human Kinetics, Champaign, IL.
Duda, J. L. & White, S. A. (1992). Goal orientations and beliefs about the causes of sprot success among elite skiers. The Sport Psychologist, 6, 334-357.
Duda, J. L. & Whitehead, J. (1998). Measurement of goal perspectives in the physical domain. In J.L.Duda (Ed.), Advances in sport and exercise psychology measurement (pp. 21-48). Morgantown, WV: Fitness Information Technology.
Dweck, C. S. (1999). Self-theories: Their role in motivation, personality, and development. Philadelphia, PA.: Psychology Press.
Dweck, C. S. & Leggett, E. L. (1988). A Social cognitive approach to motivation and personality. Psychological Review, 95, 256-273.
Elliot, A. J. (1997). Integrating the "classic" and "contemporary" approaches to achievement motivation: A hierarchical model of approach and avoidance achievement motivation. In P.Pintrich & M.Maehr (Eds.), Advances in motivation and achievement (vol. 10, pp. 143-179) Greenwich, CT: JAI Press.
Elliot, A. J. & Church, M. A. (1997). A hierarchical model of approach and avoidance achievement motivation. Journal of Personality and Social Psychology, 72, 218-232. Elliot, A. J. & McGregor, H. A. (1999). Test anxiety and the hierarchical model of approach
and avoidance achievement motivation. Journal of Personality and Social Psychology, 76, 628-644.
Elliot, A. J. & McGregor, H. A. (2001). A 2 X 2 achievement goal framework. Journal of Personality and Social Psychology, 80, 501-519.
Elliot, A. J. & Moller, A. C. (2003). Performance-approach goals: Good or bad forms of regulation? Internal Journal of Educational Research, 39, 339-356.
Ewing, M. E. (1981). Achievement motivation and sport behavior of males and females. Unpublished doctoral dissertation, University of Illinois, Urbana -Champaign.
Gill, D. L., Dzewaltowski, D. A., & Deeter, T. E. (1988). The relationship of competitiveness and achievement orientation to participation in sport and nonsport activities. Journal of Sport and Exercise Psychology, 10, 139-150.
Grant, H., & Dweck, C. S. (2003). Clarifying achievement goals and their impact. Journal of Personality and Social Psychology, 85, 541-553.
Guan, J., McBride, R., & Xiang, P. (2007). Reliability and validity evidence for achievement goal models in high school physical education settings. Measurement in Physical Education and Exercise Science, 11, 109-129.
Hardy, L. (1997). The Coleman Roberts Griffith address: Three myths about applied consultancy work. Journal of Applied Sport Psychology, 9, 277-294.
Hu, L.-T. & Bentler, P. M. (1995). Evaluating model fit. In R.H.Hoyle (Ed.), Structural equation modeling: Concepts, issues, and applications (pp. 76-99). London: Sage. Marsh, H. W. (1989). Confirmatory factor analyses of multitrait-multimethod data: Many problems and a few solutions. Applied Psychological Measurement, 13, 335-361. Marsh, H. W. (1994). Sport motivation orientations: Beware of Jingle-Jangle fallacies.
Journal of Sport and Exercise Psychology, 16, 365-380.
Marsh, H. W. (1996a). Construct validity of physical self-description questionnaire responses: Relations to external criteria. Journal of Sport and Exercise Psychology, 18, 111-131. Marsh, H. W. (1996b). Physical self description questionnaire: Stability and discriminant
validity. Research Quarterly for Exercise and Sport, 67, 249-264.
Marsh, H. W., Bar-Eli, M., Zach, S., & Richards, G. E. (2006). Construct validation of Hebrew versions of three physical self-concept measures: An extended multitrait-multimethod analysis. Journal of Sport and Exercise Psychology, 28, 310-343.
Marsh, H. W. & Grayson, D. (1995). Latent variable models of multitrait-multimethod data. In R. H. Hoyle (Ed.), Structural equation modeling (pp.177-198). Thousand Oaks, CA: Sage.
Middleton, M. J. & Midgley, C. (1997). Avoiding the demonstration of lack of ability: An underexplored aspect of goal theory. Journal of Educational Psychology, 89, 710-718. Midgley, C., Kaplan, A., & Middleton, M. J. (2001). Performance-approach goals: Good for
what, for whom, under what circumstances and at what cost? Journal of Educational Psychology, 93, 77-86.
Nicholls, J. G. (1984). Achievement motivation: Conceptions of ability, subjective experience, task choice and performance. Psychological Review, 91, 328-346. Nicholls, J. G. (1989). The competitive ethos and democratic education. Cambridge, MA:
Harvard University Press.
Nien, C.-L. & Duda, J. L. (2008). Antecedents and consequences of approach and avoidance achievement goals: A test of gender invariance. Psychology of Sport and Exercise, 9, 352-372.
Nunnally, J. C. (1978). Assessment of reliability. In J.C.Nunnally & I. Bernstein (Eds.), Psychometric theory (2nd ed., pp. 225-255). Hightstown, NJ: McGraw-Hill. Papaioannou, A. (1999). Towards multidimensional hierarchical models of motivation. In
V.Hosek, P.Tilinger, & L.Bilek (Eds.), Psychology of sport and exercise: Enhancing the quality of life (pp. 45-52). Prague, Czech Republic: Charles University Press. Papaioannou, A. (2006). Muslim and Christian students' goal orientations in school, sport,
Papaioannou, A., Dimitros, M., Kosmidou, E., & Nikolaos, T. (2002). Multidimensional structure of goal orientations: The importance of adopting a personal development goal in physical education. Psychologia, 9, 494-513.
Papaioannou, A. G., Ampatzoglou, G., Kalogiannis, P., & Sagovits, A. (2008). Social agents, achievement goals, satisfaction and academic achievement in youth sport. Psychology of Sport and Exercise, 9, 122-141..
Papaioannou, A. G., Milosis, D., Kosmidou, E., & Tsigilis, N. (2007). Motivational climate and achievement goals at the situational level of generality. Journal of Applied Sport Psychology, 19, 38-66.
Roberts, G. C. (2001). Understanding the dynamics of motivation in physical activity: The influence of achievement goals on motivational processes. In G.C.Roberts (Ed.), Advances in motivation in sport and exercise (pp. 1-50). Champaign, IL.: Human Kinetics.
Roth, P. (1994). Missing data: A conceptual review for applied psychologists. Personnel Psychology, 47, 537-560.
Smith, M., Duda, J. L., Allen, J., & Hall, H. K. (2002). Contemporary measures of approach and avoidance goal orientations: Similarities and differences. British Journal of Educational Psychology, 72, 155-190
Skaalvik, E. M. (1997). Self-enhancing and self-defeating ego orientation: Relations with task and avoidance orientation, achievement self-perceptions and anxiety. Journal of Educational Psychology, 89, 71-81.
Stoeber, J., Stoll, O., Pescheck, E., & Otto, K. (2008). Perfectionism and achievement goals in athletes: Relations with approach and avoidance orientations in mastery and performance goals. Psychology of Sport and Exercise, 9, 102-121.
Wang, C. K. J., Biddle, S. J. H., & Elliot, A. J. (2007). The 2x2 achievement goal framework in a physical education context. Psychology of Sport and Exercise, 8, 147-168.
Widaman, K. F. (1985). Hierarchically nested covariance structure models for multitrait-multimethod data. Applied Psychological Measurement, 9, 1-26.
Goal Inventories
AGQ-S AAASQ MHMGO
Facet 1 2 3 1 2 3 1 2 3 AGQ-S1.Mastery-approach (.78)a 1.Performance-approach 0.27 (0.85) 1.Performance-avoidance 0.10 0.59 (0.85) AAAS Q 1.Mastery 0.58 0.20 0.14 (0.82) 1.Performance-approach 0.19 0.71 0.47 0.23 (0.89) 1.Performance-avoidance -0.12 0.33 0.54 -0.001 0.41 (0.83) MHM GO 1.Personal development 0.55 0.15 0.11 0.63 0.11 -0.05 (0.89) 1.Ego-strengthening 0.14 0.69 0.42 0.13 0.67 0.39 0.17 (0.89) 1.Ego-protection -0.07 0.25 0.43 0.01 0.24 0.66 -0.05 0.37 (0.89) Mean 5.81 4.01 3.49 4.22 2.97 2.37 5.36 3.87 2.94 SD 0.98 1.41 1.50 0.57 0.92 0.83 1.02 1.27 1.24
Note: a. values in parentheses are coefficient alpha estimates of reliability; values less than . 10 are nonsignificant.
Achievement Goal Instruments and Augmented Models with External Criteria
Model
CHIS Q
DF CFI IFI TLI RMSEA
SRM R
Description
1 11906 861 1 factor (all 9 scales load on one factor)
3CM 6869.6 816 0.45 0.45 0.42
.131(.128-. 133)
0.1723 correlated method factors
3UM 7499.2 819 0.4 0.4 0.36
.137(.134-. 140)
0.2483 uncorrelated method factors
3 3631.2 816 0.75 0.75 0.73 .089(.086-. 092) 0.0913 trait factors 3T X 3UM 2227.4 774 0.88 0.87 0.85 .066(.063-. 069) 0.075
3 trait factors and 3 uncorrelated method factors* 3T x 3CM 2143.2 771 0.88 0.88 0.86 .064(.061-. 067) 0.068
3 trait factors and 3 correlated method factors* 3T + CU 982.53 522 0.96 0.96 0.93 .045(.041-. 049) 0.060
3 trait factors and methods as correlated uniquenesses (CUs)
Model augmented with external validity criteria (EVC)
3T+ CU 2999.5 15520.91 0.89 0.91
.047(.044 -. 049)
0.069
3T + CU model with EVC** nonsignificant CUs
* improper solution (with negative factor loadings among method factors). **EVC = external validity criteria
Factor Traits Loading** Map Pap Pav
AGQ-S .57* 0.68 0.56
1… to perform as well as I possibly can 0.83 0.53
2… to do well compared to others 0.77 0.64
3… avoid performing worse than others 0.78 0.57
4… to perform as well as it is possible 0.82 0.52
5… to perform better than others 0.87 0.73
6… avoid performing worse than everyone 0.91 0.53
7… to master all aspects of my performance 0.58 0.65
8… to do better than most other performers 0.80 0.68
9… to avoid being one of the worst 0.77 0.58
AAASQ 0.59 0.68 0.65
1…to learn as much as possible 0.68 0.61
2…to do better than the others 0.78 0.69
3… to be taken for a weak person 0.52 0.62
4…to do as well as possible everything 0.69 0.64
5…to prove that I am the best 0.68 0.62
6… could think I’m not very good 0.74 0.66
7…to improve my technique 0.76 0.60
8…try to be better than the others 0.83 0.69
9…I can’t succeed in doing 0.77 0.70
10…to do everything that I have been taught 0.71 0.59
11....interested in doing better than the others 0.79 0.68
12… I am not very good at sports 0.62 0.53
13…for me to progress 0.60 0.53
14…to get better results than the others 0.82 0.72
15… may find out I am doing badly 0.85 0.75
MHMGO 0.60 0.68 0.58
1…to always develop my skills 0.69 0.57
2… I am the only one who can do a skill 0.76 0.63
3… avoid skills for which I may be teased 0.85 0.56
4…to learn new skills and games 0.74 0.51
5… I get ahead of others 0.80 0.66
6… worry others say I don’t have any ability 0.58 0.50
7…keep learning new skills 0.79 0.64
incompetent.
10…I learn a skill and want to try even more 0.78 0.59
11…get ahead of others in skills and games 0.80 0.73
12… concerned if I look incompetent 0.75 0.67
13…I like learning new things 0.77 0.62
14… to get ahead of my friends 0.75 0.75
15… if I try a skill I may look incompetent. 0.76 0.62
16… helps me develop my abilities 0.77 0.67
17… others do not perform as well as I 0.61 0.62
18…want to avoid skills that may look incompetent 0.85 0.59
AGQ-S Q 1 1 2 3 4 5 6 7 8 Q 2 0.27 Q 3 0.10 0.27 Q 4 0.57 0.19 Q 5 0.15 0.41 0.34 0.12 Q 6 0.19 0.57 0.37 Q 7 0.18 0.18 0.21 0.17 0.18 Q 8 0.23 0.23 0.34 0.42 Q 9 0.16 0.39 0.14 0.29 0.54 0.18 0.40 AAASQ Q1 1 2 3 4 5 6 7 8 9 10 11 Q2 0.11 Q3 0.24 0.18 Q4 0.15 0.17 0.22 Q5 0.21 0.13 0.35 0.25 Q6 -0.18 -0.13 Q7 0.24 0.16 Q8 0.38 0.18 0.16 0.20 -0.16 Q9 0.19 Q10 0.19 0.12 0.31 Q11 0.29 0.14 0.13 0.26 0.32 Q12 0.23 0.14 Q13 0.17 0.17 -0.12 0.15 0.16 -0.12 Q14 0.26 0.13 0.21 0.40 0.29 Q15 0.28 0.27 MHMGO-S Q1 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 Q2 0.23 Q3 0.17 Q4 0.38 0.13 Q5 0.30 0.14 Q6 0.27 0.18 Q7 0.36 0.13 0.44 0.12 Q8 0.14 0.57 0.16 0.14 0.37 0.14 Q9 0.59 0.22 0.13 Q10 0.28 0.14 0.42 0.31 0.19
Q12 0.36 0.37 0.37 Q13 0.27 0.12 -0.18 0.27 0.31 0.15 -0.25 0.36 0.13 -0.13 Q14 0.29 0.23 Q15 0.34 0.23 0.34 0.40 -0.18 Q16 0.17 -0.12 0.32 0.29 0.12 -0.14 0.33 0.42 Q17 0.14 0.13 0.14 0.15 0.15 0.11 0.12 0.21 Q 18 0.52 0.20 0.56 0.36 -0.31 0.11 0.47 -0.19 0.20
