TITLE
A Framework for Assessing High School Students’ Intercultural Communicative Competence in a Computer-Mediated Language Learning Project
ABSTRACT
The purposes of this study are (1) to identify the essential dimensions of intercultural communicative competence (ICC) and (2) to establish a framework for assessing the ICC level of high school students that includes a self-reporting inventory and scoring rubrics for online interaction in intercultural contexts. A total of 472 high school students completed the Intercultural Communicative Competence Inventory (ICCI), adapted from the YOGA form (Fantini, 2000, 2001), and 72 students from Taiwan and the United States were selected to join the Learning of Intercultural Language over the Net (LILON) project where they participated in structured intercultural learning activities facilitated through telecommunications technology. Statistical analyses confirm that ICC is a composite of attitudes, skills, knowledge, awareness, and meta-awareness, which support our proposed framework for online intercultural interactions.
KEYWORDS
Intercultural communicative competence; foreign language learning; computer-mediated communication; high school students
INTRODUCTION
Today, in the midst of the communication age (Bonk & King, 1998), as we observe the ways in which we are currently living, working, and learning, we undoubtedly see the interconnectedness of our world. As the likelihood that a majority of today’s youth will someday be living, working, or traveling abroad increases, it is imperative that they be equipped with the skills and capabilities that enable them to work or perform with people from different cultures and languages worldwide. In other words, future denizens of the world should develop a level of intercultural communicative competence (ICC), in which they possess “the knowledge, motivation, and skills to interact effectively and appropriately with members of different cultures” (Wiseman, 2002, p. 208).
Rapid advancements in telecommunications technologies, particularly the Internet, have created opportunities for users to acquire immediate access to the world. Valuable resources via the Internet can be used to bridge the gaps of international communication and help erase the cultural and social boundaries between countries (Szente, 2003). In such a computer-mediated learning environment, Kern and Warschauer (2000) propose that “human-to-human communication is the focus” (p. 1). Specifically, language learners with access to the Internet can now potentially communicate with native speakers (or other language learners) around the world twenty-four hours a day, seven days a week, from school, home, or work (Blyth, 1999). Learners can communicate either on a one-to-one or a many-to-many basis in local-area network conferences further multiplying their opportunities for communicative practice (Kern & Warschauer, 2000). As a result, and with an emphasis on intercultural learning, a growing number of classrooms, especially language classrooms in universities or high schools, are joining online learning networks and communities where teachers and students from around the world
are able to carry out cross-cultural communication (Greenfield, 2003; Roberts, 2004).
Consequently, as more and more schools incorporate intercultural learning components in their language-learning curricula, educators will need to evaluate the extent to which their programs are successful in meeting the goals of the enhanced curricula (O’Dowd, 2003). Specifically, important elements in the evaluation process should include learners’ knowledge, skills, and attitudes toward different cultures (Byram, 1997). For instance, if learners do not have adequate knowledge and skills and do not feel that intercultural learning is a relevant part of their foreign-language learning, they are unlikely to be interested in acquiring the knowledge and the skills that underlie effective communication in intercultural encounters (Dörnyei, 2003).
This study is drawn from a larger federally-funded project, the Learning of Intercultural Language over the Net (LILON) project, whose researchers aim to create structured intercultural learning opportunities for high schools students using telecommunications technology to interact with and learn from students from another country/culture. There are only a few self-reporting inventories that ask respondents at the college level to assess their own ability to communicate effectively in intercultural situations and to date, we have been unable to locate any existing validated instrument whose design targets the assessment of high-school-level ICC. Further, besides the use of inventory-type instruments, the use of supplementary assessment approaches such as rubrics may present a more thorough picture of learners’ level of ICC. This approach allows for authentic assessment that can address and capture the multifaceted nature of the intercultural experience provided through the project.
Therefore, the current study reflects our attempt (1) to identify the dimensions of ICC of high school students and (2) to establish an assessment framework that includes a self-report inventory and scoring rubrics to measure high school students’ ICC, in the context of an
asynchronous CMC environment.
THEORETICAL FRAMEWORK
Definitions and Constructs of Intercultural Communicative Competence (ICC)
The term intercultural communicative competence (ICC) and its constructs are defined differently among different studies. Until now, researchers have not yet reached a consensus on the definition and the constructs of ICC (Hajek & Giles, 2003). Byram (1997) states that “someone with intercultural communicative competence is able to interact with people from another country and culture in a foreign language.” (p. 71). In other words, ICC refers to the competence that enables learners to convey and interpret messages and to negotiate meanings interpersonally within specific contexts (Brown, 1993), including intercultural contexts.
Byram (1997) suggests that ICC should consist of attitudes, knowledge, skills of interpreting and relating, and critical cultural awareness. According to Byram (1997), critical cultural awareness is “an ability to evaluate critically and on the basis of explicit criteria perspectives, practices and products in one’s own and other cultures and countries” (p. 53). He further discusses the related factors that are involved in ICC, and asserts that ICC components be formulated as “objectives,” meaning observable behaviors. For instance, in the construct of attitude, learners should be “willing to seek out or take up opportunities to engage with otherness in a relationship of equality” (p. 57). Byram believes that observable performance would enhance educators’ and researchers’ ability to clarify and to measure ICC.
Similarly, Fantini (2000) suggests that the development of ICC require “insights drawn from both language and intercultural areas” (p. 27). One’s language proficiency can reflect and affect how one perceives, interprets, and expresses one’s view of the world. On the basis of this
perspective, Fantini (2000) proposes four constructs of intercultural competence—awareness, attitudes, skills, and knowledge (A+ASK)—as shown in Figure 1. He stipulates that the term ICC refers to a person’s “abilities to perform effectively and appropriately with members of another language-culture background on their terms.” Among these four constructs, awareness of self and others are seen as the keystone on which effective and appropriate interaction depends. Fantini (2000) has raised the idea of performance, which indicates that, by observing certain behaviors, researchers can measure learners’ ICC-based improvements.
Figure 1. The construct of ICC developed by Fantini (2000, p. 28)
Drawing from the above-mentioned reviews of ICC definitions and constructs, we propose that ICC is a composite of five basic elements: attitudes, skills, knowledge, awareness, and meta-awareness (see Figure 2). Specifically:
1. Awareness (A+) refers to learners’ feelings about self in relation to others (whether the other is a person or a thing).
2. Attitudes (A) refers to learners’ willingness to accept information from other cultures. 3. Skills (S) refer to the degree to which a learner may use appropriate techniques while
4. Knowledge (K) refers to learners’ understanding toward the target culture.
5. Meta-awareness (MA), similar to Byram’s (1997) critical cultural awareness, serves as the monitoring mechanism behind ICC constructs in which learners demonstrate the capacity to (a) identify and interpret explicit and/or implicit values of one’s own and others culture (MA sub-dimension 1), (b) make an evaluative analysis of documents and events based on an explicit perspective and standards (MA sub-dimension 2); and (c) to interact and mediate in intercultural exchanges in accordance with explicit stands or accepted exchanges through negotiation by drawing upon one’s own knowledge, skills and attitudes (MA sub-dimension 3).
Figure 2. Meta-Awareness and the core constructs of ICC
A Framework for Assessing Intercultural Communicative Competence (ICC)
Using an Inventory to Determine High School Students’ ICC Level
To date, instruments used to determine ICC level has been scarce. The Your Objectives, Guidelines, and Assessment (YOGA) form by Fantini (2000, 2001) was created to determine a people’s developmental levels in accordance with the timeframe in which learners are exposed to an intercultural environment. The following describes each of the four levels as it corresponds to ICC development:
1. Educational Traveler Level refers to people who participate in a short-term exchange program, usually lasting from 4 to 6 weeks;
2. Sojourner Level refers to longer cultural immersion; for instance, for people who engage in longer-term internships, usually lasting from 4 to 8 months;
3. Professional Level refers to people who work in an intercultural or multicultural context; for instance, the staff at the School for International Training may be considered to be at a professional level of ICC; and
4. Intercultural/Multicultural Specialist Level refers to people who educate international students.
Thus far, almost no research studies have reported statistical results concerning the YOGA form in regards to the assessment of ICC, nor has it been validated as an instrument for the assessing of high-school-level ICC. Therefore, one of the purposes of this study is to adapt the YOGA form, modify and validate it so that it is more suitable for high school students.
Using Scoring Rubrics to Evaluate High School Students’ Meta-Awareness
Whereas the self-reporting ICC inventories allow us to determine students’ ICC level, scoring rubrics provide the opportunity to evaluate the quality of both processes and outcomes of intercultural learning (Jacobson, Sleicher, & Maureen, 1999). In this study, we examined students’ learning artifacts from online activities. The activities were designed so that students could demonstrate various types of skills and knowledge more expansively and thoroughly (Corbett, 2003), which helped us to evaluate learners’ meta-awareness abilities.
In sum, the self-reporting inventory is used to determine students’ levels of awareness, of attitudes, of skills, and of knowledge; whereas scoring rubrics are used for the assessment of
students’ ICC-based meta-awareness (see Table 1).
Table 1. Summary of the ICC-based essential dimensions and assessment methods
Dimensions Assessment Methods
Awareness (A+), Attitudes (A), Skills (S), & Knowledge (K)
Intercultural Communicative Competence Inventory (ICCI)*
Meta-Awareness (MA) Scoring rubrics
* Adapted from the YOGA Form (Fantini, 2000, 2001)
METHODS
Participants
A convenience sample of 472 high school students was comprised of 273 tenth graders from three high schools in Taiwan and 199 tenth and eleventh graders from two high schools in the U.S. to participate in the process of validating the ICCI. Following, participants were selected to participate in the LILON project that consisted of 72 high school students: 39 tenth graders from one high school in Taiwan and 33 tenth and eleventh graders from one high school in the U.S.. Permission to participate in the study was obtained from the schools, the students and their parents.
The Learning of Intercultural Language over the Net (LILON) Project The Project Setting
The LILON project extended from September 2005 to the end of May 2006, a total of nine months. What makes this project unique lies in the design of the learning environment in which students in Taiwan (Taipei) are studying English as a foreign language and in which students in the U.S. (San Francisco) are studying Chinese as an elective foreign language. We believe that this project constitutes an opportunity for high school students from one target culture to interact with high school students from the other target culture.
The Underpinning Design Principles
The design principles of the LILON project incorporate tenets of foreign-language learning, ICC as part of language proficiency, and computer-supported collaborative learning. We developed instructional-learning units for the participating high school classes. These learning units focused on themes that emphasize intercultural understanding, promote ICC, and enhance foreign-language learning beliefs. Participating students worked in collaboration with cultural partners. Similar to other cross-cultural exchange projects, this project uses telecommunications technology (i.e., online discussion boards and web-based video conferences) via Blackboard, an online course-management system (see Figure 3 and 4), and via JoinNet video conferencing system (see Figure 5), to facilitate students’ collaboration with students from another country and culture. Besides occasional synchronous video conferencing, the asynchronous discussion mode was mainly used because of its flexibility relative to students’ reflections (Arnold & Ducate, 2006; Morse, 2003) during and after the activities.
Figure 3. Screenshot of the online course-management system
Figure 5. Screenshot of the JoinNet videoconferencing system
Intercultural Collaborative Learning Activities via Telecommunications Technology Prior to the collaborative activities between students, the researchers and classroom teachers planned out the project logistics and familiarized the participants with the features of the course management system. Participating students received a project kit containing information about the project, rules they must follow, descriptions and instructions regarding the learning units, helpful resources, and rubrics detailing how their performance will be assessed. At the beginning of the project, participating students were given orientations, including training of the course management system, application software and Internet etiquette. During orientation weeks, students learned to log into the course management system, edit their online profiles, and sent greetings to each other through discussion board. The purpose was to have students from
different cultures/countries get to know each other and become more familiar with the use of tools offered through the course management system. The course management system was hosted by the server located in the researchers’ university. The researchers also provided the necessary application software to be installed in the high school computer labs. All students, classroom teachers, project facilitators, and researchers had individual accounts to access to the course management system during and after the study.
After the orientations, students were randomly assigned a learning partner and had to work collaboratively to accomplish several specific activities in the learning units. There were three learning units (i.e., Self-Introduction, School Life and Digital Video Project), each taking approximately five to six weeks to complete. During the first week of each learning unit, students would individually work on a task that consisted of gathering and putting together information relating to the learning unit and presenting the information to their learning partner. Weeks two and three were mainly online discussion activities for which dyads would spend week two comparing and contrasting points, asking and answering questions, and exchanging thoughts and comments based on the information presented from week one. In week three, dyads were asked to come up with ideas for their learning products (e.g., PowerPoints and videos), which was due at the end of last week. Therefore, during week three, dyads would engage in the process of brainstorming, negotiating, planning, and decision making via online discussions. Dyads would then use the last weeks to create and upload their final learning products to the discussion boards.
The classroom teachers agreed to integrate the LILON project into their regular course curriculum. Integration consists of (1) classroom discussions on relevant intercultural learning topics and (2) required participation in the project—participation that included contributing to
discussions’ board activities and constructing of their final learning products. Rather than develop paper-and-pencil tests, this project developed a set of rubrics for assessing the complexity of students’ online interactions (Jonassen, Howland, Moore, & Marra, 2004). In order to control for confounding factors such as student motivation, the project scores were counted toward the students’ final semester grade, which was approximately 5-10% of the total grade.
Data Collection
Validation of the Intercultural Communicative Competence Inventory (ICCI)
Prior to the implementing the LILON project, the development and validation of ICCI was conducted. We adapted the YOGA form (Fantini, 2000, 2001) specifically for the assessment of ICC of high school students. In the original YOGA form, there are 87 questions addressing four ICC levels (Educational Traveler, Sojourner, Professional, and Intercultural/Multicultural Specialist); however, considering that high school students experience and are exposed to intercultural issues and topics mainly at school, we used only questions from the Educational Traveler Level and the Sojourner Level in our revised instrument. For that reason, the first step was to ascertain that the wording and the contexts were valid and adequate for high school students from Taiwan and the United States. After removing question items from the Professional Level and the Intercultural/Multicultural Specialist Level, the revised questionnaire had a total of 37 items.
The second step involved minor editing on question rephrasing. For example, rather than state that “I demonstrate awareness of how I am viewed by members of the host culture” (question 1) in the original YOGA form, we decided to rephrase this question to “I demonstrate awareness of how I am viewed by my partner in the target culture.” For students in Taiwan, the
target culture refers to American culture, whereas for students in the U.S., the target culture refers to Chinese culture. As a result, we revised seventeen questions in a similar manner to make the terminology more appropriate for the current study.
Last, all questions were translated directly into Chinese for high school students in Taiwan. Three Taiwanese high school students reviewed the Chinese version for clarity. A few refinements were made before the final version of the inventory was distributed to the participating students. Since the original YOGA form was written in English, the researchers did not feel it was necessary for American high school students to review the questions for clarity.
The questionnaire asked participants to rate themselves in each of the four ICC subscales, from 0 (no competence) to 5 (very high competence). It took approximately 10 to 15 minutes to administer this questionnaire. Sample items of the four dimensions are presented as follows:
1. A sample item of the Awareness dimension is: “I demonstrate the awareness of how I am viewed by my partner of the target culture”.
2. A sample item of the Attitudes dimension is: “I demonstrate a willingness to try to communicate in the target language and to behave in ways judged ‘appropriate’ by the target culture”.
3. A sample item of the Skills dimension is “I use a variety of effective strategies when interacting with culturally different people”.
4. A sample item of the Knowledge dimension is “I can contrast aspects of the target culture with my own”.
Development of Scoring Rubrics to Evaluate Meta-Awareness in Online Interactions Messages on the discussion boards were captured into a word processor for later analysis.
Two content experts in instructional design developed the rubrics that provided a standard guideline to assess various aspects of students’ online interactions. The aim was to design rubrics that not only could be used to score reliably and comprehensively, but that would also provide diagnostic information. In other words, the rubrics were designed to reflect and, hopefully, construct best practice aspects of the meta-awareness learning of ICC.
The rubrics were included in the project kit that was given to the participating students before the LILON project began. Specifically, rubrics for assessing students’ online discussion board interactions include criteria such as (1) clarity of message content, (2) quality of the messages, (3) timeliness of participation to the discussions, (4) responsiveness towards others’ messages, (5) courtesy towards others, (6) posted messages that raised questions or initiated discussions, and (7) posted messages summarizing/reflecting what was learned, and (8) level of meta-awareness (Appendix A presents detailed information of rating scales, except for the level of meta-awareness). The following paragraphs deal with the three criteria on evaluating students’ meta-awareness of ICC learning: (1) Value identification and interpretation (MA sub-dimension 1), (2) Evaluative analysis (MA sub-dimension 2), and (3) Appropriate interaction, mediation and negotiation (MA sub-dimension 3).
Table 2 shows the scoring rubric for examining MA sub-dimension 1 of “the level of the student’s ability to identify and interpret values of his/her own and other cultures during discussions in LILON learning activities”, with the total possible point of 5. This scoring rubric contains four indicators describing different levels of performance, consisting of “Not proficient” (0 points), “Acceptable” (1 point), “Good” (2 to 3 points), and “Excellent” (4 to 5 points), and descriptors for each indicator. For a message displaying no evidence of value identification or interpretation was graded “Not proficient” (0 points); in contrast, a message was graded
“Excellent” (4 to 5 points), when demonstrating very clearly on identifying and interpreting values based on his/her own and other cultures. It also describes in-depth about respecting different cultures from his/her own.
Table 2. The scoring rubric of Meta-Awareness sub-dimension 1 (MA sub-dimension 1) – Value identification and interpretation
Level of Performance Descriptor
Excellent (4-5 points) Demonstrate very clearly on identifying and interpreting values based on his/her own and other cultures. Go into much detail about respecting different cultures from his/her own Good (2-3 points) Demonstrate clearly what his/her own and other cultures are Acceptable (1 point) Is vague about the ability to identify and interpret values Not proficient (0 points) No evidence of meta-awareness on the online interactions
Table 3 shows the scoring rubric for examining MA sub-dimension 2 of “the level of the student’s ability to make an evaluative analysis using explicit perspective and standards of interactions and products created through LILON learning activities”. Table 4 shows the scoring rubric for examining MA sub-dimension 3 of “the level of the student’s ability to interact and mediate in intercultural exchanges in accordance with explicit standards or accepted exchanges through negotiation by drawing upon his/her own knowledge, skills, and attitudes”.
Table 3. The scoring rubric of Meta-Awareness sub-dimension 2 (MA sub-dimension 2) – Evaluative analysis
Level of Performance Descriptor
Excellent (4-5 points) Demonstrate very clearly on judgment making. Go into much detail about respecting different cultures from his/her own Good (2-3 points) Is able to make judgment based on his/her own cultural
perspective and standards
Acceptable (1 point) Is vague about the ability to make an evaluative analysis Not proficient (0 points) No evidence of meta-awareness on the online interactions
Table 4. The scoring rubric of Meta-Awareness sub-dimension 3 (MA sub-dimension 3) – Appropriate interaction, mediation, and negotiation
Level of Performance Descriptor
Excellent (4-5 points) Demonstrate very clearly of appropriate knowledge, skills and attitudes on intercultural exchanges, and show a degree of acceptance of these exchanges
Good (2-3 points) Demonstrate appropriate knowledge, skills and attitudes on interacting and mediating in intercultural exchanges
Acceptable (1 point) Is vague about the ability to interact and mediate in intercultural exchanges appropriately
Data Analyses
Procedures to analyze the data included descriptive statistics, exploratory factor analysis, Pearson product-moment correlation, and t-tests. During the process of data screening, cases where the participants did not fill out either the pretest or the posttest inventory were deleted. Sixty-two students (86.1% response rate) completed both the pretest and posttest inventories and were the only ones included in the analysis. Of the total sample, thirty-two were male students (52%) and thirty-five students (56%) were from Taiwan.
We performed exploratory factor analysis using varimax rotation, construct validity, and reliability testing to validate the four dimensions of ICC (Awareness, Attitudes, Skills, and Knowledge). An item was retained only when it was loaded with absolute values greater than .5 and with an eigenvalue greater than 1 for the dimension. We calculated each Cronbach’s α for each dimension to test internal reliability.
Pearson product-moment correlation was conducted to identify significant relationships and to report the degree of those relationships between the four dimensions in ICC and the level of Meta-Awareness scores. Only posttest ICC inventory scores were used because we wanted to see how they relate to the Mata-Awareness scores. A paired-sample t-test of the four dimensions was performed to compare the means between pretest and posttest scores for the students. All statistical analyses were performed using the Statistical Package for the Social Sciences (SPSS 12.0).
Additionally, using the rubrics, three project facilitators graded students’ online messages on the discussion boards and discrepant scoring were resolved through discussion. All the graders had received training in scoring using the rubrics and practiced on examples prior to the implementation of the project.
RESULTS
Results of Intercultural Communicative Competence Inventory (ICCI) Validation Factor Analysis of the ICCI
A factor analysis of the ICCI reveals four dimensions—Awareness, Attitudes, Skills, and Knowledge—which are identical to the dimensions in the original YOGA form (see Table 5). The initial 37 items on the ICCI are reduced to 23, choosing items with an eigenvalue of one or higher, which are respectively 5, 6, 5, and 7 in these four dimensions. Furthermore, the reliability (alpha) coefficients for these four dimensions respectively are .83, .93, .88, and .91, with an overall alpha of .95, which indicates that 95% of total variance is explained by these four dimensions. Table 4 indicates that this revised inventory is adequately reliable for measuring high school students’ intercultural communicative competence.
Table 5. Factor loadings from principal-components analysis of the ICCI items using varimax rotation (N=472)
Factor Item Number/Item Content
1 2 3 4
Awareness Dimension
4. I demonstrate awareness of how I am viewed by my partner of the target culture
.590 6. . . . responses to my social identity (race, class, gender, age, ability, etc.)
within the context of my own culture
.778 7. . . . responses to my social identity (race, class, gender, age, ability, etc.) as
perceived by my partner of the target culture
.763 8. . . . intracultural differences (i.e., diversity aspects such as race, class, gender,
age, ability, sexual orientation, etc.) within my own culture
.644 10. . . . my choices and their consequences (which make me either more or less
acceptable to my partner of the target culture)
.506
Attitude Dimension
11. I demonstrate a willingness to interact with my partner of the target culture (I don’t avoid them, I primarily seek the company of my compatriots, etc.)
.746 12. . . . learn the language and culture of my partner from the target culture .839 13. . . . try to communicate in the target language and to behave in ways judged
“appropriate” by my partner of the target culture
.858 14. . . . try to deal with the emotions and frustrations caused by my participation
in the target culture (in addition to the pleasures which it offers)
Factor Item Number/Item Content
1 2 3 4 17. . . . adapt my behavior in accordance to what I am learning about
communication in the target culture (e.g., language, non-verbal behaviors, and sensitivity to behavioral adjustments appropriate for different contexts)
.766 18. . . . reflect on the effect that my decisions, choices, and behavior have on my
partner of the target culture .733
Skills Dimension
24. I use a variety of effective strategies when interacting with culturally different people
.693
26. I am able to cite sociopolitical factors that have shaped both my own culture and the target culture
.689
27. I employ appropriate strategies for coping with or adjusting to the target culture
.754
28. I employ appropriate strategies for coping with or adjusting to my own
culture upon returning home .540
29. I identify and effectively utilize models, strategies, and techniques to enhance my learning about the target culture and language
.699
Knowledge Dimension
30. I can cite a basic definition of culture and identify its components .733 31. I can contrast aspects of the target culture and my partner of the target
culture with my own
.637 32. I know the essential norms and taboos (greetings, dress, behavior, etc.) of the
target culture .672
33. I recognize signs of cultural stress and I know strategies for overcoming them
.592 34. I know some techniques to maximize my learning of the target culture and
my partner of the target culture .605
35. I can articulate at least one academic definition of culture and can describe the complexities of cultural systems using relevant concepts and terms
.740 36. I can describe and explain my own behavior and that of my counterpart in
various domains (e.g., social interaction, time orientation, relation to the environment, etc.)
.609
Proportion of variance explained, % 22.75 18.26 15.19 13.33
Total variance explained, % 22.75 40.41 55.60 68.93
Alpha .83 .93 .88 .91
Note. KMO = .95, α = .95. Only factor loadings greater than .5 are shown.
High School Students’ Scores on the ICCI
Table 6 shows the mean and standard deviations of pre-and posttest scores on the dimensions of ICCI. The results indicate that high school students of this study demonstrated moderate levels of competencies in awareness, attitudes, skills, and knowledge in intercultural contexts. Of the four dimensions, students demonstrated higher levels of competence in the Awareness and Attitudes dimensions, on both pre- and posttest.
A paired-sample t-test of the above four dimensions was performed to compare the means between pretest and posttest scores. Based on the results of paired-sample t-test, the pre-and-posttest scores of the Attitudes dimension had significantly decreased (p < .001) while the scores of the Awareness, Skills, and Knowledge dimensions also decreased from pretest to posttest, they were not significant differences.
Table 6. Students’ mean scores and paired-sample t-test among pretest and posttest scores on the ICCI dimensions (N=62) Pretest Posttest M SD M SD t p Awareness 3.17 .76 3.06 .63 -1.15 .256 Attitudes 3.58 .80 3.07 .78 -4.29*** .000 Skills 2.99 .79 2.95 .79 -.415 .679 Knowledge 2.77 .91 2.81 .76 .370 .710 *** p < .001, two-tailed
Correlations of the Four Dimensions on ICCI
The strength of correlation is interpreted using the criteria suggested by Gall, Borg, and Gall (1996) (refer to notes under Table 8). Posttest score of the Awareness dimension showed a substantial positive correlation with that of the Skills dimension (r = .619) at the .01 level of significance; two moderate positive correlations were found with that of the Attitudes dimension (r = .563) and with that of the Knowledge dimension (r = .555), at the .01 level of significance. Posttest score of the Attitudes dimension showed a substantial positive correlation with that of
the Skills dimension (r = .607) at the .01 level of significance; while a moderate positive correlation was found with that of the Knowledge dimension (r = .462) at the .01 level of significance. Lastly, the posttest score of the Skills dimension showed a substantial positive correlation with that of the Knowledge dimension (r = .688) at the .01 level of significance.
The results indicate the existence of moderate to substantial relationships among the four ICC dimensions, confirming the coherence of these four dimensions. In other words, the more the student demonstrates a degree of intercultural awareness, the better he/she demonstrates appropriate attitudes, skills, and knowledge in intercultural contexts.
Results of Students’ Meta-awareness Scores for Online Interactions
Table 7 shows the numbers and the percentages of students’ Meta-Awareness scores. In the MA sub-dimension 1, more than half of the students (77.4%) had scored 2 and above points, which indicates that they were able to identify and interpret values on both their own and their partners’ cultures; whereas seventeen students (27.4%) were even able to go further to express their respect towards their partners’ culture (scored 4-5 points) during discussions in LILON learning activities. The following quotes are from Vincent (from Taiwan) and Arthur (from the U.S.) (pseudonyms) when they asked and answered questions based on each other’s self-introduction PowerPoint. We translated Albert’s message into English and made minor grammatical corrections in order to improve readability. Arthur’s message was scored 5, having revealed the appreciation and value of learning a foreign language as well as recognized his partner’s limited language proficiency, therefore, offered to translate English into Chinese.
Vincent asked,
How long have you been studying Chinese? Why do you want to learn Chinese? You mentioned that you can write poetry. Can you share some with me?
Arthur responded to Vincent’s message as:
I used to speak Chinese with my parents during my childhood. Why I want to learn Chinese? I think it will be easier to find a job if being proficient in Chinese. Besides, I like to communicate with my family in Chinese, even though most of the time I use English at school and with my friends. All my poems are in English though – would you prefer them in English or should I translate them into your language? (MA sub-dimension 1)
However, in the MA sub-dimension 2, more than half of the students (58.1%) did not demonstrate proficiency (scored 0 points) with regards to making an evaluative analysis using explicit perspectives and criteria of interactions and products created through LILON learning activities. Although twenty-one students were able to demonstrate a good to excellent ability to make a judgment based on criteria of his/her own cultural perspectives. The following example is a statement from a student when discussing about different kinds of books they enjoyed reading. She explained why she prefers reading the book than watching the movie.
I like Harry Potter too. I am reading the Chinese version [of Harry Potter], and I have seen the movie as well. But I don’t think they shot this movie well – some important elements are missing in the movie… (MA sub-dimension 2, Jennifer from the U.S., scored 5)
Following this statement, the students went into discussion about what they thought were the differences between the book and the movie.
Also, in the MA sub-dimension 3, a majority of the students (83.8%) were unable to demonstrate the ability to interact and mediate in intercultural exchanges appropriately (scored 0-1 points). In other words, the students did not interact and mediate in intercultural exchanges in accordance with explicit standards or accepted exchanges through negotiation by drawing upon his/her own knowledge, skills, and attitudes. The following message is from an American student providing comments to his partner’s PowerPoint and complimenting his English.
I just looked at your school life PowerPoint and made some corrections [on grammar]. I just wanted to tell you that your English is getting better and that you have a good understanding of what certain English words mean in Chinese.
The American student then continues on to offer suggestions for improving language learning, from personal experience.
I also want to say that the thing that you need to work on right now is improving your English grammar. This is an extremely difficult thing to do because grammar isn't something that can be taught. Grammar is something learned through experience. I believe, that learning grammar, whether it be English or Chinese, French or Spanish, is best learned doing one of the following:
1. Watching videos where people talk in the language 2. Listening to songs sung in the language
3. Actually traveling to the country that speaks the language 4. Reading simple story books that are written in the language 5. Practicing a short sentence in that language each day
Last, he goes on to give encouragement and support:
Try to apply what I’m telling you right now as you continue to take English class at your school. I believe that by following [these] steps, you will be able to speak English more clearly and effectively. Don’t feel bad though. Learning English grammar can really difficult, and to tell the truth, I don’t understand it sometimes. Don’t give up. Keep it going! (MA sub-dimension 3, Albert from the U.S., scored 5)
When looking at the online interactions, students in Taiwan usually completed the minimum project requirements (e.g., posted one discussion board message per week) or provided superficial descriptions or observations of their own cultures—which may have inhibited continuous and in-depth discussions. As one Taiwanese student pointed out,
I didn’t know what questions to ask [my learning partner], but the teacher asked us to ask three questions [based on the content of her PowerPoint]… so I just posted whatever I could think of to fulfill the minimum requirements.
This finding seems to support the study by Yildiz and Bichelmeyer (2003) that Asian students are usually not good at expressing themselves on the discussion board, and “posted only
as many times as was required by the instructor” (p. 184). Moreover, cultural differences may have impacted EFL students’ comfort with learner-centered discussions. Students coming from Asian countries are used to learning directly from the teachers. They find “learning through peer discussion unusual and challenging” (p. 191), and public disagreement, questioning, and negotiating are not a part of their culture, either. In our study, Taiwanese students claimed that they were not able to write as lengthy postings in English as their learning partners in the U.S. For that reason, they prefer to agree with their learning partners and avoid questioning and challenging, which may have affected their performance in the process of interacting and negotiating.
Table 7. Results of students’ sub-dimensions scores on meta-awareness (N=62) Meta-Awareness Dimension Scored 0 Number (%) Scored 1 Number (%) Scored 2-3 Number (%) Scored 4-5 Number (%) M SD MA sub-dimension 1 (Value identification and interpretation) 0 (0%) 14 (22.6%) 31 (50.0%) 17 (27.4%) 2.81 1.39 MA sub-dimension 2 (Evaluative analysis) 36 (58.1%) 5 (8.1%) 13 (21.0%) 8 (12.9%) 1.19 1.67 MA sub-dimension 3 (Appropriate interaction, mediation and negotiation)
42 (67.7%) 10 (16.1%) 8 (12.9%) 2 (3.2%) .65 1.20
Relationship between the Scores of ICCI and the Scores on Meta-Awareness
Meta-Awareness and the posttest scores of the Awareness, Attitudes, Skills, and Knowledge dimensions. The total score of Meta-Awareness shows significant but low correlations with the Awareness dimension (r = .372, p < .01), with the Skills dimension (r = .314, p < .05), and with the Knowledge dimension (r = .360, p < .01). The results support the relationships between Meta-Awareness and the four dimensions identified by Fantini (2000, 2001), at the high school level.
Table 8. Correlation table between ICCI posttest scores and the total score of Meta-Awareness (N=62) Variables 1 2 3 4 5 1. Meta-Awareness - 2. Awareness .372 ** - 3. Attitudes .227 .563 ** - 4. Skills .314 * .619 ** .607 ** - 5. Knowledge .360 ** .555 ** .462 ** .688 ** - ** p < .01, two-tailed; * p < .05, two-tailed.
Strength of correlation is interpreted as follows (Gall, Borg, & Gall, 1996) .01-.1 = very low
.2 - .3 = low .4 - .5 = moderate .6 - .7 = substantial .8 - .9 = very high
CONCLUSIONS
In this study, we adapted a self-reporting inventory in order to validly and reliably measure the ICC of Taiwan and U.S. high school students. Original items were taken from the YOGA form, and as expected, the constructs of the revised inventory appear similar to that of the original YOGA form. Apparently, the YOGA form can be relatively easily adapted to use in both Western and East Asian cultures. However, minor editing of wording is needed for different contexts.
Our intention for the revised inventory is that it be short and concise, therefore, easy to administer. These attributes will enable (1) educators to use it quickly and easily in their assessment of students’ ICC level and (2) researchers to incorporate it into large-scale studies. The twenty-three ICCI items appear to be a valid measure of high school students’ ICC in the context of online intercultural learning. The differences between the original YOGA form and the revised ICCI inventory are relatively minor.
We also provide a supplementary assessment method using scoring rubrics to evaluate high school students’ Meta-Awareness during online interactions in the process of completing intercultural learning activities. When using scoring rubrics to evaluate high school students’ Meta-Awareness, researchers and educators may gain more in-depth information and a thorough picture of complex levels of Meta-Awareness competencies in a systematic and reliable way. The purpose of evaluating students’ meta-awareness is based on our assumption that it serves as an overarching mechanism on high school students’ awareness, attitudes, skills, and knowledge in intercultural contexts.
Statistical analyses have confirmed the construct of high school students’ ICC that includes four dimensions (Awareness, Attitudes, Skills, and Knowledge). They also revealed relationships
between meta-awareness and the four dimensions, therefore supporting our assumption and framework of assessing high school students’ ICC, which consists of a self-reporting inventory and scoring rubrics for online intercultural interactions.
Although there was a significant decrease in the students’ pre- and posttest scores of the Attitudes dimension, this could be due to the frustration that students sometimes felt, as a result of miscommunication because of language barriers and untimely responses from their partners. Therefore, it could have affected their willingness to accept or exchange information from and with their learning partners of the other culture. Additionally, cultural differences in terms of patterns of online participation may have hindered the progress of communicative exchanges; hence, further extensive studies are needed to compare the different communication styles and interaction patterns across cultural groups in intercultural settings. Even though some studies on computer-mediated cross-cultural exchanges have also reported similar outcomes such as communication breakdown and unresponsiveness (Reeder, Macfadyen, Roche, & Chase, 2004; Solem et al., 2003), however, the pragmatic and instructional benefits of telecommunications technology in facilitating cross-cultural learning have been well noted and are worthy of further exploration.
As teachers and researchers consider incorporating intercultural learning in their language curriculum, either in traditional classroom settings or computer-mediated environments, it is important that they implement a framework to assess students’ learning processes and outcomes. Therefore, this framework that we propose may serve as a model in guiding the design and development of their curriculum.
ACKNOWLEDGEMENT
This study was supported by the National Science Council (NSC), Taiwan, under grant numbers NSC93-2520-S-009-007 and NSC94-2520-S-009-008. Thanks are given to: Chih Yen and Chia-Yen Chong of the Web-based Academic Affairs Division at the National Chiao Tung University (NCTU); high school principals and administrators, teachers (Marlisa Lee, Mariam Fan, Stephanie Moore, and Steve Grad); and students at the Ta-Tung High School, Zhong-Lung High School, and Chien-Kung High School in Taiwan, Villa Duchesne High School in St. Louis, and Saratoga High School in San Francisco, U.S.A.
REFERENCES
Arnold, N. & Ducate, L. (2006). Future foreign language teachers’ social and cognitive collaboration in an online environment. Language Learning and Technology, 10(1), 42-66.
Blyth, C. (1999). Implementing technology in the foreign language curriculum: Redefining the boundaries between language and culture. Journal of Educational Computing Research, 20(1), 39-58.
Bonk, C.J. & King, K.S. (Eds.). (1998). Electronic collaborators: Learner-centered technologies for literacy, apprenticeship, and discourse. Mahwah, NJ: Lawrence Erlbaum Associates.
Brown, H. D. (1993). Principles of language learning and Teaching (3rd
ed.). Englewood Cliffs, NJ: Prentice-Hall.
Byram, M. (1997). Teaching and assessing intercultural communicative competence. Clevedon, England: Multilingual Matters Ltd.
Corbett, J. (2003). An intercultural approach to English language teaching. Buffalo, NY: Multilingual Matters.
Dörnyei, Z. (2003). Attitudes, Orientations, and Motivations in Language Learning: Advances in Theory, Research, and Applications. Language Learning, 53(S1), 3-32.
Fantini, A. (2000). A central concern: Developing intercultural competence. Retrieved, August 1, 2005 from http://www.sit.edu/publications/docs/competence.pdf
Fantini, A. (2001). Exploring intercultural competence: a construct proposal. Paper presented at NCOLCTL Fourth Annual Conference, April 2001, Las Vegas, NV, USA.
Gall, M.D., Borg, W.R., & Gall, J.P. (1996). Educational research (6th ed.). White Plains, NY: Longman.
Greenfield, R. (2003). Collaborative e-mail exchange for teaching secondary ESL: A case study in Hong Kong. Language Learning and Technology, 7(1), 46-70.
Hajek, C. & Giles, H. (2003). New directions in intercultural communication competence: The process mode. In J.O. Greene & B.R. Burleson (Eds.), Handbook of communication and
social interaction skills (pp. 935-57). Mahwah, NJ: Lawrence Erlbaum Associates.
Jacobson, W., Sleicher, D., & Maureen, B. (1999). Portfolio assessment of intercultural competence. International Journal of Intercultural Relations, 23(3), 467-92.
Jonassen, D.H., Howland, J., Moore, J., & Marra, R.M. (2003). Learning to solve problems with technology: A constructivist perspective (2nd ed.). Englewood Cliffs, NJ: Prentice-Hall.
Kern, R. & Warschauer, M. (2000). Introduction: Theory and practice of networked-based language teaching. In M. Warschauer & R. Kern (Eds.) Networked-based language
teaching: Concepts and practice, (pp. 1-19). Cambridge, UK: Cambridge University Press.
Morse, K. (2003). Does one size fit all? Exploring asynchronous learning in a multicultural environment. Journal of Asynchronous Learning Networks, 7(1), 37-55.
O’Dowd, R. (2003). Understanding the “other side”: Intercultural learning in a Spanish-English e-mail exchange. Language Learning & Technology, 7(2), 118-144.
Reeder, K., Macfadyen, L.P., Roche, J. & Chase, M. (2004). Negotiating cultures in cyberspace: Participation patterns and problematics. Language learning and technology, 8(2), 88-105.
Roberts, L.G. (2004). Harnessing information technology for international education. Phi Delta Kappan, 86(3), 225-28.
Solem, M.N., Bell, S., Fournier, E., Gillespie, C., Lewitsky, M. & Lockton, H. (2003). Using the Internet to support international collaborations for global geography education. Journal of Geography in Higher Education, 27(3), 239-53.
Szente, J. (2003). Teleconferencing across borders: Promoting literacy—and more—in the elementary grades. Childhood Education, 79(5), 299-306.
Warschauer, M., & Kern, R. (2000) (Eds.). Network-based language teaching: Concepts and practice. Cambridge: Cambridge University Press.
Wiseman, R.L. (2002). Intercultural communication competence. In W.B. Gudykunst & B. Moody (Eds.), Handbook of international and intercultural communication, (2nd ed.)(pp.
207-24). Thousand Oaks, CA: Sage.
Yildiz, S. & Bichelmeyer, B.A. (2003). Exploring electronic forum participation and interaction by EFL speakers in two web-based graduate-level courses. Distance Education, 24(2), 175-91.
