Research Design and Methodology

Chapter Three

Research Design and Methodology

From the discussions in the previous two chapters, it can be concluded that researchers agree that CD-ROM titles are important instructional aids in language teaching and learning. In order to apply this kind of multiple interactive software in language classrooms, I try to locate a set of applicable and easy-to-use criteria as a reference in choosing a piece of appropriate software and to make the process of evaluation more efficient. Therefore, as mentioned in 1.4, the purpose of this study is to construct a set of criteria to evaluate instructional CD-ROM titles. By means of the evaluation processes, teachers would have a better grasp of the textbooks, making

teaching more effective and interesting.

This chapter presents the research designs and the methodology used in this study. It comprises of four parts: research design, data collection, pilot testing, and evaluation procedure. First, this study will introduce how I would like to proceed with a series of evaluation procedure that includes the determination of methods, the choice of criteria, and the target CD-ROM titles. The second step is the

implementation of initial plans. To put these plans into action, I will describe the process of studying related research and theories, the construction of the instrument, the steps of the pilot testing, and the processes of the evaluation. The result of evaluation will be presented and analyzed in Chapter Four, and the suggestions for application and further development will be discussed in Chapter Five.

3.1 Research Design

Since the purpose of this study is to choose appropriate instructional CD-ROM

titles for language classes, I begin to collect and analyze related research and data.

After thorough consideration, an evaluation instrument is formed and the process of evaluation follows.

3.1.1 Method

As mentioned in 2.2, Chapelle (2001) contends that CALL activities should be evaluated through two perspectives: the judgmental analysis of software and teachers’

planned tasks, and the empirical analysis of learners’ performance. According to the discussion in 2.2.3, the experimental analysis of examining learners’ data was

excluded because it is difficult to collect and analyze learners’ data in a limited period of time. Therefore, the subjective evaluation will be used in this study. On the other hand, the evaluation criteria are applicable to most situations and deserve a full discussion in this study. Hence, subjective judgment on the criteria, guidelines, and checklists will be the focus of this study.

3.1.2 Object of Evaluation

Most English textbooks adopted by schools come with language learning

materials such as MP3, audio CD or website connection. Some bookstores have also published CD-ROM titles constructed mainly on the formats of their textbooks and usually serve as a subsidiary to their print copies. Duber (1998) also comments that most published reviews of software programs have followed the form of textbook reviews. These programs are treated as major supplementary teaching materials to their textbooks and the aims of the CD-ROM titles are clear since they are mainly a simulation of their textbooks. As a result, there is a need to evaluate the frequently used CD-ROM titles in terms of multimedia presentation, suitability, and

effectiveness of their application.

3.1.3 Instrument

In order to construct a set of evaluation criteria as an instrument, a variety of software evaluation criteria, guidelines, and checklists are collected, analyzed, and organized. After reviewing and discussing the literature of CALL and criteria in 2.3.3.3, Ioannou-Georgiou’s selecting software for language classes is considered as the most suitable one and chosen as the instrument of this study because of its complete design especially for language classes.

The checklists are originally published in an article in the journal Modern English Teacher. It lists the criteria for evaluating software and discusses their

application in language classes. It is practical and valuable for teachers to consult with while applying software in their curricula. In order to make it applicable and easy-to-use for teachers, I transformed it into a checklist style, in which teachers or users can answer ‘yes’ or ‘no’ to the evaluation questions and make some comments if needed.

The checklist is divided into three main parts, each with its own criteria for evaluation. The first part is the Technical Aspect, which informs teachers on computer requirements and factors that might influence the performance of the program while it is applied on computers. The second part focuses mainly on language learning and teaching, that is, the Methodology Aspect. Lastly, the third part is on the Pedagogical Aspect. When applying it to language classrooms,

teachers need to take pedagogical criteria into account in terms of learners’ motivation and suitability, variety of activities, and their effectiveness, as well as the integration of software and the curricula.

In order to make sure that the form is appropriate for this study, a pilot test of

evaluating a Junior High School CD-ROM was conducted, and some revisions were

included to make it more appropriate for the school setting.

3.2 Data Collection and Analysis

As the previous research design in 3.1, in order to search for its evaluation criteria of instructional software and assess its effectiveness, I reviewed the research of the fields of CALL, SLA and applied linguistics.

In the first chapter of this study, the application of computers and the role they play in pedagogy as well as language learning practice on computers have been discussed. After reviewing and summarizing the research, the result of applying computer technology into language learning has been proved effective, advantageous, and closely connected as discussed in Chapter One and Two.

For the purpose of choosing a set of systematic evaluation criteria, a variety of criteria, guidelines, and checklists made by researchers, educators, or teachers have been reviewed. According to the characteristics of the criteria, these forms are categorized into four types: (1) General Criteria and Guidelines; (2) General Guidelines and Checklists with Numerical System; (3) Criteria of E/FSL Instructional CD-ROM Evaluation; (4) Checklists of E/FSL Instructional CD-ROM Evaluation including the Technical Aspect. As in the summary in 2.3.5, a set of criteria by Sophie, Ioannou-Georgiou was chosen as the instrument of this study.

There are two types of instructional CD-ROM titles for language learning in

terms of their written copies, namely magazines and textbooks. Most of the

language magazines published in Taiwan are sold each with a piece of multimedia

CD-ROM, serving as outside reading or as supplement materials assigned by teachers

to enhance their language learning or as self-instruction materials. The other type is

the one published according to textbook formats. In Taiwan, almost all of the junior

high schools select textbooks for their English classes. Some bookstores provide

CD-ROM titles that correspond to their textbook versions for teachers or sell their

reference books with a piece of CD-ROM. These two types of CD-ROM titles are considered beneficial to language learning and teaching. However, the latter is chosen as the object of evaluation because it is directly connected to the English curricula in junior high schools in Taiwan and has much influence on students’

language learning.

The next step is to assess those CD-ROM titles to determine if they fit the criteria, to identify their strengths and limitations assessing for their application to language classrooms. Before putting it into practice, the instrument was tested to ensure its suitability for language classes in school settings.

3.3 Pilot Testing

In this study, a checklist was used as an evaluation instrument and Nan-I Junior High School Book One CD-ROM was served as the object of the pilot testing to check the inter-rater reliability and the appropriateness of the evaluation. As

discussed in Chapter 2.2, Ioannou-Georgiou’s selecting software for language classes was chosen as the instrument of this study by reason of its concentration on SLA, practical criteria, and a wide range discussion of technical, methodological, and pedagogical aspects.

In order to assess whether Ioannou-Georgiou’s form was appropriate for this study, I conducted a pilot study. The procedures are:

1. Construct a set of checklists. The original version of her evaluation form was an article describing how to evaluate software. I first transformed it into a checklist style for practicality and ease of use.

2. Choose the object of the pilot testing. The latest version of Nan-I CD-ROM Book

One published in the year 2002 was chosen as the object because its textbook is

one of the popular junior high school textbooks and currently used at the two

raters’ school. The familiarity of the textbook was beneficial for raters to evaluate.

3. Find the other evaluator. In order to assess the reliability criteria, another evaluator joined the process of the pilot testing. This evaluator, Ann Wang graduated from English Department, National Taiwan Normal University and has had eight years of experiences in teaching English in Taipei County. She is still an English teacher now, and also teaches courses on the first year English this year (2004).

4. Discuss the checklists. In order to check whether the meaning of the criteria was being fully understood, the two raters discussed every question. When discussing the 107 questions, the different interpretations or the discrepancies were discussed thoroughly until the two raters reached a consensus.

5. Evaluate the CD-ROM. The two evaluators ran and assessed the Nani CD-ROM separately, then filled out the evaluation form. After the evaluation form was filled, both two copies were compared and the comments made by the evaluators were recorded.

6. Result of the pilot test. Among all the criteria, a total 107 questions, the

evaluators almost always shared the same answers on yes-no questions, as well as similar comments, except 4 items on the compatibility of methodology. The main disagreement was on whether it was a communicative program or merely a drill and practice. Finally, the two evaluators agreed that the CD-ROM was mainly a drill and practice, although they both agreed that it was relatively difficult for programmers to design communicative activities for beginners.

7. Revise the evaluation form. Ioannou-Georgiou’s form was a full coverage of

criteria needed for evaluating software. Nevertheless, the two evaluators still

thought that 17 supplementary questions should be added to the form for the

specific conditions in the junior high school setting in Taiwan and recent

technology development. Some of the supplementary questions include: How is the software to be implemented in a classroom? Does the software offer a variety of activities with options of increasing complexity? Does it require a recording system? Does it require a web-camera? Is sound or music relevant to screen displays? The revised questions were presented in italicized words in the form (see Appendix 1).

8. Construct the evaluation form. Based on the Ioannou-Georgiou article, after pilot testing and through the revision of the form, a formal evaluation tool was made for this study.

3.4 Evaluation Procedure

In the discussion of methodology, I will describe how I collected information, which included the review of related literature, the review of the evaluation criteria and checklists, and the choice of target CD-ROM titles. The second step is to perform a pilot study to ensure the chosen instrument worked and to test the

inter-rater reliability. The final step was to assess those CD-ROM titles by means of the chosen instrument.

In the process of the evaluation, I first ran through every interface and presentation of those CD-ROM titles to gain a comprehensive view of what the program was and to check if there was any potential problem that users might encounter while running the program on their computers. I then ran the program in details to compare the similarities and differences of the three CD-ROMs according to the Ioannou-Georgiou’s evaluation criteria. The last step is to discuss their

application in class.

I ran the three CD-ROM titles and made the judgment on the basis of the chosen

checklist to see whether they fit the question asked by checking on ‘yes’ or ‘no’

column to each question. When I had different opinions on the listed questions that cannot be answered simply by ‘yes’ or ‘no’, then the comments would be added to elaborate my viewpoints. After the evaluation process, I synthesized and analyzed the data and drew a conclusion. The conclusion will be presented in Chapter Four.

3.4.1 Description of Objects: Three Junior High School CD-ROMs, Book One

Currently, there are about seven publishers in the current junior high school English textbook market. This study will evaluate three CD-ROM titles, namely, Kang Hsuan (康軒), Nani (南一), Joy’s English @ Hand (佳音) for two reasons: (1) Their textbooks are among the best sellers in the junior high school English textbook market; (2) These companies are the only three software developers who have published CD-ROM titles based on their textbook versions.

The objects of evaluation are Book One CD-ROM titles published by different bookstores. Each bookstore has published four CD-ROM titles from Book One to Book Four, and they are almost presented in the same format and interfaces with few variations; therefore, it is fair to say that every piece of CD-ROM title of different publication can represent its respective others, that is, Book Two to Book Six. Thus, that is why I only choose those of Book One as the evaluation objects and make the inference to the other CD-ROM titles.

From August 2002 on (the beginning of free choice of textbooks), some of the

publishers have released CD-ROMs related to their respective textbooks. Until now

there are four CD-ROMS from Book One to Book Four, two books for every school

year. Each publisher has its own design of content and interfaces. But the format

of those four CD-ROMs (from Book One to Book Four) for each publisher is almost

the same. I adopted those of Book One as the objects of evaluation.

There are seven to ten lessons in every textbook. Most of them contain language sections such as dialogues, readings, vocabulary, grammar, and games in every lesson, which correspond to the daily teaching modes teachers usually adopt in junior high school settings. Each item and the overall structure will be shortly described below:

Item 1/ Main Page & Overall Structure: Most of the main page is aesthetically appealing to the users for the attractive colors, sound, animated characters from the textbooks, and exit & help icons. For example, Kang Hsuan’s CD-ROM has an intuitive presentation of arranging the same language sections in different lessons on the same interface. It will be convenient for the users who want to practice a certain language skill, such as dialogue, grammar or pronunciation. Besides the connection to different lessons and language sections, Kang Hsuan’s CD-ROM also contains teacher’s and student’s worksheets, lesson plans, and designs of the curricula. Apart from the common arrangement listed by lessons, Joy’s English @ Hand is arranged by different sections, such as Talk it out, Listen it up, and then by lessons.

This might be confusing to students at first, but after a few tries, it will be easy to operate.

Item 2/ Dialogue: Most of the dialogues are presented with animation and sound.

The student has choices of captions, English, Chinese, or none. The

programs also provide an option of single sentence practice. Just click on the sentence and it will be read automatically. There is an attractive design in Kang Hsuan’s CD-ROM; the dialogue will be read when moving the cursor on the animated character.

Item 3/ Reading: The reading is presented with texts, animation, and sound. The

student has the choice of captions of Chinese, or none. Some key words are

listed with pronunciation. In Joy’s English @ Hand, the continuous short stories based on The Adventures of Tom Sawyer in a dialogue form are

presented. Moreover, the sections connect with each other readily. Clicking from unit 1 to 7, students are able to read the whole story easily.

Item 4/ Vocabulary: A box of listed vocabulary is for students to choose from. Click on the word, the sound, pronunciation, Chinese, and examples with Chinese translation will be shown in the other box. There is also easy connection to click on the vocabulary of other units. Students can choose the vocabulary they want to learn, with unlimited number of times.

Item 5/ Grammar: Two to five listed sentence patterns are in each lesson. Each grammar item has clear explanations and examples. These examples are also added with the option of Chinese translation.

Item 6/ Games: Only a few games are in these CD-ROMs, in which the same game is used almost in every lesson. For example, in Nani’s ‘Having Fun’ of lesson 7, there is a memory challenge game with cards of unseen words; after flipping and matching cards with the same words, the user will gain one point.

System requirements: Microsoft Windows 95/ 98/ 2000/ ME/ XP Chinese version.

Microsoft Internet Explorer 5.5 and above. Micromedia Flash Plugin 5.0 and above.

Microsoft Office 97 and above. PC Pentium II 400 Hz CPU or faster. 64 MB RAM or above. 8x CD-ROM drive or above. 100 MB hard disk space or above.

16-bit color display, sound card, mouse, and speaker.