Based on the methodologies mentioned in chapter 2, this chapter explains the framework of the research, followed by instrument and the research method.
3.1 Research Design
According to the research motivation, research purpose and methodologies, the research framework of this study is shown in Fig. 3.1.
Educational Decision-making evaluation
GRA English coursebook
chosen
AHP English Test
OT/IRS GRA
ISM
Optimal remedial instruction path
GSP GSM
Experts’ Concept Structure
Students’ Item Response
Structure GSP GSM
GRA/GSP CTT
Test Item difficulties/
Test Design Review//
Fig. 3.1 Framework of the thesis
46
Fig. 3.1 shows the framework of the study which starts from taking English tests, including grammar tests and reading tests. Then the
gamma
values of test items are calculated via Matlab toolbox (Wen, et al, 2006; Wen, et al, 2010). Based on thegamma
values, GSM structure of test items can be established, which shows the hierarchy and relationship between test items.Finally, test designers can review the test items and decide to revise the test or provide remedial instructions.
Moreover, the study also uses GRA to the educational decision-making field, that is, the study tries to find the objective method for choosing an English coursebook. By combing GRA and AHP, the experts’ opinions are analyzed in an objective, scientific way.
47
3.2 Significance of the Research
The originality of this research could be concluded as follows:
1. The students’ test performances are clustered based on the
gamma
values obtained from GRA calculation, and the results are shown clearly in the GSP figure through Matlab GUI toolbox. Besides, the
gamma
values are between 0 to 1, and the GSP can be applied to both large pool of data and small data, which is better than traditional S-P chart.
2. Also, the GSM figures present the learning concepts in different levels, and it is an innovative way. In addition, the paper draws two-dimensional space figures which not only show the relationship between elements, but also present the hierarchy of each element (Wang et al, 2011a~2011d;
Wang et al, 2012a~2012c) which proves that this is a prospective research of cross-domain integration with the originality and feasibility.
3. In current English teaching field, it seldom uses engineering software to assist and analyze the data. In past software development, there are some achievements, but it lacks the discussion of Matlab GUI toolbox to cluster students’ test performances. The proposed toolbox in this paper aims to assisted analysis and validation. The Matlab GUI toolbox used in this paper is not only user-friendly, but also has powerful analytical functions.
4. The interface of Matlab GUI toolbox shows both the students’ test performances and test item difficulties in a user-friendly interface. This novelty approach is efficient, and it is suggested to become an information science tool in the educational field.
5. The paper uses Grey-AHP method (GRA and AHP) to evaluate the English coursebook chosen process in a more objective way. It presents
48
the coordinates’ distance from the origin. Comparing with the traditional math methods, the gamma values reached by GRA can be positioned and sorted correctly. Also, the results are clear to present both the weighting of each professional and English coursebook selection criteria.
This method is also suggested to be used in other decision-making fields, such as business or medical fields.
49
3.3 Research Instrument
The research instrument and the relevant experimental tests are shown in Table 3.1.
Table 3.1 Research instrument and the relevant experimental tests
Instrument Relevant experimental tests
• English Proficiency Test (one-hour test), including 30 listening questions, 10 grammar questions and 30 reading questions (please see Appendix 1 &
Appendix 2).
• Matlab GUI toolbox
1. Concept diagnosis of English grammar
2. Evaluate English test item difficulties
• English coursebook selection criteria
3. Evaluate the English coursebook chosen process (weighting)
In the first experimental test, there are four verb concepts included in the grammar test, and they are introduced in Table 3.2.
Table 3.2 The verb concepts in the grammar test (Adapted from Harmer, 2007)
Verb forms Examples
Present tense I go to work every day.
He always goes to school by bus.
Simple past tense They went to Taipei last weekend.
She visited the museum last year.
(table continues)
50 Table 3.2 (continued)
Verb forms Examples
Be verbs I am from the United States.
They are my grandparents.
Present continues and present tense
Look! They are crossing the road.
She can’t answer the phone because she is taking a shower.
Next, the purpose of this English proficiency test is to measure whether freshman students' English ability reach CEF B1 level (Common European Framework B1 Level) or not, approximately equivalent to the TOEIC Test score 600. To distinguish the proficient learners from the less proficient ones, a second instrument, Matlab GUI toolbox, is used to calculate students’ gamma values and cluster them.
For experiment 3, the author uses the English coursebook selection criteria to summarize five important factors (Cunningsworth, 1995; Ellis, 1997; Richards, 2001). Then the soft-computing calculation methods of AHP and GRA are used to decide the order of importance for the factors in an objective and scientific way.
51
3.4 Research Method
The research mainly uses the test scores obtained from freshman English classes, and the interview outcomes from experienced teachers. Next, the data are evaluated through the soft-computing calculation methods, such as OT/IRS, GRA, GSP, GSM, ISM and AHP. Their explanations are as follows:
1. OT/IRS: to present students’ knowledge structure based on their responses toward test items;
2. ISM: to present the experts’ concepts and structural analysis;
3. GRA: to calculate gamma values of test items and students’ responses and sort the sequence of objects;
4. GSP: to cluster students’ performances and test item difficulties;
5. GSM: to present the hierarchy of students, students’ and experts’ concept structures;
6. AHP: to decide the weighting based on experts’ opinions.
According to the methodologies and relevant definitions, the experimental tests of the paper are explained in Table 3.3.
52
Table 3.3 The experimental tests of the paper
Experimental tests Purposes Analysis
methods 1. Concept diagnosis of
English grammar
It aims to propose the integrated approach on concept diagnosis for English grammar to ESL students in a Taiwanese classroom.
GRA, ISM, GSM, GSP, IRS
2. Evaluate English test item difficulties
It is to introduce Grey Relational Analysis method (GRA) in evaluating test item difficulties and students’ performances in the educational field.
GRA, CTT, Matlab toolbox
3. Evaluate the English coursebook chosen process (weighting)
It is to apply soft-computing method to the educational decision-making field.
GRA, GSP, AHP, GSM
Table 3.3 shows that there are three experimental tests being included in this thesis, and the methods, like GRA, ISM, GSM, GSP, IRS, CTT and AHP are used in the experimental tests for different purposes, respectively.
53