Chapter 3 Design of This Research
3.2 Method of this research
國
立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
professionals and technical societies. The evaluation/accreditation process is based on those standards.
Graduate attributes （畢業生核心能力）: These are the skills and understandings that the students should develop upon graduation for the social goods in unknown future. The graduate attributes are also known as core competencies.
Self-evaluation report（自評報告）: This report, prepared by the educational program, describes its educational objectives, practices, procedures and outcomes.
Continuous improvement（持續改善）: It is a process of data collection and interpretation of the facts in order to achieve the outcomes continuously.
Professional mobility （ 專 業 人 員 流 動 ） : This is a mission to facilitate professionals to across international borders through mutual recognition of credentials.
Substantial equivalence（實質相當）: It means that the education programs have comparable contents and experiences. It implies that all graduates have reasonable confidence as entry level professionals.
Program evaluators （ 評 審 委 員 ） : The evaluators are well qualified and committed professionals to conduct evaluation or accreditation and to give balanced professional judgments.
3.2 Method of this research
3.2-1 Design of interview problems
‧
國立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
The trend of evaluation by HEEACT, IEET, AACSB, or other recognized organizations by the Ministry of Education will be directed to the outcomes-based accreditation. The spirit of the outcomes-based accreditation is to change the traditional view that inputs were equivalent to quality. The outcomes-based model also changes the traditional in-classroom assessment to a much broader assessment of the program. For example, the outcomes-based accreditation examines what the students have learned and what special skills the students have developed upon their graduation from the programs.
The outcomes-based accreditation has been effectively applied by IEET for evaluating the higher education engineering programs in Taiwan. It was stated by IEET (劉曼君，2007) that outcomes-based method was used in examining the core competencies of graduates. The author pointed out that the engineering accreditation agency of USA, ABET, Inc., has applied the outcomes-based method since year 2000 and established the Engineering Criteria 2000. The ABET, Inc. has impelled the other members in the Washington Accord to initiate the Washington Accord Graduate Attributes in order to meet the equivalent core abilities among all members. The accreditation mechanism of IEET is shown in Fig. 3.1 (ieet.org.tw/download). It is demonstrated in Fig. 3.1 that the accreditation conducted by IEET examines the objectives of the program and the outcomes of the students through appropriate assessment methods. The traditional input-based evaluation approach emphasize on the quantitative indexes such as the grades of student and the ratio of students to faculties. The outcomes-based method, however, weights more on the ability of students after they finished their learning processes.
‧
國立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
Fig. 3.1 The accreditation mechanism of IEET accreditation.
It is also noticed from Fig. 3.1 that each program has its own objectives of education. The outcomes were derived from these objectives to present the special characters of the individual program. After establishing the objectives and student outcomes, the design of curriculum and assessment methods can be achieved. The cyclic sign in Fig. 3-1 demonstrates the continuous improvement process. IEET has stated that this outcomes-based accreditation does not compare various programs by common standards. It examines if the program does satisfy the objectives and student outcomes defined by the program itself (劉曼君，2008).
The outcomes-base education has also been discussed by Hsu and Lin (2005). It was stated that the outcomes-based education (OBE) was an important subject of US education reform in 1990’s. It paid more attention on the behavior of students and the development of their abilities through courses design. It was taken as a key point to enhance the quality of education (Spady, 1994). The major components of OBE include criterion-referenced measurement, mastery learning, accountability and competency-based education. It is observed that the spirit of OBE has fully been
‧
國立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
applied by the accreditation mechanism of IEET.
Hsu and Lin (2005) stated that the progress of OBE in US met obstacles because the dissenters argued the assessment of learning outcomes had difficulties, especially the standard of value and the learning attitude. The OBE has been discussed in US for 10 years without significant success. Hsu and Lin described that Taiwan also has experienced education reform through different steps since 1987. Although the concept of OBE was not directly implemented in Taiwan, the competency-directed education has gradually replaced the knowledge-oriented phase. The authors listed some difficulties that Taiwan has to face: the practice of required competencies, the argument of standard of value and teaching of knowledge, and the progress of education reform. There should be in-depth discussions of course design and development of assessment methods.
From the above observations, it is confirmed that outcomes-based learning and accreditation is the future trend of higher education evaluation in Taiwan. The objective of this research is to collect information through interviews and analyze the opinions of reaching the ultimate goal of outcomes-based higher education.
Generally, the outcomes-based evaluation or accreditation is new to the higher education institutions in Taiwan. We have been accustomed to the traditional input-based teaching format. It is the main purpose of this research to investigate the opinions and criticisms of higher education professionals, teaching faculties, and policy makers about the outcomes-based evaluation and accreditation. This study will be conducted through interviews. During the interviews, many related issues will also be included. These issues include if the higher education evaluation could be conducted by government recognized professional organizations. From the viewpoint
‧
of faculty members, their expectations for the outcomes-based evaluation will be collected and analyzed. The following Table 3.1 lists the proposed questions that will be delivered in the interviews. The Chinese version of Table 3.1 with detail subtitles is listed in Appendix A of this study.
Table 3.1 The proposed questions for the interviews
No. Questions during interview Proposed
interviewees 1. Do you agree that the outcomes-based evaluation/accreditation is appropriate
for various fields of higher education programs in Taiwan?
Faculty members
2.
Do you agree that the evaluation of higher education programs by HEEACT can be conducted by individual professional organizations? What are the advantages and drawbacks of this change?
Faculty members, Government officers,
Policy makers
3.
The government intends to recognize professional commissions in various fields to carry out higher education evaluation or accreditation. Do you think this will cause any arguments?
Faculty members, Government officers
4.
Do you think there should be a certain percentage of higher education programs failed in the evaluation/accreditation in order to maintain the education quality? What should be the penalty to the failed programs?
Faculty members, Government officers
5.
Do you agree that the higher education evaluation/accreditation is not a burden to the faculty members? Does it impair the free development of education programs?
Faculty members
6.
If the outcomes-based evaluation/accreditation is desirable, what are the best ways to help education programs to identify their graduate attributes and complete the continuous improvement?
Faculty members
7.
What are your opinions on the global connections through higher education evaluation/accreditation?
Faculty members, Government officers
8. To your opinion, what are the benefits to the students through program evaluation/accreditation?
Faculty members, Government officers
9.
Do you agree that the program evaluation/accreditation is the exercise of civil right?
Faculty members, Government officers,
Policy makers
10.
What are your suggestions about higher education evaluation/accreditation?
Besides engineering programs, Should other fields be accredited by internationally recognized commissions?
Faculty members, Government officers
‧
3.2-2 Quantitative questionnaire to the interviewees
In order to obtain some quantitative feedbacks from interviews, a questionnaire is designed that the interviewees will provide quick overall answers before the end of interviews. An example is shown in Table 3.2. In the following table, number 1 indicates no degree of consensus, while number 5 represents the highest degree of consent. The Chinese version of Table 3.2 with detail 27 questions delivered in the interviews is listed in Appendix B.
Table 3.2 An example of questionnaire in the interview
Questions 5 4 3 2 1
The outcomes-based evaluation or accreditation is applicable to all fields of higher education programs.
The evaluation or accreditation commissions should maintain a certain percentage of disqualified programs.
The conduct of higher education evaluation or accreditation is an execution of civil right.
There should be penalty for the disqualified programs from the government in additional to the market driven mechanism.
The evaluation or accreditation for technically oriented programs should be regionally or globally connected.
The evaluation or accreditation for non-technically oriented programs should be regionally or globally connected.
The evaluation or accreditation results should be involved in national examinations for degree recognition.
There should be division of labor for program evaluation of various fields. Government should play the role of supervision.
Mutual recognition of cross border higher education degrees and professional mobility should be a national policy.
The results of evaluation or accreditation should be associated with national progress policy to modulate the route of
disqualified higher education programs.
‧
3.2-3 Proposed list of interviewees
According to the proposed questions in the interviews listed in Table 3.1, the interviewees should include professional faculties, government officers, and policy makers. A list of interviewees in this study is shown in Table 3.3.
Table 3.3 The list of interviewees of this study Field of
Classification Code of Names Title and Professional Area
Academic
A Professor, Engineering
B Professor, Education
C Professor, Education
D Professor and Dean, Business School
E Professor, Law School
F Professor, Liberal Art
Academic and Evaluation Agency
G Professor, Education
H Professor, Engineering
I Professor, Education
Evaluation Agency
J Professor, Medicine
K Professional, Education
L Professional, Education
Government M Official, Education
N Official, Education
People’s Representative
Body
O People’s Representative
The proposed interviewees include the administration officers of the government, and the executive officers of the evaluation/accreditation organizations
‧
國立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
for professional areas of engineering, medical and business schools. The above members also include professors who are experts for educational administration or who have participated in the accreditation. There is a member from the Legislative Yuan who can express their opinions on the civil rights of evaluation/accreditation, as well as the legislation issue of HEEACT. This study interviews the departments that passed the evaluation by HEEACT or accreditation by IEET. The interviews also include department that was put into the pending list of HEEACT. Their arguments about the evaluation results reflect the voices from the programs being evaluated.
3.2-4 Hypotheses of this study
The in-depth interview is a commonly used qualitative research method in social sciences. In this research, the aim is to collect and analyze the responses of higher education evaluation or accreditation from experts. The following items are the most important hypotheses in this study:
(1) The outcomes-based evaluation or accreditation will replace the traditional input-based method for all the academic fields in Taiwan.
(2) The Higher Education Evaluation & Accreditation Council (HEEACT) will soon change its role to a higher rank supervision position.
(3) The aim of higher education evaluation is to help the programs for continuous improvement. No specific percentage of elimination is required.
(4) The documentation works brought heavy burden to the programs being evaluated.
(5) Training of evaluators is the most important issue. This is the key point to the
‧
國立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
success of higher education evaluation or accreditation.
(6) Global connection of higher education evaluation or accreditation results is an important reason to attract programs. It is important for professional mobility.
(7) The conduct of higher education evaluation or accreditation by non-governmental commissions is not a form of exercising civil rights.
(8) The higher education evaluation or accreditation results can be taken as a requirement for license examination, at least for engineering programs.
3.2-5 Restrictions of this method
There are some restrictions for this research using interview method, as listed below:
(1) The interviewees have various academic backgrounds including humanity, business, science and engineering. Since business management, medical and engineering institutions have international link for accreditation at the present time, it is expected that major interview comments will be focused on these professional area.
(2) This study intends to interview primarily the professionals of high ranks in universities, evaluation/accreditation commissions, or people’s representative bodies. The responses from the authorities of programs (Deans of Colleges, Chairpersons of Departments) being evaluated or accredited may not be effectively collected.
‧
國立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
(3) Similar to the above statement, the responses from the faculties and students of the programs being evaluated or accredited will not be collected in the present study.
The higher education evaluation or accreditation has been lunched in Taiwan in the past 5 to 6 years. The purpose of this research is to collect and analyze the comments from high rank officials about the policy of higher education quality control. Extensive studies by questionnaires will be expected as future work.