Chapter 7 Quantitative Analysis of Interview Results and Comparison
7.1 Quantitative Analysis of Interview Results
國
立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
Chapter 7 Quantitative Analysis of Interview Results and Comparison with Literature Evidences
7.1 Quantitative Analysis of Interview Results
7.1.-1 Quantitative results from questionnaires
Upon interviewing process of this study, the interviewees have also been invited to express their opinions in a quantitative way. Eleven interviewees gave their rating of importance for each item in the questionnaire. The results are presented in Table 7.1. Rank 5 represents the highest degree of agreement to the question. Rank 1, on the other hand, denotes no degree of consensus. The numbers in Table 7.1 indicate the number of interviewees who gave a certain rank to each specific question. Take question number 1 as an example, 2 interviewees gave rank 5, 7 interviewees gave rank 4. This result means 9 out of 11 interviewees had the similar view that the outcomes-based accreditation is better than the traditional input-based method. Two interviewees, however, had different opinion that they thought the input-based accreditation was more desirable.
Examining the distribution of rankings for each question listed in Table 7.1, it is reasonable to state that interviewees had consensus about a question if the numbers of (rank 5 + rank 4) or (rank 1 + rank 2) are greater than 8 (more than 70% of the total interviewees who expressed their opinion for this questionnaire). Following this definition, it is observed that interviewees had consensus for question numbers 1, 8, 12, 14, 15, 17, 21, 22, 23 and 27. Generally speaking, all interviewees had consensus for one third of the questions.
‧
Table 7.1 Quantitative results for the questionnaires from the in-depth interviews.
No. Question Rank a better method than traditional input-based method?
Will there be difficulties for the Ministry of Education to conduct the recognition of domestic and international accreditation agencies?
1 0 3 5 2
4
Should the HEEACT transform his role as a higher rank
supervision?
3 4 1 2 1
5
Would there be argument on the fairness of accreditation results from various agencies?
2 0 5 1 3
6
Should there be a specific percentage of programs that fail in the accreditation?
1 0 1 2 7
7
Should there be punishment rules for the programs that fail in the accreditation?
2 2 0 2 5
8
The domestic and international accreditation agencies play both roles of evaluator as well as counselor.
5 3 2 0 1
9
Accreditation always brings burden to the programs been accredited.
2 2 2 4 1
10
There is increase in financial burden for the programs to participate the accreditation.
1 2 5 2 1
‧
Accreditation will limit the freedom of progress of programs.
1 1 0 4 5
12
Diversified assessment methods are important to the program accreditation.
3 5 3 0 0
13
Core competencies should be the most important item for program accreditation.
2 5 3 0 0
14
Continuous improvement should be an important index in the program accreditation.
7 4 0 0 0
15
Training of evaluators should be an important mission in
program accreditation.
8 3 0 0 0
16
International connection of program accreditation is a main incentive for participation.
Professional mobility will bring negative effect for the job market of domestic students.
1 0 1 4 5
19
The government should provide financial support to the
international activities of entrusted by the government to exercise civil rights.
1 2 1 4 2
‧
The accreditation results can be used as the criteria for taking actions on the programs by the government.
1 2 0 2 6
22
The government should establish the guidelines for the professional accreditation agencies.
5 4 0 0 2
23
It is appropriate for Taiwan to initiate a society of professional accreditation agencies.
4 5 1 1 0
24
The accreditation results are suitable for the recognition of academic degrees in
professional license application.
1 4 2 1 3
25
Program accreditation should adopt the approval system instead of the ranking system.
5 0 4 0 2
26
The government should support the accreditation fees for the programs.
4 1 2 1 3
27
There should be the
participation of enterprises in the program accreditation.
7 4 0 0 0
7.1-2 Discussions on the consensus of questionnaire results
Among the 27 questions, the highest consensus of interviewees is that training of evaluators should be an important mission in program accreditation (question number 15). This is really true because the evaluators have to understand the spirit and
‧
國立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
process of accreditation. He or she has to conduct the accreditation according to the criteria without personal bias. A graphical presentation for the distribution of opinions for question number 15 of Table 7.1 is shown in Fig. 7.1.
Fig. 7.1 Distribution of opinions from interviewees for question number 15 of Table 7.1: Training of evaluators should be an important mission in program accreditation.
(Rank 5: highest degree of agreement; Rank 1: no degree of consensus)
The interview results for question number 14 also indicated that most interviewees agreed with the importance of continuous improvement. The higher education accreditation has recently been implemented in Taiwan. If this policy is carried out for longer time, say 10 years, most programs will be directed into a correct track to satisfy all the accreditation criteria. Faculty members will be expected to continuously think of new ideas of teaching or assessment, rather than routinely filling up the statements in self evaluation reports. This will be a major driving force to maintain the enthusiasm of faculty members to participate the accreditation process.
‧
國立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
They will always find out better ways of teaching to meet the requirements of the changing world. The up-to-date teaching methods and materials will be essential for educating the professionals in the future.
Similar to the results of question number 14, all interviewees agreed that there should be enterprises participation in the program accreditation (question number 27).
As it is recorded in Chapter 4, major part of university graduates will join companies or industries. Programs being accredited should provide evidences that core competencies, approved by the related enterprises to this program, have successfully been obtained by the graduates. Invitation of enterprises to join the program accreditation is not easy because of the busy schedules of their high rank leaders.
They should be persuaded that without their participation, there could never be supply of qualified employees to their business.
Most interviewees also agreed that the international connection of program accreditation is important for professional mobility (question number 17). According to my personal discussions with interviewees and academic faculties, Mutual recognition of higher education degrees by foreign countries is a main reason for universities in Taiwan to participate program accreditation. Accreditation of engineering programs is a good example due to the international connection through Washington Accord. Graduates from Taiwanese universities are recognized by other signatories in the Accord such as Japan, Singapore and Malaysia. This international recognition is valuable for Taiwanese graduates to apply for licenses in foreign countries where they desire to engage in a profession in those countries. There was only one interviewee who took a completely opposite viewpoint. Overall, all interviewees had very good consensus on the above four questions (question numbers
‧
國立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
15, 14, 27 and 17).
There are scattering viewpoints from interviewees for many questions shown in Table 7.1. Take question 4 as an example, interviewees had no consensus at all if the HEEACT should change its role to a higher rank supervisor. The quantitative results for question 4 in Table 7.1 reflect the interview results presented in section 4.2 of Chapter 4. Although many interviewees agreed that HEEACT should change its role as a higher rank supervisor like CHEA in the United States, there are still problems to be solved concerning the legal status of HEEACT. The graphical illustration for the distribution of opinions on question number 4 of Table 7.1 is shown in Fig. 7.2
Fig. 7.2 Distribution of opinions from interviewees for question number 4 of Table 7.1: Should the HEEACT transit his role as a higher rank supervision?
(Rank 5: highest degree of agreement; Rank 1: no degree of consensus)
Interviewees showed scattering opinions about question 20, whether government should entrust the accreditation agencies to exercise civil rights. The scattering results
‧
國立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
again are consistent with the interview statements shown in section 4.4 of Chapter 4.
Generally, more than 50% of interviewees did not agree that the accreditation or evaluation agencies were entrusted by the government to exercise civil rights. There would be no dispute that The Ministry of Education in Taiwan is the only authority to take actions on higher education programs. The graphical presentation for the distribution of opinions on question number 20 of Table 7.1 is shown in Fig. 7.3.
Fig. 7.3 Distribution of opinions from interviewees for question number 20 of Table 7.1: Domestic or international accreditation agencies are entrusted by the government to exercise civil rights.
(Rank 5: highest degree of agreement; Rank 1: no degree of consensus)
Finally, the results for question 24 in Table 7.1 present scattering ideas of interviewees that are also consistent with the interview statement shown in section 4.6 of Chapter 4. Half of the interviewees showed positive opinion on the issue that the accreditation results can be used for the recognition of academic degree in professional license application. The other half provided negative viewpoint. Since no
‧
國立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
consensus is observed in this study, the accreditation results have very low possibility to be used by the Ministry of Examination in revising the present law of approving licenses.
7.1-3 Comparison between in-depth interview and quantitative analysis results
The major efforts of this study have been put into the qualitative in-depth interviews. The supplementary quantitative questionnaire analyses, however, also yielded certain direct recognition for the degree of agreement from interviewees on various topics. It is interesting to make a general comparison between these qualitative and quantitative interview results.
The comparison of in-depth interviews and questionnaires has been generally examined on topics listed in Chapters 4 to 6 and Table 7.1. Generally speaking, the similarity between the results from these two approaches is greater than 80%. This means, although the interviewees spent little time to answer the 27 questions in the questionnaire, their comments are consistent with those from the in-depth interviews usually more than one hour for each interviewee. Five typical topics are selected for comparison in this section. Owing to that the questionnaires had been completed by only 11 interviewees (out of the total number of 15), the comparison is not on a strictly accurate basis. The other reason is that the qualitative in-depth interviews are not easy to quantify. The aim of such a comparison intends only to demonstrate the general consistency of interview comments.
Take the outcomes-based evaluation or accreditation as an example, approximately 55-60% of the in-depth interview results agreed it is a better method than the traditional input-based approach. The quantitative results from the
‧
questionnaire showed a similar trend of 60-70% agreement. This comparison indicates that although more than half of the interviewees favored the outcomes-based method, they still did not agree that the input-based method can be completed replaced. The other example is about the importance of training evaluators. Both the qualitative and quantitative results showed the same degree of agreement of 90-100%.
The comparison results are shown in Table 7.2.
Table 7.2 Comparison between in-depth interviews and quantitative analyses on 5 selective topics
Selected topics Percentage of agreement from 1. The outcomes-based evaluation or
accreditation is a better method than the traditional approach?
55 - 60% 60-70%
2. Should HEEACT transform his role to a higher rank supervision?
60-65% 55-60%
3. Is it important for international connection of evaluation or accreditation?
70-80% 80-90%
4. Is it important for training the evaluators?
90-100% 90-100%
5. Should there not be a passing ratio of evaluation or accreditation?
80-90% 80-90%
‧
國立 政 治 大 學